This article covers connecting Talend to Git repositories using the SSH protocol. Git is a version control system that Talend supports, similar to SVN. Git can store repositories locally or remotely. The key difference is centralization. In SVN, the Repository is in one centralized location. The advantage with Git is there can be a local copy of a repository, and you can push, pull, clone, and merge not just from a central location, but also from another developer’s code base. You can look at diffs to compare code changes and decide which to keep.
While using Talend, you may not need to do most of these; you will likely just be cloning, pushing (committing), and pulling from a centralized location (either from a remote URL or a local Git Server installation).
Content:
- Requirements
- Operating System Recommendations
- Connecting to Git with SSH
- Connecting Talend Administration Center
- Caution
- Git Troubleshooting Guide
Requirements
-
Git Bash
-
Git GUI client (required if using Windows)
https://git-scm.com/download/gui/windows
If you are using Windows to house your TAC, then Git Bash or a Git client needs to be installed to create an SSH Key. If you are using Linux, OpenSSH comes automatically with the OS.
-
One of the following supported distributions of Git
-
Bit Bucket (Talend version 6.2.1+)
-
Git Hub
-
Git Lab
-
-
Talend version 8.x (refer to the license email you received from Qlik Support for your version)
Operating System Recommendations
-
Red Hat Linux 7
-
Windows 7
-
Windows 10
Connecting to Git with SSH
Connections to Git must be set up individually for each Talend Studio user. For example, if your team has five developers who wish to connect to SSH-enabled Git from Talend Studio, then all five need to configure Git on their machines. If multiple developers use the same Talend Studio installed on a single machine, then each of them must follow this procedure because each user must have their own SSH keys.
If any users configured Git on a previous machine, but now are using a new laptop/desktop, they need to configure Git again on the new machine.
-
To verify that an RSA key can be created, open a Git Bash window and run the command:
ssh -v
The output should be similar to the following:
-
List the contents of the ~/.ssh directory by running the command:
ls -a ~/.ssh
The result should be similar to the following:
$ ls -a ~/.ssh ls: /c/Users/emmap1/.ssh: No such file or directory
If a .ssh folder doesn't exist, you will see a No such file or directory error. You must create a .ssh folder under c:\Users\user\.ssh.
-
Type ssh-keygen. You will be prompted to save the key in the default location: c:\Users\user\.ssh\id_rsa.
Entering a passphrase is optional when prompted. The complete command will look like the following:
$ ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/c/Users/emmap1/.ssh/id_rsa): Created directory '/c/Users/emmap1/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /c/Users/emmap1/.ssh/id_rsa. Your public key has been saved in /c/users/emmap1/.ssh/id_rsa.pub. The key fingerprint is: e7:94:d1:a3:02:ee:38:6e:a4:5e:26:a3:a9:f4:95:d4 emmap1@EMMA-PC
-
After the RSA key pair is generated, review the .ssh folder and verify that the id_rsa and id_rsa.pub files are listed:
-
Create an SSH config file and add these parameters:
Host bitbucket.org IdentityFile ~/.ssh/id_rsa
The space before and after IdentityFile is required – do not delete this.
-
In Git Bash, type the following command to add bitbucket and create a known_hosts file:
ssh -T git@bitbucket.org
If you are using Git 2.10 on Windows, there is an issue with ssh-agent meaning it will not start automatically.
-
To start it in Git Bash, type ssh-agent -s.
-
Type eval $(ssh-agent).
-
Type ssh-add –l. This should state that you have no identity.
-
Add an identity by typing ssh-add ~/.ssh/id_rsa.
-
Run ssh-keygen to overwrite the id_rsa file. Your ssh-agent was not running when the previous file was created, so an improper key may have been generated as a result.
-
-
From your Git distribution, click your Avatar > Settings.
-
Scroll down to the SSH keys portion in the Security tab.
Bit Bucket
Git Hub
Git Lab
-
Perform a cat command on the id_rsa.pub file:
cat ~/.ssh/id_rsa.pub
-
Copy the contents of the id_rsa.pub file and place it into the Key portion of the SSH Keys tab. Click Save.
Bit Bucket
Git Hub
Git Lab
If you are using GIT Bash 2.18 or newer, open the id_rsa (private key) file in Notepad or any other text file editor. If the key type is OPENSSH (shown on the first line of the file), then you need to convert the private key to a RSA type key.
-----BEGIN OPENSSH PRIVATE KEY-----
To convert the private key to a RSA type key, perform the following steps:
-
Launch PuTTYgen.
-
Load the id_rsa file that you created in Step 3 by selecting File > Load private key.
-
Click OK.
-
Click Conversions, then select Export OpenSSH key. You are prompted to save the key, overwriting the existing id_rsa (private key) file stored in the SSH directory, for example, c:\Users\user\.ssh\id_rsa.
-
Open the new id_rsa file, and confirm that the first line reads:
-----BEGIN RSA PRIVATE KEY-----
-
Studio/TAC will now be able to connect successfully with this RSA type private key.
-
Git is now set up for use with SSH. The next steps will connect TAC to it.
Connecting Talend Administration Center
-
Create a new repository.
Bit Bucket
Git Hub
Git Lab
-
Expand the I’m starting from scratch section, then copy the Git URL shown after git remote add origin, in this case:
git@bitbucket.org:arthur-talend/bitbucket-test.git
For Bit Bucket and Git Hub, the Git URL will be searchable in the project itself, refer to the images below:
Bit Bucket
Git Hub
Git Lab
-
Create a new repository in TAC, enter the correct details (including the URL you just copied), check the connection, and then click Save. You don't need to provide the password in the project settings. TAC will be using the Private key to authenticate.
If you are not using your own account to run the Talend services, your .ssh folder may be defined as a different directory. The environment can be set up to work around this issue.
-
In the System Environment Variables > PATH, add your Git cmd path, normally: C:\Program Files\Git\cmd.
-
Alternatively, use the HOME variable, and establish your HOME variable to a folder of your choice. For details, see the Git Environment Variables page.
-
-
Refresh your repository page for the repo; it should look a little different now.
Notice the SSH URL, and the Branch and Watcher.
-
Log in to Studio (remember to give your user the correct credentials as well as project authorizations).
Continue with the instructions in this step if you are using SSH2; if not, skip to the next step.
These instructions will be performed on a machine with Talend Studio installed, but that is unable to connect or obtain a connection.
-
If you are having issues connecting Studio to Git, use the ssh-keyscan command to add the host URL to the known_hosts file by typing the following:
ssh-keyscan –H {host.url} >> known_hosts -
Create and append the public RSA key to the authorized keys file.
Type the following command:
cat id_rsa.pub>>authorized_keys
-
If you still can't connect to Git, open Git Bash and type this command to find the location from which Git is trying to read the keys:
ssh –vT git@bitbucket.org
The first two parameters in the output show from where the configurations and files are being read.
After running the ssh –vT command, you should be logged in as the Git user if it is properly set. However, sometimes the ssh -vT command isn’t sufficient, and your user will be logged in as “anonymous”. In this case, you will need to use this command (substitute the correct email address):
ssh-keygen -t rsa -C “you@email.com”
-
Repeat the steps above to place all of the files–known_hosts, authorized_keys, config, and id_rsa.pub–within the folder from which Git is reading the files (identified above) and then try logging in again.
-
-
To finish, navigate to the source tab within Bitbucket, then click your repository. Notice that the structure now looks like that of a correct Talend repository.
Caution
Talend 6.1.1 and Bitbucket on SSH does not work. Talend supports Bitbucket starting in 6.2.1, and thus it is not possible to set up 6.1.1 with SSH in Bitbucket, only HTTPS. If you try to set up Bitbucket, SSH, and Talend 6.1.1, you will receive a mismatched protocol error in the console.
Git Troubleshooting Guide
What is the first troubleshooting information needed?
The most important piece of information are the logs. The logs will normally tell you what the issue is, such as the RSA fingerprint is not recognized, or there is an unknown host in the logs. Other possible issues are that the user is wrong or the host is incorrect. For instance, in the URL:
git@bitbucket.org:user-talend/bitbucket-test.git
- The first git is the user; if you have a local server implementation, you might not have called your user git so try the user you used to log in (or try the service account).
- The host in the URL is bitbucket.org, but sometimes the host is not found within the hosts file. Check to ensure that there are no inconsistencies.
The new URL could look something like this:
talend@user-pc.talend.org:user-talend/talend-repo.git
What issues have been encountered when trying to set up SSH?
The most common issues are due to connectivity with the central repository. The Log in to Studio section offers a couple of troubleshooting steps that can check for inconsistencies in the implementation of SSH. Another issue is if you are not using OpenSSH as the SSH client. For example, Tectia does not have the same SSH commands, and there are some limitations that exist. This will require assistance from the Administrator for the specific SSH implementation.
Git is set up, my client is implemented, and when I check the connection in TAC it says that the connection is OK. Yet when I try to connect, why doesn’t it allow me to save?
The most common reason for this is that the RSA key is not trusted. When this occurs, you will need to go back to the SSH key deployment section of your account. Perhaps you may want to read the key, as there may be an issue with it. Another option is to add the key directly to the project, instead of to the user. To do this, go to the project itself, and under Settings > Deploy keys, add the RSA key to the project.
I'm getting an error that states “stashing local changes did not successfully complete”.
There are two things to check:
- Permissions can be an issue. Check the folder permissions on the folder being complained about, and see what the distributed locking mechanism is. For instance, if you are on an NFS, try using the Git Repo URL from Bit Bucket or Git Hub instead to see if it works. If it does, then that's how the OS was configured.
- This could be an issue with the workspace. Try using a new workspace, by deleting the existing one and creating a new one, or by specifying a new workspace. If it’s CommandLine, make a copy of the commandline-workspace and then delete it, letting CommandLine recreate the entire workspace.
I'm getting the error "Accessing the same repository with different protocols is not supported" after following all the configuration steps.
If you try to establish two Projects in TAC that point to the same Git project, but using two different protocols, for example, the https:// URL and then the ssh:// URL, you might get the following message:
Accessing the same repository with different protocols is not supported.
You might get this message even if you deleted the original project; you are not trying to create a duplicate project, but you still cannot add the project. To fix the problem, stop TAC, clear the TAC cache, then restart TAC.
You can add a HOME variable as a workaround; for details, see the Git Environment Variables page.
Background: If you are running TAC (or another Talend server) as a Windows service, it will most likely be running under the Local System account, so this is the account in which Talend will look for SSH keys when it tries to connect to the remote Git server. This means you either need to set up the Local System account to have its own SSH keys, or you can create the HOME variable and point it to another user that has their SSH keys already set up (they have a public key created and added to the remote Git server as described in previous steps).
Environment
