Talend Studio: Custom libraries process

TalendSolutionExpert — Tue, 23 Jan 2024 09:57:00 GMT

This document describes the process for working with the third-party libraries (custom libraries) used within Talend.

Custom libraries are needed for Talend components to work properly. These libraries are not delivered by Talend for contractual and licensing reasons, so they must be downloaded by the Studio.

This content is applicable for old 6.4.1 versions of Talend.

Content and process:

This document also describes what you can do if Talend is installed in an environment without an internet connection.

Configure TAC

Among other things, TAC needs the following configuration:

Note: Software Update is not part of the custom library process. It is the mechanism used to download patches and updates.

So you have a configuration for Nexus, the artifact repository, but this configuration is only the place for TAC to look for artifacts for deployment in the Job and ESB conductors. Studio has a separate configuration for the artifact repository to which it publishes, as detailed later.

Set up a user

The user needs to be set up with the appropriate credentials for the source code system.

Set up a project in TAC

Set up a project as shown below:

Assign rights to the appropriate user(s)

Assign rights to users as follows:

Start Studio

Before Studio starts for the first time, the Maven .m2 repository has not been set up:

When Studio starts for the first time, it sets up its local Maven repository. This is where it stores libraries delivered by Eclipse plugins that are not transferred to Nexus with the custom libraries.

At this point, you can see that the custom libraries directory – stored in org.talend.libraries – is still (almost) empty:

and the custom libraries in Nexus are also still empty:

Log in to Studio

This is what happens behind the scenes:

Here you can see why it is a bad idea to leave hostnames as localhost. When the Nexus location is passed to Studio, it would get localhost as the host name, and because Nexus is not local to Studio, Studio is unable to find Nexus to interact with it. As soon as the user has logged on to Studio, you see some libraries appearing in Nexus, but not the third-party libraries:

Now Studio will request that the custom libraries are downloaded, which need to have the licenses accepted. These are the custom libraries:

Clicking Finish starts the download and you must now accept the licenses, which is the crucial part:

Note: the licenses will need to be explicitly accepted whether or not Studio downloads the libraries from the internet or internally from Nexus. Then you see the libraries downloading into the Studio’s Maven repository:

And you see them appearing in the Nexus custom libraries repository:

A note about the command line

Even though the command line is running, the custom libraries do not appear automatically in its Maven repository.

It will only bring these across from Nexus when it needs to build them into an artifact, such as when a Publisher or Job Conductor task is run. As mentioned earlier, try to avoid using these methods to publish or build artifacts and as much as possible, publish artifacts to Nexus and deploy those.

Publish from Studio

Before publishing from Studio, you must configure the Nexus artifact repository in the Preferences.

Installing in an environment without an internet connection

Many companies do not allow software such as Talend to have access to the internet. These are most typically financial institutions, but any company or organization handling sensitive data will probably impose internet access restrictions. This obviously poses a problem for a mechanism that relies on access to the internet to access third-party files. How can you solve this problem?

There are two main ways:

Install at least one Studio into an internet-connected environment.
Upload a backup of the Nexus custom libraries repository.

Install Studio into an internet-connected environment

As you have already seen, once the third-party libraries are uploaded into Nexus, there is no further requirement for access to the internet, even for Studio. So the easiest solution is to install one Studio on a machine and configure it with access to TAC and the Nexus installation. This will enable third-party libraries to be downloaded into Nexus and from there, the libraries are accessible from any Studio that connects to TAC, regardless of whether or not that Studio has internet access.

Upload a backup of the custom libraries

This involves having an “external” installation with access to the internet – often your own laptop - where the third-party libraries have already been downloaded into Nexus. Once the libraries are loaded into Nexus in the “external” installation, a backup of the libraries is made and then restored to the unconnected installation.

The process to make a backup

By default, Nexus is installed with the TAC, in the TAC directory:
Within that, navigate to the storage for the Talend libraries in the custom libs repository:
Make a Zip backup of the entire directory.

The process to “restore” the backup to the target installation

Log in to the Nexus in the target installation as an admin user: the default username is admin, the default password is Talend123.
Click the Repositories menu item on the left:
Right-click the talend-custom-libs-release repository and select Put out of service:
Navigate to the storage for the Talend libraries in the custom libs repository, and unzip the backed up libraries archive:
Right-click the talend-custom-libs-release repository and select Put in service:
Right-click the talend-custom-libs-release repository, select Rebuild Metadata, then right-click and select Repair Index.

The libraries are now available for any Studio that logs in to the TAC.

article Talend Studio: Custom libraries process in Official Support Articles