This learning path is an introduction to continuous integration and how to implement it in a Talend environment. It focuses on the basic capabilities of Maven and Jenkins, illustrating how to implement an automated build that reacts to code changes in the source repository. It shows you how to connect Talend software to these tools to implement a complete CI pipeline, starting with the development of a Job and its unit tests, then building and deploying it to an artifact repository, and finally deploying it to Talend Management Console.

A second section illustrates how to use Talend Studio to build and automate these tasks using Jenkins and Azure DevOps.

• Target Audience: Data Integration developers
• Prerequisites: Completion of Talend Data Integration Basics and Talend Data Integration Advanced, and basic knowledge of computing.
• Availability: This learning plan is available as part of a Talend Academy Learning Subscription, and as a private virtual or private onsite instructor-led course.

If you are already a Talend Academy subscriber or want to access the publicly available content on the platform, go to the Talend Academy Welcome page to log in or create an account.

Overview

Talend Continuous Integration (CI) is now fully compliant with Maven standards, and Continuous Integration and Deployment (CI/CD) with Talend has never been easier.

This tutorial illustrates how to automate CI/CD with Talend 7 and Jenkins, by complementing current Talend documentation. For more information, see the Talend Software Development Life Cycle Best Practices Guide.

Requirements

• Talend platform Edition 7.0.1

• Jenkins

• Source code management supported by Talend

Architecture

This example uses a Continuous Integration server (Jenkins) by leveraging Talend CI Builder, Bitbucket as a service for the code repository, and the Talend Nexus Artifact Repository.

You can continue to use TAC to publish Jobs to a Nexus repository using the Publisher page, but the Publisher page is deprecated in TAC, so Talend recommends using a Maven build.

Talend CI Builder is a Maven plugin, delivered by Talend, that transforms the Talend Job sources to Java classes using the Talend CommandLine application. This allows you to execute your tests in your own company Java factory.

• The overall high-level architecture:

  
  

• Talend platform architecture and components on a Windows Server machine:

  
  

• CI server architecture and components on a Red Hat machine:

  
  

Installing the Talend platform

Install TAC and CommandLine

1. Download the Talend-Installer, the Dist file, and your License.

2. Extract and store them in the same folder.
   

    

3. Double-click the installer.

4. Choose Advanced Install, choose Custom, then browse to your License File.
   

    

5. Select Talend Administration Center and Talend Command Line. Click Next.
   

    

6. Select Talend Administration Center, Talend Command Line, and Talend Server Services. Click Next.
   

    

7. Choose Install an embedded tomcat8 server on the drop-down list. Keep the default Create TAC administrator user setting. Click Next.
   

    

8. Choose Embedded H2 database on the drop-down list and select Install Nexus server with TAC.
   

    

9. Keep the default Nexus Port.
   

    

10. Keep the default CommandLine port.
    

     

11. Select Install Talend Administration Center as a service and Install Talend Command Line as a service.
    

     

12. The Talend platform will install.
    

Set up Bitbucket as a service

1. Create a Bitbucket account or use your existing one.

2. Click the plus [+] sign.
   

    

3. From the CREATE menu, select Repository.

   

    

4. Create a new repository and give it a name. Select Git as the version control system, then click Create repository.

   

Set up Talend Administration Center

1. Go to the TAC web interface.

2. Login with security@company.com and your password.
   

    

3. Create a new user. Complete your Git login with your Bitbucket login, click Validate, then click Save.
   

    

4. Log out, then log back in as the user you just created.

5. Create a new project.
   

    

6. Navigate to your Bitbucket repository and find the Git URL.
   

    

7. Complete your project details. Check the connection to your Bitbucket repository, if it is OK, click Save.
   

    

8. Grant your user write privileges on the newly created project.

   

Set up Talend Nexus

1. Open the Nexus web interface.

2. Login with the user admin and password Talend123.

3. Navigate to Server Administration and configuration, click Repositories, then select Create repository.
   

    

4. Create a new repository, choose maven2 (hosted) and configure it, as shown below. Use this repository to deploy snapshot versions of your Talend Job.
   

    

5. Create a second repository, choose maven2 (hosted) and configure it, as shown below. Use this repository to deploy release version of your Talend Job.
   

    

6. Create a third repository, choose maven2 (hosted) and configure it, as shown below. This repository is used by Talend CI Builder and is defined later in the maven_user_settings.xml file.
   

    

7. When the installation is complete, open Studio.

Installing Talend Studio

Install Studio and connect to TAC

1. On another machine, install Talend Studio using the installer.

2. When the installation is complete, open Studio.

3. Click Manage Connection, click the green plus [+] sign to create a Remote TAC connection. Enter the User name, User Password, and the Web-app Url. Click Check url to make sure the connection is working.
   

    

4. Select the connection you created. Click Finish.
   

Create a sample Job

Create a simple Job that reads two flat files and uses a tMap component to join the files.

1. Add two tFileInputDelimited components to the Designer.

   

    

2. Drag and drop a tMap component and connect your input files to the tMap. Use one as a source table and the other as a lookup table.
   

    

3. Use an ID common for both files and join them using the tMap interface.
   

    

4. Drag and drop a tFileOutputDelimited component and connect it at the output of the tMap component to store the results of the join into a file.
   

    

5. Run your Job and make sure that it runs successfully.

Create test cases

1. Right-click the tMap component within the Job, then select Create Test Case.
   

    

2. Give it a name, then click Finish.
   

    

   A test case Job has been generated automatically for you.
   

    

3. As the note in Step 2 indicates, you need sample data to complete the step. Create another Job with the files you used in the tFileInputDelimited components, but only extract 200 rows by using a tSampleRow component.
   

    

4. Create another Job with the sample files as input, connect the same tMap component as before, and connect a tOutputfiledelimited component at the output to get a sample output for your test cases.
   

    

5. After the sample files are generated, double-click the testcase. Select the TestCase tab and open the Default test case.
   

    

6. For each input_file and reference_file, click File Browse and select each previously generated sample file.
   

    

7. Right-click the Default test, select Run Instance, and make sure that your test runs successfully.
   

   Your test cases are now set up, and you can automate them within the Jenkins pipeline.

    

8. Before going to the next steps, make sure you have saved your Job and pushed the modification to your Bitbucket repository.
   

Configuring the CI server

Install Maven

1. Login to the Red Hat machine.

2. From the Apache Maven Project web page, download Maven.

3. Extract the ZIP archive to the directory where you want to install Maven.

4. Open a terminal and add the M2_HOME environment variable by entering the following command:

   export M2_HOME=/opt/apache-maven/apache-maven-3.0.x

5. Add the M2 environment variable by entering the following command:

   export M2=$M2_HOME/bin

6. Add the M2 environment variable to your path by entering the following command:

   export PATH=$M2:$PATH

7. Make sure that JAVA_HOME is set to the location of your JDK.

8. Verify that Maven is installed successfully on your machine, by running the command:

   mvn --version

Install Git

1. Ensure that your system is up-to-date with the latest version of packages by running the YUM package manager update command, as shown below:

    yum update

2. Install Git by running the command:

    yum install git 

3. Verify that Git is installed successfully on your machine, by running the command:

   git --version

Install and configure Talend CommandLine

1. Ensure that the dist file is in the same folder as the Talend-Tools-Installer-YYYYYYYY_YYYY-VA.B.C-linux64-installer.run file.

2. Make the Talend-Tools-Installer-YYYYYYYY_YYYY-VA.B.C-linux64-installer.run file executable, using the following command. If you want to install Talend server modules as services, execute this command with the super-user rights.

   chmod +x Talend-Tools-Installer-YYYYYYYY_YYYY-VA.B.C-linux64-installer.run

3. Launch Talend Installer by running the command:

   ./Talend-Tools-Installer-YYYYYYYY_YYYY-VA.B.C-linux64-installer.run

4. Accept the License Agreement, and choose the directory where you want your Talend product to be installed.

5. Choose Advanced Install from the installation style list, and Custom from the installation type list.

6. Add your license file and launch the installation.

7. Install only Talend CommandLine with default port 8002.

8. Start Talend CommandLine at least once to initialize its default Maven repository, then close it.

9. Edit the commandlinePath/configuration/maven_user_settings.xml file and add the connection information to the Nexus repositories. In your case, Nexus is on a remote server where you installed the Talend platform, so replace localhost with the private EC2 IP address.

   <?xml version="1.0" encoding="UTF-8"?>
   <settings xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd" xmlns="http://maven.apache.org/SETTINGS/1.1.0"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
     <localRepository>/<.m2Path>/repository</localRepository>
   
   <servers>
       <!-- credentials to access the default releases/snapshots repositories -->
       <server>
           <id>releases</id>
           <username>admin</username>
           <password>Talend123</password>
       </server>
       <server>
           <id>snapshots</id>
           <username>admin</username>
           <password>Talend123</password>
       </server>
       <!-- credentials to access the repositories holding external jars -->  
       <server>
           <id>talend-custom-libs-release</id>
           <username>admin</username>
           <password>Talend123</password>
       </server>
       <server>
           <id>talend-custom-libs-snapshot</id>
           <username>admin</username>
           <password>Talend123</password>
       </server>
       
       <!-- credentials to access the repositories holding maven plugins -->
       <server> <!-- central (as proxy) -->
           <id>central</id>
           <username>admin</username>
           <password>Talend123</password>
       </server>
       <server>
           <id>thirdparty</id>
           <username>admin</username>
           <password>Talend123</password>
       </server>
       
   </servers>
   
   <mirrors/>
   <proxies/> <!-- http proxies, not maven proxy repositories -->
   
   <profiles>
       <profile>
           <id>talend-ci</id>
           <repositories>
               <repository>
                   <id>central</id>
                   <name>central</name>
                   <url>http://localhost:8081/repository/maven-central/</url>
                   <layout>default</layout>
               </repository>
               <repository>
                   <id>talend-custom-libs-release</id>
                   <name>talend-custom-libs-release</name>
                   <url>http://localhost:8081/repository/talend-custom-libs-release</url>
                   <layout>default</layout>
                   <releases>
                       <enabled>true</enabled>
                   </releases>
                   <snapshots>
                       <enabled>false</enabled>
                   </snapshots>
               </repository>
               <repository>
                   <id>talend-custom-libs-snapshot</id>
                   <name>talend-custom-libs-snapshot</name>
                   <url>http://localhost:8081/repository/talend-custom-libs-snapshot</url>
                   <layout>default</layout>
                   <releases>
                       <enabled>false</enabled>
                   </releases>
                   <snapshots>
                       <enabled>true</enabled>
                   </snapshots>
               </repository>
           </repositories>
           
           <pluginRepositories>            
               <pluginRepository>
                   <id>central</id>
                   <name>central</name>
                   <url>http://localhost:8081/repository/maven-central/</url>
                   <layout>default</layout>
               </pluginRepository>
               <pluginRepository>
                   <id>thirdparty</id>
                   <name>thirdparty</name>
                   <url>http://localhost:8081/repository/thirdparty</url>
                   <layout>default</layout>
               </pluginRepository>
           </pluginRepositories>
       </profile>
   </profiles>
   
   <activeProfiles>
       <activeProfile>talend-ci</activeProfile>
   </activeProfiles>
   
   </settings>

Install CI Builder

1. Extract the Talend-CI-Builder-V7.0.1.zip archive file in the directory of your choice.

2. Browse to the installation directory and execute the following command:

   mvn install:install-file -Dfile=ci.builder-7.0.1.jar -DpomFile=ci.builder-7.0.1.pom

3. Browse to the CI Builder installation directory and execute the following command to deploy the new repository on Nexus:

   mvn deploy:deploy-file -Dfile=ci.builder-7.0.1.jar -DpomFile=ci.builder-7.0.1.pom -DrepositoryId=thirdparty -Durl=http://127.0.0.1:8081/repository/talend-custom-libs-release/ -s <commandlinePath>/configuration/maven_user_settings.xml

   where the -Durl parameter value corresponds to your repository URL on Nexus, and the -s parameter value corresponds to the path to your maven_user_settings.xml file.

4. Log back in to the Nexus web UI, click Browse, then navigate to the thirdparty repository.
   

    

5. This Maven plugin is now available for anyone and can be incorporated in your builds.

Install Jenkins

1. From the Jenkins web page, download Jenkins.

2. Download the appropriate version for your environment.
   

    

3. Before starting the installation, verify that you have installed a Java JDK, and set your JAVA_HOME and PATH, then follow the Jenkins documentation on Installing Jenkins on Red Hat distributions.

Install Jenkins plugins

1. After installing Jenkins with the default setup, log in and install additional plugins.
   

    

2. Click Manage Jenkins then select Manage Plugins.
   

    

3. Click the Available tab.
   

    

4. Select and install the following plugins: Bitbucket, GitLab, Pipeline, Build Pipeline, Green Balls, Publish Over SSH, SSH, and Workspace Cleanup.

Configuring Jenkins

1. Navigate to the Jenkins main page.

2. Click Manage Jenkins, then Global Tool Configuration.
   

    

3. Click JDK > JDK installations, fill in Name and JAVA_HOME, as it is set up on your CI server, then Save.
   

    

4. Click Git, fill in the Name and Path to Git executable where Git is installed on your CI server, then Save.
   

    

5. Click Maven installations, fill in the Name and MAVEN_HOME of your CI server, then Save.
   

    

6. You are all set to start building a Jenkins CI Pipeline. You will specify your Bitbucket credential within your Jenkins Jobs.

Create a Jenkins CI Pipeline

Add a Bitbucket webhook

1. Login to Bitbucket.

2. Select your repository.

3. Click Settings.
   

    

4. From WORKFLOW select Webhooks.
   

    

5. Click Add webhook, specify a title, and build the URL as follows: http://PUBLIC IP of JENKINS:8080/bitbucket-hook/. Configure the remaining options as shown below, then Save.

   

    

6. To the right of your webhook, click View requests.

   

    

7. Go back into Studio and make a small change to your Job, for example, change the name of a component, then push the change to Git, come back to the Bitbucket webhooks request, and make sure that the webhook is working.

   

    

Compile

1. Go to Jenkins main page and select New Item.
   

    

2. Enter the Job name: 01_BitBucket_Compile, select Maven Project, click OK.
   

    

3. Scroll down to Source Code Management, select Git, enter your Repository URL, add your credentials, and specify the branch, for example, */master.
   

    

4. Scroll down to Build Trigger, then select Build when a change is pushed to BitBucket.
   

    

5. Scroll down to Build Environment, then select Delete workspaces before starts.
   

    

6. Scroll down to Build, and complete the first section as shown below: add the Root POM of your Talend project, define your Maven Goals and options, and point to your cmdline instance for MAVEN_OPTS.
   

    

7. Complete the section as shown below: locate your maven_user_settings.xml file.
   

    

8. Click Save. Your first Jenkins Job is complete.
   

    

9. To run it manually, click the green arrow with a clock, on the right side of the screen.
   

    

10. Click the console output to see the results, and you should end up with the status: Finished: SUCCESS.
    

     

11. As you did earlier, make a small change to your Job from the Studio, push the modification to Git, and make sure that the compile of the Jenkins Job has been triggered.

    

     

Test

1. Create a new Maven project, name it 02_Test, scroll down to Build Triggers, configure the project as shown below:
   

    

2. Set up the Build Environment:
   

    

3. Configure Build as shown below, then click Save:
   

Package

1. Create a new Maven project, name it 03_Package, scroll down to Build Triggers, and configure the project as shown below:
   

    

2. Set up the Build Environment:
   

    

3. Configure Build as shown below, then click Save:

   

Install

1. Create a new Maven project, name it 04_Install, scroll down to Build Triggers, and configure the project as shown below:
   

    

2. Set up the Build Environment:
   

    

3. Configure Build as shown below, then click Save:
   

    

4. Create a new Maven project, name it 05_Deploy, scroll down to Build Triggers, and configure the project as shown below:

   

5. Set up the Build Environment:
   

6. Configure Build as shown below, then click Save:
   

   MAVEN_OPTS:

   -Dproduct.path=/opt/cmdline
   -Dgeneration.type=local
   -DaltDeploymentRepository=snapshots::default::http://YOUR NEXUS PUBLIC IP:8081/repository/ben_snapshots/

7. Create a new Pipeline view from the Jenkins main page by clicking the plus [+] sign next to the tabs.
   

    

8. Give it a name, Pipeline View, select Build Pipeline View, then OK.
   

    

9. Specify the Pipeline Flow, as shown below. Keep the rest of the default settings. Click OK.

   

10. Run the first job manually, or make a small change to your Talend Job, and push the update to Bitbucket. Click the Pipeline view you just created.
    

11. Make sure that your pipeline ran successfully, and check your repository to confirm that your Talend Job has deployed.
    

     

    

Overview

When auditing a project using a remote CommandLine that doesn't have access to the H2 audit database, a NullPointerException will be returned.

Symptoms/Description

Below is the exception returned while fetching an audit report for a project where a remote CommandLine is used and where the audit database location is not on a shared location:

java.lang.RuntimeException: java.lang.NullPointerException
   at org.talend.commandline.command.ExtensionCommandSwitch.caseExtensionServerCommand(ExtensionCommandSwitch.java:148)
   at org.talend.commandline.client.command.extension.AbstractExtensionCommandSwitch.doSwitch(AbstractExtensionCommandSwitch.java:31)
   at org.talend.commandline.command.CommandProcessorSwitch.caseExtensionCommand(CommandProcessorSwitch.java:161)
   at org.talend.commandline.client.util.CommandAbstractSwitch.doSwitch(CommandAbstractSwitch.java:70)
   at org.talend.commandline.command.CommandConsumer.executeCommand(CommandConsumer.java:57)
   at org.talend.commandline.command.CommandConsumer.execute(CommandConsumer.java:35)
   at org.talend.commandline.mode.ServerCommandLine$CommmandConsumerRunnable.run(ServerCommandLine.java:139)
   at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException
   at org.talend.commandline.audit.datasource.HibernateUtils.checkAuditDB(HibernateUtils.java:141)
   at org.talend.commandline.audit.datasource.HibernateUtils.init(HibernateUtils.java:106)
   at org.talend.commandline.audit.TalendAuditSelectMode2.createHibernateSessionFactory(TalendAuditSelectMode2.java:305)
   at org.talend.commandline.audit.command.PopulateAuditExecuteCommand.execute(PopulateAuditExecuteCommand.java:130)
   at org.talend.commandline.command.ExtensionCommandSwitch.caseExtensionServerCommand(ExtensionCommandSwitch.java:146)
   ... 7 more

Resolution

With a remote CommandLine, ensure that the H2 audit database location is shared so that both tomcat and commandline can access it.

This error can also be reported if a CommandLine local to TAC is used without enough permissions on the audit database directory.

Ensure that the CommandLine user has sufficient permissions on H2 audit database directory and is able to read and write files.

The exact cause of the error can be found in the .log file of the CommandLine directory available at CMD_LINE/commandline-workspace/.metadata

Documentation suggests using the same user to start the CommandLine and Tomcat: to ensure correct management, make sure to launch Tomcat using the same administrator account as for the CommandLine.

For example: Create an account TISAdmin for both Tomcat and CommandLine.