Talend: Continuous Integration and Continuous Deployment of data services, routes, and microservices

[TalendSolutionExpert]

Talend 7.1.1 supports Continuous Integration and Continuous Deployment (CI/CD) of data services and routes, and you can use CI to build Microservices or Talend Runtime (OSGi) artifacts. If you implemented CI/CD for standard DI/BD Jobs, you can use the same steps for ESB Data Services and Routes.

This article shows you how to automatically build, test, package, and deploy data services, routes, and microservices using Talend CommandLine, CI Builder Maven plugin, Jenkins, and Nexus.

Sources for the project are attached to this article.

Content:

• Prerequisites
• Objectives and logical architecture
• Installing Talend CI Builder
• Configuring Jenkins for CI/CD
• Manage Plugins
• Global Tool Configuration
• Configure System
• Configure Credentials
• Managed files
• Preparing a Job, route, and microservice in Studio
• Creating a Jenkins Job and publishing artifacts to Nexus
• Creating a build pipeline and publishing artifacts to Nexus
• Create a pipeline project in Jenkins

Prerequisites

Install and configure the servers:

1. Create two VMs and install Ubuntu Linux 18.04.

2. Name the servers ciserver and adminserver respectively.

   In the /etc/hosts file, map the domain name of the server to a given IP address.

   sudo vi /etc/hosts 
   192.168.238.4 ciserver
   192.168.238.5 adminserver

3. On the ciserver, install the following:

   • OpenJDK Zulu-8
   • Jenkins
   • Talend CommandLine
   • Latest patches for Talend CommandLine and Studio

4. On the adminserver, install the following:

   • OpenJDK Zulu-8
   • Nexus
   • Talend Administration Center (Optional)
   • Talend Studio
   • Install Git and create a repository, for example, CI_ESB_DEMOS.

Note:

• Steps in this article can be executed on any Linux OS.

• For information on installing Talend Software modules on the adminserver, see Talend Data Services Platform Installation Guide for Linux on the Talend Help Center.

Objectives and logical architecture

1. Configure Jenkins for CI/CD

2. Create a Job in Jenkins that builds OSGi bundles, routes, and microservices, then publish it to Nexus

3. Create a pipeline Job that builds OSGi bundles, routes, and microservices, then publish it to Nexus

   

Installing Talend CI Builder

Note: Talend CI Builder is now embedded in Studio as a plug-in and doesn't require a manual installation.

Log in to the CI server and start the CommandLine process, doing this initializes your CommandLine local Maven repository and installs the CI Builder plug-in automatically. As you will be using local generation type mode, you can stop the CommandLine process after it initializes. Each build starts a new CommandLine process.

Configuring Jenkins for CI/CD

Jenkins needs additional tools and configurations to automate the build of Jobs and routes, to execute unit tests, and to package and deploy the artifacts to a repository. To accomplish this, you will do the following:

• Use Manage Plugins to install additional plug-ins

• Use Global Tool Configuration to install or configure required tools, for example, Maven and Git

• Configure System parameters, for example, PATH

• Configure Credentials, for example, Git

• Use Managed files to configure Maven settings to deploy Jobs

  

Manage Plugins

1. In the Manage Jenkins view, select Manage Plugins.

2. Select the Available tab.

3. Select and install the required plug-ins, for example:

   • Maven Integration
   • Config File Provider
   • Green Balls
   • Bitbucket (Note: If you choose a different Git provider then the Bitbucket plug-in is optional.)
   • Multiple SCMs
   • Pipeline
   • Pipeline Maven Integration
   • Blue Ocean
     
     

   

Global Tool Configuration

1. In the Manage Jenkins view, select Global Tool Configuration to configure the tools required for your Maven build.

2. Click Add JDK and provide the location of your JAVA_HOME.

   

3. Click Add Git and add JGit as a client.

   

4. Click Add Maven, provide the path to the existing MAVEN_HOME or install it automatically.

   

Configure System

1. In the Manage Jenkins view, select Configure System and configure your system variables.

2. Define a global property PATH+extra for locating Java executable.

   • On the Global properties section, click Environmental variables.
   • Click Add and fill in the Name and Value properties.
     
     

   

Configure Credentials

Use the Jenkins Credentials Provider to save your Git repository credentials.

1. Navigate to Jenkins > Credentials > System > Global credentials.

   

2. Select Add Credentials, fill in your Git Username, Password, ID, and a Description. Click OK.

   

Managed files

The Config File Provider plug-in provides the ability to add configuration files loaded through the Jenkins UI that will be copied to the Job's workspace.

Here you need to create and configure a Maven settings file that contains the path to your local Maven repository and remote repository definitions with credentials.

Note: Repository settings are configured at two levels, that is, user local settings or global Maven settings.

1. In the Manage Jenkins view, select Managed files.

   

2. Click Add a new Config, select Maven settings.xml, and provide a valid ID for the settings file. Click Submit.

   

3. Configure the local and remote repositories as shown below, then click Submit.

   Note: The ID and Name of the settings file is referenced in your Jenkins Job during the Build phase. Use the settings file, ci_maven_settings.xml, attached to this article.

   

Preparing a Job, route, and microservice in Studio

1. Create a new TAC project.

   

2. Grant write access to authorized users.

   

3. Log in to Talend Studio.

4. Import the ESB Demo Jobs by selecting Import on the Studio toolbar.

5. Open the DemoREST Job service. Click the Job tab and observe that Deployment > Build Type is set to OSGi Bundle.

   

6. Open the TestDemoRest Job. Click the Test Cases tab, then right-click Customer2 to run the test, and verify the test results.

   

7. Open the DemoResourceRoute route. Click the Job tab and observe that the Deployment > Build Type is set to Route.

   

8. Create a microservice by right-clicking DemoRESTRoute, then click Duplicate, rename the route to MS_DemoRESTRoute.

   You can run a route as microservice by changing the Deployment > Built Type.

9. Click the Job tab and change the Deployment > Build Type to RouteMicroservice.

   

10. Test the Job, route, and microservice in Studio.

Creating a Jenkins Job and publishing artifacts to Nexus

In the previous section, you prepared a data service, route, and microservice in Studio.

Before you check your changes into SCM, run the Jobs and routes in Studio, and verify that all the tests are working, then push the changes to the Git repository.

In this section, you create a Jenkins Job and verify the build and test execution results on a CI server.

1. Click New Item.

2. In the Enter an item name text box, name the Job, ESB_DeploymentType. Select Maven project, then click OK.

   

3. On the General tab, enter the Job Description.

   

4. Click Source Code Management, choose Git or Multiple SCMs, provide the Git repository URL and branch specifier.

   

5. For Git credentials, select the credentials with id bitbucket, by using the down arrow to the right of Credentials.

   

6. Click Build Triggers and configure when Jenkins should automatically run the Job.

   Note: An alternate option is to build every two hours based on the size and complexity of the project.

   

7. Click Build Environment, then select the Delete workspace before build starts check box.

   

   

8. Click Build:

   1. For root pom, fill in the relative URL of your pom.xml project:

      CI_ESB_DEMOS/poms/pom.xml

      Note: The project name is in uppercase.

   2. For Goals and options, fill in the Maven goal and additional CLI options:

      deploy -X -pl jobs/process/demorest_0.1,jobs/routes/demoresourceroute_0.1,jobs/routes/ms_demorestroute_0.1 -fae -amd

      Note: modify the options as per your convenience, for example:

      1. Use the -ff option to stop the build at the first failure.

      2. Remove the -pl option for a full build.

      Maven command line options:

      -pl: Comma-delimited list of projects to build, or remove the -pl option for complete build

      -fae: Only fail the build at the end, allow all non-impacted builds to continue

      -ff: Stop at first failure in reactorized builds

      -amd: Using -pl if project list is specified, then also build projects required by the list

      -X: Debug log output

   3. For Maven_OPTS, fill in the location of your local CommandLine, and remote Nexus repository URLs.

      -Dgeneration.type=local
      -Dproduct.path=/opt/talend/Talend-7.1.1/cmdLine/cmdline/studio
      -Xmx3076m
      -DaltDeploymentRepository=releases-ci::default::http://adminserver:8081/repository/releases-ci/
      -Dproject.distributionManagement.repository.url=http://adminserver:8081/repository/releases-ci/
      -Dproject.distributionManagement.repository.id=releases-ci

      

9. Click Build Settings, optionally for unstable builds, and configure E-mail Notification.

   

10. Save the changes, by clicking Apply, then click Save, to save the Job.

11. Run the Job and verify the results.

    1. Review the test results:

       

    2. Verify that the artifacts are published to Nexus, by logging in to the Nexus console at, http://adminserver:8081/

       

Creating a build pipeline and publishing artifacts to Nexus

The Pipeline plug-in allows you to express your build process as a script and implement complex workflows. The Pipeline Maven plug-in provides Maven integration by using the withMaven step, which configures a Maven environment to use within a pipeline Job.

The Blue Ocean plug-in provides sophisticated visualizations of the build and pipeline status with dashboards.

Create a pipeline project in Jenkins

1. Click New Item, enter the name of the pipeline, for example, Pipeline_ESB_Deployments, then click Pipeline and click OK.

   

2. In the Description field, provide the purpose of this pipeline.

   

3. Click Build Triggers, select Poll SCM, for example, every night.

   

4. Click Pipeline, then select pipeline script from the Definition drop-down menu. Copy the contents of the Pipeline_Nexus_deploy_groovy.txt file, attached to this article, into the script area.

   

5. Click Build now, and run the pipeline.

6. Click Full Stage View to see the pipeline stages.

   

7. Select the Open Blue Ocean pipeline and visualize the dashboards.

   

8. Select Pipeline_ESB_Deployments, then click Run.

   

9. Observe the pipeline status is blue while the Job is running.

   

10. After successful execution, the pipeline status turns to green.