This course concentrates on Big Data Spark Jobs. It is mainly focused on Big Data Streaming Jobs but also introduces you Big Data Batch Jobs.

After an introduction to Apache Kafka and Apache Spark, you work on a log processing use case, a common Big Data use case. You see how to publish messages to Kafka, subscribe to receive messages, insert data into ElasticSearch and use Kibana to create charts and dashboards. You also see how to save data to and read data from HBase tables.

• Target Audience: Anyone who wants to use Talend Studio to interact with big data systems.
• Prerequisites: Completion of Talend Big Data Basics.
• Badge: Complete this learning path on your way to earning the Talend Big Data Developer Practitioner badge. To know more about the criteria to earn this badge, refer to the Talend Academy Badging Program page.
• Availability: This learning plan is available as part of a Talend Academy Learning Subscription.

If you are already a Talend Academy subscriber or want to access the publicly available content on the platform, go to the Talend Academy Welcome page to log in or create an account.

Overview

This article demonstrates how to integrate Zeppelin Notebooks in a Talend DI Job by leveraging the Zeppelin RESTful API.

Use the two notebooks prepared for you, in the Archive.zip file that is attached to this article. The first notebook trains, using historical data stored on HDFS, a Machine Learning model and saves it to HDFS. The second notebook loads the trained Machine Learning model and uses it to score the new data. Then the article explains how to develop two Talend DI Jobs to integrate those notebooks. The first Job ingests historical data to HDFS, and then triggers its execution. The second Job uploads the new data from S3 to HDFS, runs the notebook that scores the new data, and saves the results to HDFS.

Assumptions

1. Amazon Web Services (AWS):

   • You should be familiar with the AWS platform.

2. Talend:

   • You should have a basic knowledge of Talend Studio.
3. Restful API:
   • You should have a basic understanding of Restful API.

Prerequisite

• AWS account

• EMR 5.11.1 with Hadoop, Zeppelin, Livy, Hive, Hue, and Spark

• AWS S3 bucket

• IAM Role to access S3 bucket (Access Key / Secret Key)

• EC2 machine with Talend Studio 7.0.1 and above

• Source materials: Archive.zip file attached to this article

Apache Zeppelin

Getting started with Zeppelin on EMR

1. Connect to your AWS Management Console, click Services, then search for EMR and select it.
   

    

2. Click Create cluster, on the following screen, select Go to advanced options.
   

    

3. On the Software and Steps page, select emr-5.11.1 from the Release pull-down menu, then select Hadoop, Zeppelin, Livy, Hive, Hue, and Spark from the Software Configuration list. Click Next.
   

    

4. On the Hardware configuration page, keep the default setting or if needed choose a specific Network and Subnet, then click Next.

5. On the General Cluster Settings page, under General Options, provide a Cluster name for your cluster. Click Next.
   

    

6. On the Security page, under the Security Options settings, select an EC2 key pair from the drop-down menu, or create one, then click Create cluster.
   

    

7. Wait a few minutes for your cluster to be up and running.
   

    

8. Edit the network security rules by navigating to the EC2 dashboard.
   

    

9. Select the EC2 machine of your master node, then under the Description tab, click Security groups.
   

    

10. Select the Inbound tab and click Edit.
    

     

11. Create a new rule to allow all traffic from the EC2 where Talend will be installed, make sure to use the Security Group ID.
    

     

12. Create a new rule to allow all traffic from your local machine to the cluster.
    

     

13. Go back to your EMR cluster home page and select your cluster. On the Summary tab, notice that you can now access your cluster web UI connections from your local machine.
    

     

14. Click the Hue link, for the first connection, create the Hue superuser, using admin as username, and a password of your choice. Remain on this screen.

15. Your EMR cluster is all set.

     

Training a Machine Learning model

1. Before getting into Zeppelin, you need to upload the training data. Select Files, under Browser, from the menu on the left, and navigate to /user/admin.
   

    

2. Click the New button, and select Directory. Name the directory, Input_Data, then click Create.
   

    

3. Navigate to Input_Data, click Upload > Files, and browse to the TrainingData.csv file and select it.
   

   
   

    

4. Return to the EMR Management Console, and click the Zeppelin link.
   

    

5. From the Welcome to Zeppelin page, select Create new note to open the properties box.

   1. In the Note Name field, name the note Model_Training.

   2. Select spark from the Default Interpreter pull-down menu.

   3. Click Create Note.

   

    

6. In the first paragraph, import the TrainingData.csv file and create a Spark DataFrame, by copying and pasting the following code into the note:

   val df_data = spark.
   read.format("org.apache.spark.sql.execution.datasources.csv.CSVFileFormat").
       option("header", "true").
       option("inferSchema", "true").
       load("/user/admin/Input_Data/TrainingData.csv")
   

    

7. Run the paragraph by clicking the Play button on the right-hand side of the paragraph. You can observe the results output below your code.
   

    

8. In the second paragraph, split your data in to three parts, training_data, test_data, and prediction_data, using the code below, then run the paragraph.

   val splits = df_data.randomSplit(Array(0.8, 0.18, 0.02), seed = 24L)
   val training_data = splits(0).cache()
   val test_data = splits(1)
   val prediction_data = splits(2)
   
   
   

    

9. In the third paragraph, import machine necessary libraries, create features using indexers, and assemble your features in a vector, using the code below, then run the paragraph.

   import org.apache.spark.ml.classification.RandomForestClassifier
   import org.apache.spark.ml.feature.{OneHotEncoder, StringIndexer, IndexToString, VectorAssembler}
   import org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
   import org.apache.spark.ml.{Model, Pipeline, PipelineStage, PipelineModel}
   import org.apache.spark.sql.SparkSession
   
   val stringIndexer_label = new StringIndexer().setInputCol("PRODUCT_LINE").setOutputCol("label").fit(df_data)
   val stringIndexer_prof = new StringIndexer().setInputCol("PROFESSION").setOutputCol("PROFESSION_IX")
   val stringIndexer_gend = new StringIndexer().setInputCol("GENDER").setOutputCol("GENDER_IX")
   val stringIndexer_mar = new StringIndexer().setInputCol("MARITAL_STATUS").setOutputCol("MARITAL_STATUS_IX")
   
   val vectorAssembler_features = new VectorAssembler().setInputCols(Array("GENDER_IX", "AGE", "MARITAL_STATUS_IX", "PROFESSION_IX")).setOutputCol("features")
   
   
   

    

10. In the last paragraph, select your model, create the pipeline, train it, and save it to disk, using the code below, then run the paragraph. For permission reasons, the model is saved under Hadoop.

    val rf = new RandomForestClassifier().setLabelCol("label").setFeaturesCol("features").setNumTrees(5)
    val labelConverter = new IndexToString().setInputCol("prediction").setOutputCol("predictedLabel").setLabels(stringIndexer_label.labels)
    val pipeline_rf = new Pipeline().setStages(Array(stringIndexer_label, stringIndexer_prof, stringIndexer_gend, stringIndexer_mar, vectorAssembler_features, rf, labelConverter))
    val model_rf = pipeline_rf.fit(training_data)
    model_rf.write.overwrite().save("/user/hadoop/Model/MyModel")
    
    
    

     

11. If you want to check the predicted results on the test data split, add the following code to the next paragraph and run it. Don't forget to remove this paragraph after testing.

    val prediction_test= model_rf.transform(test_data)
    prediction_test.show(10)
    
    
    

     

12. Your training model notebook is all set. Go to Hue and remove the training data because you will upload it from S3 to the cluster using a Talend Job.

    

     

Scoring a Machine Learning model

1. Create a new note, by clicking Notebook and selecting Create new note.
   

    

2. Name the note Model_Scoring, then select spark from the Default Interpreter pull-down menu. Click Create Note.
   

    

3. From Hue, upload the NewData.csv file into the /user/admin/Input_Data/ folder.
   

    

4. Go back to the notebook, and read the NewData.csv file from HDFS, store the data in a Spark DataFrame, and filter out the product line, by copying and pasting the following code into the note, then run the paragraph.

   val df_data = spark.
      read.format("org.apache.spark.sql.execution.datasources.csv.CSVFileFormat").
       option("header", "true").
       option("inferSchema", "true").
       load("/user/admin/Input_Data/NewData.csv")
   
   val df_newdata = df_data.select("GENDER","AGE","MARITAL_STATUS","PROFESSION")
   
   

    

5. Import the necessary libraries, and load the trained model, using the code below, then run the paragraph.

   import org.apache.spark.ml.classification.RandomForestClassifier
   import org.apache.spark.ml.feature.{OneHotEncoder, StringIndexer, IndexToString, VectorAssembler}
   import org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
   import org.apache.spark.ml.{Model, Pipeline, PipelineStage, PipelineModel}
   import org.apache.spark.sql.SparkSession
   
   val model_rf = PipelineModel.read.load("/user/hadoop/Model/MyModel")
   
   
   

    

6. Score your new data using the trained model, and export result to HDFS, using the code below, then run the paragraph.

   val newprediction = model_rf.transform(df_newdata)
   val output_pred = newprediction.select("GENDER","AGE","MARITAL_STATUS","PROFESSION","predictedLabel")
   output_pred.coalesce(1).write.mode("overwrite").csv("/user/hadoop/Output_Data/output.csv")
   
   
   

    

7. Using File Browser, navigate to /user/Hadoop/Output_Data/output.csv and check the results.
   

    

8. Your model scoring notebook is all set, go to Hue and remove the NewData.csv because you will upload it from S3 to the cluster using a Talend Job.

    

Talend Studio

Getting started

1. Before installing Studio make sure that your EC2 can access the EMR cluster, by going to the inbound network rules of your cluster and allowing all traffic from the Studio security group.

2. To install Studio on an EC2 machine, see the Installing Talend Studio with the Talend Studio Installer page on the Talend Help Center.

3. When the installation is complete, open Studio, and create a new local project.

4. Provision an S3 bucket and upload the TrainingData.csv and NewData.csv files.

Creating a Machine Learning training Job

1. Expand Job Designs, then right-click Standard, and select Create Standard Job.
   

    

2. Name your Job ML_training and click Finish.
   

    

3. On the Repository view, expand Metadata, and right-click Create Hadoop Cluster.

   1. Give your cluster a name, then click Next.

   2. Select the distribution, Amazon EMR, and version, EMR 5.8.0, of your Hadoop cluster.

   3. Select Enter manually Hadoop services.

   4. Click Finish.
      

       

4. Enter the connection information manually, using the admin username for authentication. Click Next.
   

    

5. Click Check Services to ensure that Studio can successfully connect to the cluster.
   

    

6. In the designer, add a tPrejob component, a tS3Connection component, and a tHDFSConnection component.
   

    

7. Connect the tPrejob to the tS3Connection using an OnComponentOK trigger, then connect the tS3Connection to tHDFSConnection using an OnSubjobOK. trigger.
   

    

8. Double-click the tS3Connection component, on the Basic settings tab, fill in the Access Key and Secret Key.
   

    

9. Double-click the tHDFSConnection component.

   1. On the Basic settings tab, select Repository as the Property Type.

   2. To the right of Repository, click the […] button, and navigate to Metadata.

   3. Select EMR_HDFS.

   4. Click OK.

   

    

10. Add a tS3Get, tHDFSPut, tREST and a tLogRow component to the canvas.
    

     

11. Using OnSubjobOK triggers, connect them as shown below:
    

     

12. Double-click the tS3Get component, on the Basic settings tab, fill in the Bucket, Key, and File fields with the appropriate settings.
    

     

13. Double-click the tHDFSPut component, on the Basic settings tab, configure the settings as shown in the screenshot below:
    

     

14. Before setting up the tRest component, retrieve the note id using the List of the notes API from the Apache Zeppelin web site.

15. Open up a new tab on your web browser, and using the EMR IP and Zeppelin port of your instance, enter the URL as follows: http://zeppelin-server:zeppelin-port/api/notebook.
    

     

16. Double-click the tRest component, to use the method to run a note, enter the URL, for example, http://zeppelin-server:zeppelin-port/api/notebook/job/NOTE_ID, then select POST for the HTTP Method.
    

     

17. Edit and define the schema, as shown in the screenshot below:
    

     

18. Configure the HTTP Headers, by setting the name to "Content-type", and the value to "application/json".
    

     

19. Run the Job and confirm results.
    

     

Creating a Machine Learning scoring Job

1. Create a new Standard Job, and name it ML_Scoring. Copy and paste the tPreJob, tS3connection, and tHDFSConnection components from the previous Job into this Job.
   

    

2. Add a tS3Get, tHDFSPut, tREST, and a tLogRow component to the canvas and connect them as shown below:
   

    

3. Double-click the tS3Get component, and configure the Basic settings as shown below:
   

    

4. Double-click the tHDFSPut component, and configure the Basic settings as shown below:
   

    

5. Before setting up the tREST component, call the Zeppelin API, http://zeppelin-server:zeppelin-port/api/notebook, to list the notes and find the id related to the Model_Scoring.
   

    

6. Double-click the tRest component, to use the method to run a note, enter the URL, for example, http://zeppelin-server:zeppelin-port/api/notebook/job/NOTE_ID, then select POST for the HTTP Method.

   

    

7. Configure the HTTP Headers, by setting the name to "Content-type", and the value to "application/json".

   

    

8. Run the Job and confirm results.
   

    

9. Confirm your results in Hue, by navigating to /user/Hadoop/Ouput_Data/output.csv.
   

    

Conclusion

You’ve learned how to integrate Zeppelin Notebooks with Talend, and you created a data science pipeline with Talend Jobs to train a Machine Learning model and to score new incoming data.

Introduction

Talend Jobs that are developed using context variable and dynamic SQL queries are not supported; therefore, Talend Data Catalog (TDC) is unable to harvest metadata and trace data lineage from a Talend dynamic integration Job using the Talend ETL bridge.

This article shows you how to work around these limitations in Talend Jobs that use resources from a Cloudera Cluster using Cloudera Navigator and Talend Data Catalog bridge.

Sources for the project are attached to this article.

Prerequisites

• Cloudera Cluster CDH 5.10 and above

• Cloudera Navigator 2.15.1 and above

• MySQL server to store metadata table of the dynamic integration framework

• Talend Big Data Platform 7.1.1 and above

• Talend Data Catalog 7.1 Advanced (or Plus) Edition and above, with latest cumulative patches

Setting up Talend Studio

1. Open Talend Studio and create a new project.

2. In the Repository, expand Metadata, right-click Hadoop Cluster, then select Create Hadoop Cluster.

3. Using the Hadoop Cluster Connection wizard, create a connection to your Cloudera Cluster, and make sure that you select the Use cloudera navigator check box.
   

    

4. Click the ellipsis to the right of Use Cloudera Navigator, then set up your connection to Cloudera Navigator, as shown below:
   

   For more information on leveraging Cloudera Navigator in Talend Jobs, see the How to set up data lineage with Cloudera Navigator page of the Talend Big Data Studio User Guide available in the Talend Help Center.

    

Building the dynamic integration Job

This use case uses MySQL to store metadata such as table source/target, queries, and filters, then stores these values in context variables that are used to build integration Jobs at runtime. The dynamic Job reads data from source tables in Hive and writes data to target tables in Hive.

1. Upload the metadata for the dynamic integration Job to the MySQL server (or any other DB of your choice), using the Metadata_Demo_forMySql.xlsx file attached to this article.
   

    

2. Upload the source data, located in the employees.csv and salaries.csv files attached in this article, to Hive.

3. Create a standard Job, then add a tDBConnection component to connect to the metadata database. Note: The complete preparation Job, located in the prepare_load_dwh_Hive.zip file, is attached to this article.

4. Replicate all of the fields in the metadata table by creating the following Context variables:
   

    

5. Add a tDBInput component.

6. Connect the tDBConnection component to the tDBInput component using the OnSubjobOk trigger.

7. Double-click the tDBInput component to open the Basic settings view. Click the [...] button next to the Table name text box, then select the table name where you've uploaded the metadata, in this case, meta_tables, apply the appropriate schema, and use the following query:

   "SELECT 
     `meta_tables`.`Job`, 
     `meta_tables`.`business_name`, 
     `meta_tables`.`db_in`, 
     `meta_tables`.`tabel_in`, 
     `meta_tables`.`db_out`, 
     `meta_tables`.`table_out`, 
     `meta_tables`.`select_args`, 
     `meta_tables`.`query`, 
     `meta_tables`.`conditions`, 
     `meta_tables`.`db_lookup`, 
     `meta_tables`.`table_lookup`
   FROM `meta_tables`
   WHERE  `meta_tables`.`business_name`='Agg'"

   Notice that the value for bussiness_name is hardcoded with Agg. Depending on the type of dynamic query you want to run, you could use a context variable so that at runtime, the Job uses the context variable value and filters the metadata table on the business_name (in this case, Agg or Dwh).

8. Add a tMap component after the tDBInput component, then connect it using a Main row. The tMap component acts as a pass-through and creates output that contains all the input fields.
   

   
   

    

9. Connect the tMap to a tFlowToIterate component, then create a key-value pair for each of the fields in the metadata table.
   

   
   

    

10. Add a tRunJob component. Connect the tFlowToIterate component to the tRunJob component using Row > Iterate.
    

     

11. Set up the tRunJob component to transmit the whole context to the child Job for each iteration, as shown below:
    

     

Building the data integration Job

In this section, you build a Job that is triggered by the tRunJob component from the previous Job.

Note: The complete integration Job, located in the load_dwh_Hive.zip file, is attached to this article.

1. Create a new standard Job, then add a tPreJob component and a tHiveConnection component.

2. Connect tPreJob to tHiveConnection using the OnComponentOK trigger.
   

    

3. Add a tHiveRow component below the tPreJob component.
   

    

4. Configure the tHiveRow component, as shown below:
   

    

5. Use the context parameter transmitted by the parent Job by entering the following query in the Query text box.

   "INSERT OVERWRITE TABLE  "+context.BB_W_db_out+"."+context.BB_W_table_out+" "+context.BB_W_query+" "

   The integration Job (child Job) runs as many times as the number of rows returned by the metadata table filtered by the context business_name in the parent Job.

6. Run the Job.

7. Open Cloudera Navigator, then search for Hive Jobs. Locate the Talend Job and review trace the data lineage.
   

    

Configuring Talend Data Catalog

1. Open Talend Data Catalog (TDC).

2. Create a new configuration.
   

    

Cloudera Hive bridge

1. For the bridge to work, you'll need to the Cloudera JDBC connector for Hive and the JDBC driver of your Hive metastore (in this case, Postgres).

2. Ensure both of the drivers are accessible by TDC server or an Agent.

3. Create a new Physical Data Model to harvest the Hive metastore.

4. On the Properties tab, select the Cloudera Enterprise Hadoop Hive Database.
   

    

5. On the Import Setup tab, in the User and Password fields, enter your Hive metastore credentials (typically set up during the cluster installation). If you are not able to retrieve it, use the StackExchange, connect to PostgreSQL server: FATAL: no pg_hba.conf entry for host tutorial for a Hive metastore with Postgres.

6. Review your settings, when you're finished, the Import setup tab should look like this:
   

   Click Import to start the metadata import.

7. After a successful import, navigate to Data Catalog > Metadata Explorer > Metadata Browser, and locate the harvested metadata, in this case, employee_male (Table).
   

    

Cloudera Navigator bridge

1. Create a new model, then in Model Type, select Cloudera Navigator - New Beta Bridge.
   

    

2. On the Import Setup tab, fill in the Navigator URL*, Login, Password, then filter the Operations you want to harvest, in this case, hive.
   

    

3. After a successful import, you'll find metadata harvest for Navigator, the Hive (connection), and the dynamic Data Integration Job (Di Model).
   

    

Stitching

1. Make sure that your harvested models belong to your configuration, by dragging them to the configuration you set up earlier.

2. On the Manage menu, select Manage Contents, click the Nav model, then select the Connection tab on the right.
   

    

3. Select the Connection Name, in this case, Hive.

4. Click Edit, then, from the Database pull-down list, select Hive (click Edit Schemas if it wasn't done before), then click OK.
   

    

5. Click Build, then click the Diagram to see the connection between models.
   

   
   

    

6. Trace the data lineage of the Talend Job containing dynamic queries executed against Hive.
   

    

Conclusion

This article showed you how to handle the Talend Data Catalog Harvesting process of DI Jobs using context variables and dynamic queries on a Cloudera Cluster leveraging Navigator and Hive bridges to trace data lineage.

Overview

Talend Continuous Integration (CI) is now fully compliant with Maven standards, and Continuous Integration and Deployment (CI/CD) with Talend has never been easier.

This tutorial illustrates how to automate CI/CD with Talend 7 and Jenkins, by complementing current Talend documentation. For more information, see the Talend Software Development Life Cycle Best Practices Guide.

Requirements

• Talend platform Edition 7.0.1

• Jenkins

• Source code management supported by Talend

Architecture

This example uses a Continuous Integration server (Jenkins) by leveraging Talend CI Builder, Bitbucket as a service for the code repository, and the Talend Nexus Artifact Repository.

You can continue to use TAC to publish Jobs to a Nexus repository using the Publisher page, but the Publisher page is deprecated in TAC, so Talend recommends using a Maven build.

Talend CI Builder is a Maven plugin, delivered by Talend, that transforms the Talend Job sources to Java classes using the Talend CommandLine application. This allows you to execute your tests in your own company Java factory.

• The overall high-level architecture:

  
  

• Talend platform architecture and components on a Windows Server machine:

  
  

• CI server architecture and components on a Red Hat machine:

  
  

Installing the Talend platform

Install TAC and CommandLine

1. Download the Talend-Installer, the Dist file, and your License.

2. Extract and store them in the same folder.
   

    

3. Double-click the installer.

4. Choose Advanced Install, choose Custom, then browse to your License File.
   

    

5. Select Talend Administration Center and Talend Command Line. Click Next.
   

    

6. Select Talend Administration Center, Talend Command Line, and Talend Server Services. Click Next.
   

    

7. Choose Install an embedded tomcat8 server on the drop-down list. Keep the default Create TAC administrator user setting. Click Next.
   

    

8. Choose Embedded H2 database on the drop-down list and select Install Nexus server with TAC.
   

    

9. Keep the default Nexus Port.
   

    

10. Keep the default CommandLine port.
    

     

11. Select Install Talend Administration Center as a service and Install Talend Command Line as a service.
    

     

12. The Talend platform will install.
    

Set up Bitbucket as a service

1. Create a Bitbucket account or use your existing one.

2. Click the plus [+] sign.
   

    

3. From the CREATE menu, select Repository.

   

    

4. Create a new repository and give it a name. Select Git as the version control system, then click Create repository.

   

Set up Talend Administration Center

1. Go to the TAC web interface.

2. Login with security@company.com and your password.
   

    

3. Create a new user. Complete your Git login with your Bitbucket login, click Validate, then click Save.
   

    

4. Log out, then log back in as the user you just created.

5. Create a new project.
   

    

6. Navigate to your Bitbucket repository and find the Git URL.
   

    

7. Complete your project details. Check the connection to your Bitbucket repository, if it is OK, click Save.
   

    

8. Grant your user write privileges on the newly created project.

   

Set up Talend Nexus

1. Open the Nexus web interface.

2. Login with the user admin and password Talend123.

3. Navigate to Server Administration and configuration, click Repositories, then select Create repository.
   

    

4. Create a new repository, choose maven2 (hosted) and configure it, as shown below. Use this repository to deploy snapshot versions of your Talend Job.
   

    

5. Create a second repository, choose maven2 (hosted) and configure it, as shown below. Use this repository to deploy release version of your Talend Job.
   

    

6. Create a third repository, choose maven2 (hosted) and configure it, as shown below. This repository is used by Talend CI Builder and is defined later in the maven_user_settings.xml file.
   

    

7. When the installation is complete, open Studio.

Installing Talend Studio

Install Studio and connect to TAC

1. On another machine, install Talend Studio using the installer.

2. When the installation is complete, open Studio.

3. Click Manage Connection, click the green plus [+] sign to create a Remote TAC connection. Enter the User name, User Password, and the Web-app Url. Click Check url to make sure the connection is working.
   

    

4. Select the connection you created. Click Finish.
   

Create a sample Job

Create a simple Job that reads two flat files and uses a tMap component to join the files.

1. Add two tFileInputDelimited components to the Designer.

   

    

2. Drag and drop a tMap component and connect your input files to the tMap. Use one as a source table and the other as a lookup table.
   

    

3. Use an ID common for both files and join them using the tMap interface.
   

    

4. Drag and drop a tFileOutputDelimited component and connect it at the output of the tMap component to store the results of the join into a file.
   

    

5. Run your Job and make sure that it runs successfully.

Create test cases

1. Right-click the tMap component within the Job, then select Create Test Case.
   

    

2. Give it a name, then click Finish.
   

    

   A test case Job has been generated automatically for you.
   

    

3. As the note in Step 2 indicates, you need sample data to complete the step. Create another Job with the files you used in the tFileInputDelimited components, but only extract 200 rows by using a tSampleRow component.
   

    

4. Create another Job with the sample files as input, connect the same tMap component as before, and connect a tOutputfiledelimited component at the output to get a sample output for your test cases.
   

    

5. After the sample files are generated, double-click the testcase. Select the TestCase tab and open the Default test case.
   

    

6. For each input_file and reference_file, click File Browse and select each previously generated sample file.
   

    

7. Right-click the Default test, select Run Instance, and make sure that your test runs successfully.
   

   Your test cases are now set up, and you can automate them within the Jenkins pipeline.

    

8. Before going to the next steps, make sure you have saved your Job and pushed the modification to your Bitbucket repository.
   

Configuring the CI server

Install Maven

1. Login to the Red Hat machine.

2. From the Apache Maven Project web page, download Maven.

3. Extract the ZIP archive to the directory where you want to install Maven.

4. Open a terminal and add the M2_HOME environment variable by entering the following command:

   export M2_HOME=/opt/apache-maven/apache-maven-3.0.x

5. Add the M2 environment variable by entering the following command:

   export M2=$M2_HOME/bin

6. Add the M2 environment variable to your path by entering the following command:

   export PATH=$M2:$PATH

7. Make sure that JAVA_HOME is set to the location of your JDK.

8. Verify that Maven is installed successfully on your machine, by running the command:

   mvn --version

Install Git

1. Ensure that your system is up-to-date with the latest version of packages by running the YUM package manager update command, as shown below:

    yum update

2. Install Git by running the command:

    yum install git 

3. Verify that Git is installed successfully on your machine, by running the command:

   git --version

Install and configure Talend CommandLine

1. Ensure that the dist file is in the same folder as the Talend-Tools-Installer-YYYYYYYY_YYYY-VA.B.C-linux64-installer.run file.

2. Make the Talend-Tools-Installer-YYYYYYYY_YYYY-VA.B.C-linux64-installer.run file executable, using the following command. If you want to install Talend server modules as services, execute this command with the super-user rights.

   chmod +x Talend-Tools-Installer-YYYYYYYY_YYYY-VA.B.C-linux64-installer.run

3. Launch Talend Installer by running the command:

   ./Talend-Tools-Installer-YYYYYYYY_YYYY-VA.B.C-linux64-installer.run

4. Accept the License Agreement, and choose the directory where you want your Talend product to be installed.

5. Choose Advanced Install from the installation style list, and Custom from the installation type list.

6. Add your license file and launch the installation.

7. Install only Talend CommandLine with default port 8002.

8. Start Talend CommandLine at least once to initialize its default Maven repository, then close it.

9. Edit the commandlinePath/configuration/maven_user_settings.xml file and add the connection information to the Nexus repositories. In your case, Nexus is on a remote server where you installed the Talend platform, so replace localhost with the private EC2 IP address.

   <?xml version="1.0" encoding="UTF-8"?>
   <settings xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd" xmlns="http://maven.apache.org/SETTINGS/1.1.0"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
     <localRepository>/<.m2Path>/repository</localRepository>
   
   <servers>
       <!-- credentials to access the default releases/snapshots repositories -->
       <server>
           <id>releases</id>
           <username>admin</username>
           <password>Talend123</password>
       </server>
       <server>
           <id>snapshots</id>
           <username>admin</username>
           <password>Talend123</password>
       </server>
       <!-- credentials to access the repositories holding external jars -->  
       <server>
           <id>talend-custom-libs-release</id>
           <username>admin</username>
           <password>Talend123</password>
       </server>
       <server>
           <id>talend-custom-libs-snapshot</id>
           <username>admin</username>
           <password>Talend123</password>
       </server>
       
       <!-- credentials to access the repositories holding maven plugins -->
       <server> <!-- central (as proxy) -->
           <id>central</id>
           <username>admin</username>
           <password>Talend123</password>
       </server>
       <server>
           <id>thirdparty</id>
           <username>admin</username>
           <password>Talend123</password>
       </server>
       
   </servers>
   
   <mirrors/>
   <proxies/> <!-- http proxies, not maven proxy repositories -->
   
   <profiles>
       <profile>
           <id>talend-ci</id>
           <repositories>
               <repository>
                   <id>central</id>
                   <name>central</name>
                   <url>http://localhost:8081/repository/maven-central/</url>
                   <layout>default</layout>
               </repository>
               <repository>
                   <id>talend-custom-libs-release</id>
                   <name>talend-custom-libs-release</name>
                   <url>http://localhost:8081/repository/talend-custom-libs-release</url>
                   <layout>default</layout>
                   <releases>
                       <enabled>true</enabled>
                   </releases>
                   <snapshots>
                       <enabled>false</enabled>
                   </snapshots>
               </repository>
               <repository>
                   <id>talend-custom-libs-snapshot</id>
                   <name>talend-custom-libs-snapshot</name>
                   <url>http://localhost:8081/repository/talend-custom-libs-snapshot</url>
                   <layout>default</layout>
                   <releases>
                       <enabled>false</enabled>
                   </releases>
                   <snapshots>
                       <enabled>true</enabled>
                   </snapshots>
               </repository>
           </repositories>
           
           <pluginRepositories>            
               <pluginRepository>
                   <id>central</id>
                   <name>central</name>
                   <url>http://localhost:8081/repository/maven-central/</url>
                   <layout>default</layout>
               </pluginRepository>
               <pluginRepository>
                   <id>thirdparty</id>
                   <name>thirdparty</name>
                   <url>http://localhost:8081/repository/thirdparty</url>
                   <layout>default</layout>
               </pluginRepository>
           </pluginRepositories>
       </profile>
   </profiles>
   
   <activeProfiles>
       <activeProfile>talend-ci</activeProfile>
   </activeProfiles>
   
   </settings>

Install CI Builder

1. Extract the Talend-CI-Builder-V7.0.1.zip archive file in the directory of your choice.

2. Browse to the installation directory and execute the following command:

   mvn install:install-file -Dfile=ci.builder-7.0.1.jar -DpomFile=ci.builder-7.0.1.pom

3. Browse to the CI Builder installation directory and execute the following command to deploy the new repository on Nexus:

   mvn deploy:deploy-file -Dfile=ci.builder-7.0.1.jar -DpomFile=ci.builder-7.0.1.pom -DrepositoryId=thirdparty -Durl=http://127.0.0.1:8081/repository/talend-custom-libs-release/ -s <commandlinePath>/configuration/maven_user_settings.xml

   where the -Durl parameter value corresponds to your repository URL on Nexus, and the -s parameter value corresponds to the path to your maven_user_settings.xml file.

4. Log back in to the Nexus web UI, click Browse, then navigate to the thirdparty repository.
   

    

5. This Maven plugin is now available for anyone and can be incorporated in your builds.

Install Jenkins

1. From the Jenkins web page, download Jenkins.

2. Download the appropriate version for your environment.
   

    

3. Before starting the installation, verify that you have installed a Java JDK, and set your JAVA_HOME and PATH, then follow the Jenkins documentation on Installing Jenkins on Red Hat distributions.

Install Jenkins plugins

1. After installing Jenkins with the default setup, log in and install additional plugins.
   

    

2. Click Manage Jenkins then select Manage Plugins.
   

    

3. Click the Available tab.
   

    

4. Select and install the following plugins: Bitbucket, GitLab, Pipeline, Build Pipeline, Green Balls, Publish Over SSH, SSH, and Workspace Cleanup.

Configuring Jenkins

1. Navigate to the Jenkins main page.

2. Click Manage Jenkins, then Global Tool Configuration.
   

    

3. Click JDK > JDK installations, fill in Name and JAVA_HOME, as it is set up on your CI server, then Save.
   

    

4. Click Git, fill in the Name and Path to Git executable where Git is installed on your CI server, then Save.
   

    

5. Click Maven installations, fill in the Name and MAVEN_HOME of your CI server, then Save.
   

    

6. You are all set to start building a Jenkins CI Pipeline. You will specify your Bitbucket credential within your Jenkins Jobs.

Create a Jenkins CI Pipeline

Add a Bitbucket webhook

1. Login to Bitbucket.

2. Select your repository.

3. Click Settings.
   

    

4. From WORKFLOW select Webhooks.
   

    

5. Click Add webhook, specify a title, and build the URL as follows: http://PUBLIC IP of JENKINS:8080/bitbucket-hook/. Configure the remaining options as shown below, then Save.

   

    

6. To the right of your webhook, click View requests.

   

    

7. Go back into Studio and make a small change to your Job, for example, change the name of a component, then push the change to Git, come back to the Bitbucket webhooks request, and make sure that the webhook is working.

   

    

Compile

1. Go to Jenkins main page and select New Item.
   

    

2. Enter the Job name: 01_BitBucket_Compile, select Maven Project, click OK.
   

    

3. Scroll down to Source Code Management, select Git, enter your Repository URL, add your credentials, and specify the branch, for example, */master.
   

    

4. Scroll down to Build Trigger, then select Build when a change is pushed to BitBucket.
   

    

5. Scroll down to Build Environment, then select Delete workspaces before starts.
   

    

6. Scroll down to Build, and complete the first section as shown below: add the Root POM of your Talend project, define your Maven Goals and options, and point to your cmdline instance for MAVEN_OPTS.
   

    

7. Complete the section as shown below: locate your maven_user_settings.xml file.
   

    

8. Click Save. Your first Jenkins Job is complete.
   

    

9. To run it manually, click the green arrow with a clock, on the right side of the screen.
   

    

10. Click the console output to see the results, and you should end up with the status: Finished: SUCCESS.
    

     

11. As you did earlier, make a small change to your Job from the Studio, push the modification to Git, and make sure that the compile of the Jenkins Job has been triggered.

    

     

Test

1. Create a new Maven project, name it 02_Test, scroll down to Build Triggers, configure the project as shown below:
   

    

2. Set up the Build Environment:
   

    

3. Configure Build as shown below, then click Save:
   

Package

1. Create a new Maven project, name it 03_Package, scroll down to Build Triggers, and configure the project as shown below:
   

    

2. Set up the Build Environment:
   

    

3. Configure Build as shown below, then click Save:

   

Install

1. Create a new Maven project, name it 04_Install, scroll down to Build Triggers, and configure the project as shown below:
   

    

2. Set up the Build Environment:
   

    

3. Configure Build as shown below, then click Save:
   

    

4. Create a new Maven project, name it 05_Deploy, scroll down to Build Triggers, and configure the project as shown below:

   

5. Set up the Build Environment:
   

6. Configure Build as shown below, then click Save:
   

   MAVEN_OPTS:

   -Dproduct.path=/opt/cmdline
   -Dgeneration.type=local
   -DaltDeploymentRepository=snapshots::default::http://YOUR NEXUS PUBLIC IP:8081/repository/ben_snapshots/

7. Create a new Pipeline view from the Jenkins main page by clicking the plus [+] sign next to the tabs.
   

    

8. Give it a name, Pipeline View, select Build Pipeline View, then OK.
   

    

9. Specify the Pipeline Flow, as shown below. Keep the rest of the default settings. Click OK.

   

10. Run the first job manually, or make a small change to your Talend Job, and push the update to Bitbucket. Click the Pipeline view you just created.
    

11. Make sure that your pipeline ran successfully, and check your repository to confirm that your Talend Job has deployed.
    

     

    

Create an example Job

Create an example Job called TraceRecordsWithBreakpoint. Use a tFixedFlowInput to generate some source data such as:

1;Shong
2;Elise
3;Dave
4;Mike
5;Pedro

The detailed Job settings are shown in the following figure:

Set a breakpoint

To set a breakpoint on the data flow, proceed as follows:

1. Right-click the connector between two components and select Show Breakpoint Setup.

   

    

   Note: This feature is available only in Talend Data Integration (on subscription only).

   
   

2. In the Breakpoint tab, check Active conditional breakpoint and/or Use advanced mode to set a breakpoint. In this example, if you check Active conditional breakpoint and set a breakpoint in the Condition table, the Job will pause when the value of the Name column equals Mike.

   

Trace records with breakpoint

Now run the Job in Traces Debug mode and trace records. Follow these steps:

1. In the Run view, click Debug Run tab, select Traces Debug from the Debug list, and click Traces Debug to run the Job.

   

   
   

2. As the figure shows, the Job pauses when the value of the Name column equals Mike, when it matches the condition of the breakpoint.

   

Result

Now you can trace the records by clicking Previous, Next, and Breakpoint.

• Previous: return back to the previous record.
• Next: go to the next record.
• Breakpoint: continue to run and pause until next breakpoint.
• Basic Run: continue to run until ends.
• Kill: kill the Job.