AWS System Manager (SM), an AWS service, can be used to view and control infrastructures on AWS. It offers automation documents to simplify common maintenance and deployment tasks of AWS resources.

AWS SM consists of a collection of capabilities related to automation, such as infrastructure maintenance and deployment tasks of AWS resources as well as some related to Application Management and Configuration. Among them, is a capability called Parameter Store.

AWS System Manager Parameter Store

AWS Systems Manager (SM) Parameter Store provides secure, hierarchical storage for configuration data management and secrets management.

It allows you to store data such as passwords, database strings, and license codes as parameter values.

AWS SM Parameter Store benefits

Parameter Store offers the following benefits and features for Talend Jobs.

• Secured, highly scalable, hosted service with NO SERVERS to manage: compared to the setup of a dedicated database to store Job context variables.

• Control access at granular levels: specify who can access a specific parameter or set of parameters (for example, DB connection) at the user or group level. Using IAM roles, you can restrict access to parameters, which can have nested paths that can be used to define ACL-like access constraints. This is important for the control access of Production environment parameters.

• Audit access: track the last user who created or updated a specific parameter value.

• Encryption of data at rest and in transit: parameter values can be stored as plaintext (unencrypted data) or ciphertext (encrypted data). For encrypted value, KMS: AWS Key Management Service is used behind the scenes. Hence, Talend context variables with a Password type can be stored and retrieved securely without the implementation of a dedicated encryption/decryption process.

Another benefit of the AWS SM Parameter Store is its usage cost.

AWS SM Parameter Store pricing

AWS SM Parameter Store consists of standard and advanced parameters.

Standard parameters are available at no additional charge. The values are limited to 4 KB size, which should cover the majority of Talend Job use cases.

With advanced parameters (8 KB size), you are charged based on the number of advanced parameters stored each month and per API interaction.

Pricing example

Assume you have 5,000 parameters, of which 500 are advanced. Assume that you have enabled higher throughput limits and interact with each parameter 24 times per day, equating to 3,600,000 interactions per 30-day month. Because you have enabled higher throughput, your API interactions are charged for standard and advanced parameters. Your monthly bill is the sum of the cost of the advanced parameters and the API interactions, as follows: Cost of 500 advanced parameters = 500 * $0.05 per advanced parameter = $25 Cost of 3.6M API interactions = 3.6M * $0.05 per 10,000 interactions = $18 Total monthly cost = $25 + $18 = $43.

For more information on pricing, see the AWS Systems Manager pricing web site.

About parameters

A Parameter Store parameter is any piece of configuration data, such as a password or connection string, that is saved in the Store. You can centrally and securely reference this data in a Talend Job.

The Parameter Store provides support for three types of parameters:

• String
• String List
• Secure String

Organizing parameters into hierarchies

In Talend, context variables are stored as a list of key-value pairs independent of the physical storage (Job, file, or database). Managing numerous parameters as a flat list is time-consuming and prone to errors. It can also be difficult to identify the correct parameter for a Talend Project or Job. This means you might accidentally use the wrong parameter, or you might create multiple parameters that use the same configuration data.

Parameter Store allows you to use parameter hierarchies to help organize and manage parameters. A hierarchy is a parameter name that includes a path that you define by using forward slashes (/).

The following example uses three hierarchy levels in the name:

/Dev/PROJECT1/max_rows

AWS SM Parameter Store with Talend Job

Parameter Store can accede from the AWS Console, AWS CLI, or the AWS SDK, including Java. Talend Studio leverage the AWS Java SDK to connect numerous Amazon Services, but, as yet, not to Amazon System Manager.

Implementation of AWS SM Parameter Store connector

This initial implementation solely uses the current capabilities of Studio, such as Routines and Joblets.

A future version will leverage the Talend Component Development Kit (CDK) to build a dedicated connector for AWS System Manager.

Routine

The connector was developed in Java using the AWS SDK and exported as an UberJar (single JAR with all his dependencies embedded in it).

The AWSSSMParameterStore-1.0.0.jar file (attached to this article) is imported into the Studio local Maven Repository and then used as a dependency in the AwsSSMParameterStore Talend routine.

The routine provides a set of high-level APIs/functions of the Parameter Store for Talend Jobs.

package routines;

import java.util.Map;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

import com.talend.ps.engineering.AWSSMParameterStore;

public class AwsSSMParameterStore {
	private static final Log LOG = LogFactory.getLog(AwsSSMParameterStore.class);
	private static AWSSMParameterStore paramsStore;

	/*
	 * init
	 *
	 * Create a AWSSMParameterStore client based of the credentials parameters.
	 * Follows the "Default Credential Provider Chain".
	 * See  https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html
	 *
	 * Parameters:
	 * 	accessKey : (Optional) AWS Access Key
	 * 	secretKey : (Optional) AWS Secret Key
	 * 	region    : (Optional) AWS Region
	 *
	 * Return:
	 *  Boolean : False  if invalid combination of parameters
	 */
	public static boolean init(String accessKey, String secretKey, String region) { ...
	}

	/*
	 * loadParameters
	 *
	 * Retrieve all the parameters recursively with the path a prefix in their name
	 *
	 * Parameters:
	 * 	path : Parameter path prefix for the parameters
	 *
	 * Return:
	 *  Map of name, value pair of parameters
	 */
	public static Map<String, String> loadParameters(String path){ ...
	}

	/*
	 * saveParameter
	 *
	 * Retrieve all the parameters recursively with the path a prefix in their name
	 *
	 * Parameters:
	 * 	name        : Name of the parameter
	 * 	value       : Value of the parameter
	 * 	encrypt     : Encrypt the value the value in the Parameter Store
	 *
	 * Return:
	 *  Boolean : False if the save failed
	 */
	public static boolean saveParameter(String name, Object value, boolean encrypt) { ...
        }

}

The init function creates the connector to AWS SSM using the AWS Default Credential Provider Chain.

The loadParameters function connects to the Parameter Store and retrieves a set/hierarchy of parameters prefixed with a specific path (see the naming convention for the parameters below).

The result is returned as a Map key-value pair.

Important: In the returned Map, the key represents only the last part of the parameter name path. If the parameter name is: /Dev/PROJECT1/max_rows, the returned Map key for this parameter is max_rows.

The saveParameter function allows you to save a context parameter name and value (derived from a context variable) to the Parameter Store.

Joblets

Two Joblets were developed to connect to the AWS Parameter Store through the routine. One is designed to initialize the context variables of a Job using the parameters from the AWS Parameter Store. The other, as a utility for a Job to store its context variables into the Parameter Store.

Joblet: SaveContextVariableToAwsSSMParameterStore

The Joblet uses a tContextDump component to generate the context variables dataset with the standard key-value pair schema.

The tJavaFlex component is used to connect to the Parameter Store and save the context variables as parameters with a specific naming convention.

Parameter hierarchies naming convention for Talend context variables

In the context of context variables, the choice is to use a root prefix (optional) /talend/ to avoid any potential collision with the existing parameter name.

The prefix is appended with a string representing a runtime environment, for example, dev, qa, and prod. This to mimic the concept of the context environment found in the Job Contexts:

The parameter name is then appended with the name of the Talend Project (which is extracted from the Job definition) and, finally the name of the variable.

Parameter naming convention:

/talend/<environment name>/<talend project name>/<context variable name>

Example Job: job1 with a context variable ctx_var1 in a Talend Project PROJECT1.

The name of the parameter for the ctx_var1 variable in a development environment (identified by dev), is:

/talend/dev/PROJECT1/ctx_var1

For a production environment, prod, the name is:

/talend/prod/PROJECT1/ctx_var1

One option is to use the Job name as well in the hierarchy of the parameter name:

/talend/prod/PROJECT1/job1/ctx_var1

However, due to the usage of Talend Metadata connection, Context Group, and other that are shared across multiple Jobs, the usage of the Job name will result in multiple references of a context variable in the Parameter Store.

Moreover, if a value in the Context Group changes, the value needs to be updated in all the parameters for this context variable, which defies the purpose of the context group.

Joblet context variables

The Joblet uses a dedicated context group specific to the interaction with the Parameter Store.

• AWS Access & Secret keys to connect to AWS. As mentioned earlier, the routine leverages AWS Default Credential Provider Chain. If these variables are not initialized, the SDK looks for Environment variables or the ~/.aws/Credential (user directory on Windows ) or EC2 roles to infer the right credentials.

• AWS region of the AWS SM Parameter Store.

• Parameter Store prefix and environment used in the parameter path as described above in the naming convention.

Joblet: LoadContextVariablesFromAwsSSMParmeterStore

The second Joblet is used to read parameters from The Parameter Store and update the Job context variables.

The Joblet uses a tJavaFlex component to connect to SSM Parameter Store, leveraging the AwsSSMParameterStore.loadParameters routine function described above. It retrieves all the parameters based on the prefix path (see the defined naming convention above).

The tContextLoad use the tJavaflex output key-value pair dataset, to overwrite the default values of the context variables.

Joblet context variables

The load Joblet uses the same context group as the save counterpart.

Sample Talend Job

The sample Talend Job, generates a simple people's dataset using the tRowGenerator (first name, last name, and age), applies some transformations, and segregates the rows by age to create two distinct datasets, one for Adults ( age > 18) and one for Teenagers.

The two datasets are then inserted into a MySQL database in their respective tables.

The Job contains a mix of context variables, some are coming from a group defined for the MySQL Metadata Connection and some are specific to the Job: max_rows, table_adults, and table_teenagers.

Create Parameter Store entries for the context variables

The first step is to create all the parameters in the Parameter Store for the Job context variables. This can be done using the AWS console or through the AWS CLI, but those methods can be time-consuming and error-prone.

Instead, use the dedicated SaveContextVariableToAwsSSMParameterStore Joblet.

You need to drag-and-drop the Joblet into the Job canvas. There is no need to connect it to the rest of the Job components. It lists all the context variables, connects to AWS SM Parameter Store, creates the associated parameters, and stops the Job.

When the Job is executed, the System Manager Parameter Store web console should list the newly created parameters.

On the AWS console, the first column is not resizable, to see the full name of a parameter, you'll need to hide some of the columns.

You can also click a specific parameter to see the details.

For context variables defined with a Password type, the associated parameter is created as SecureString, which allows the value to be encrypted at rest in the store.

Talking about security, IAM access control can be leveraged to restrict access to a specific Operation team or to restrict access of a specific set of parameters such as production parameters: /talend/prod/*; developers will have access solely to the dev environment-related parameters, for example:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                // Allows to decrypt secret parameters
                "kms:Decrypt",
                "ssm:DescribeParameters"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "ssm:PutParameter",
                "ssm:LabelParameterVersion",
                "ssm:DeleteParameter",
                "ssm:GetParameterHistory",
                "ssm:GetParametersByPath",
                "ssm:GetParameters",
                "ssm:GetParameter",
                "ssm:DeleteParameters"
            ],
            // Grant access only to dev parameters
            "Resource": "arn:aws:ssm:AWS-Region:AWS-AccountId:parameter/talend/dev/*"
        }
    ]
}

Talend Cloud Job

In the context of a Talend Cloud Job/Task, the context variables don't need to be exported as connections or resources for Talend Cloud as they are initialized from the AWS Parameter Store.

You can only create a connection for the AWS SM Parameter Store credentials and config parameters.

Custom connection for AWS SM Parameter Store

The context group for the AWS SM Parameter Store, is externalized as Talend Cloud Custom Connection because, as yet, Talend Cloud doesn't have a native connector for AWS System Manager.

Talend Cloud Task

In Studio, you create a new Talend Cloud task by publishing the Job artifact to the cloud.

You'll then add the custom connection for AWS SM.

The additional context variables are exposed as advanced parameters, including the database connection parameters that are initialized from the Parameter Store.

A successful task execution on a cloud or Remote Engine means that the Job can connect to AWS SM, retrieve the parameters based on the naming convention set above, and initialize the corresponding context variables to allows the Job to connect to the MySQL database and create the requested tables.

Over the next few weeks, many of us will take some time to recharge before heading into the New Year. Why not take some of that time to invest in yourself and explore some of the learning resources available to Talend Academy subscribers?

Got a few minutes? Watch a video!

Best Practices videos are a quick way to sharpen your Talend skills in between other activities. They take only a minute or two, but they distill expertise from Talend consultants to deliver a meaningful learning experience.

Find them by logging in to Talend Academy, clicking the Search Talend Academy tile, then selecting Best Practices from the menu.

If you're a Talend partner, log in to the Partner Academy, then click Browse Catalog and select Best Practices from the menu.

Investigate Talend 8

Get a head start on learning more about Talend 8 by spending time with these resources, now upgraded to reflect the new functionality.

Talend Cloud Administration Learning Plan

The learning plan was upgraded to a Talend 8.0 environment, and the Logging and monitoring in Talend Management Console module was updated to integrate the new logging interface. Personal access tokens are now used to connect from Talend Studio to Talend Cloud throughout the learning plan.

 

Talend Data Fabric Installation Foundations Learning Plan

This learning plan now includes a new video, Talend Studio Installation, covering how to maintain a Talend Studio installation in Talend 8 using the new Feature Manager, and explaining how to manage and apply patches.

 

Talend Cloud Hands-on Migration Advanced Use Case

This use case was upgraded to demonstrate migration from a 6.4 on-premises environment to an 8.0 Talend Cloud environment (instead of 7.3) after the release of Talend 8. 

 

Talend Cloud Administrator Professional Exam

This professional exam was upgraded to test participants on their ability to migrate from a 6.4 Talend environment to an 8.0 Talend Cloud environment after the release of Talend 8.

 

Earn a new badge

Sharpen your skills by taking advanced Talend Academy courses in a familiar topic, or by starting a learning plan that's new to you.

Read more about our badging program on the Talend Community — you will gather trophies along the way, and earn a badge that you can share with the world!

 

 

Prepare to get certified

To challenge yourself even further, get ready to pass a certification exam in the new year.

Talend Academy offers certification preparation courses that provide review materials and a practice test to validate your knowedge and increase your confidence in passing the exam. When you do, you'll be awarded a Certified badge.

 

 

Experienced Talend professional?

Even after you've completed courses and created Talend projects, Talend Academy has advanced resources to help you hone your skills and continue learning!

Implementation Accelerators

Talend experts create Implementation Accelerators based on their extensive experience out in the field. They develop tools, frameworks, and scripts that help you optimize tasks such as implementing CI/CD workflows, automating creation of data pipelines, and verifying and cleansing addresses.

Talend Academy's extensive collection of Solution Templates delivers software artifacts and Talend Jobs, as well as instructions for how to use them. You can extend and customize them to fit your own environment and requirements, ensuring that your projects follow Talend best practices.

Find them all by clicking the tile on your Talend Academy home page.

 

Advanced Use Cases

Talend experts also write advanced use cases that describe solutions to real-world scenarios that you might encounter in your own production environment or at a client's site. These solutions use several Talend products together to solve a complex problem, or show how to use Talend products with third-party software to achieve your goals.

This tile is available on your Talend Academy home page.

Introduction

If you need to request to reschedule your Talend Certification Exam because you could not clear it on your first attempt, follow the steps outlined below. This guide will help you understand the process and the necessary details you need to provide to get the support you need.

Steps to Request Assistance

1. Gather Necessary Information:

   • User Email ID: Ensure you have the email address of the person who needs to reschedule the exam.

   • Exam Name: Provide the name of the exam that needs to be rescheduled.

   • Reason for Reschedule: Provide a brief description of why you need to reschedule the exam.

   • Preferred Reschedule Date and Time: Mention the preferred date and time for rescheduling the exam, along with the region.

   • Point of Contact (POC): Mention the point of contact for this request.

   • Internal Person's Email ID: Include the internal person's email address who should be informed once the request is completed.

2. Send an Email Request:

   • Email your request to customersupport@qlik.com.

Email Template for Requesting an Exam Reschedule

Use the following template to structure your email request:

Subject: Request to Reschedule Talend Certification Exam

Dear Support Team,

I hope this email finds you well.

I am writing to request a reschedule of my Talend Certification Exam. Below are the details:

• User Email ID: [Insert User Email]

• Exam Name: [Insert Exam Name]

• Reason for Reschedule: [Provide a brief description of the reason for the reschedule]

• Preferred Reschedule Date and Time: [Mention preferred date and time, along with the region]

• Point of Contact (POC): [Insert POC Name and Email]

• Internal Person's Email ID: [Insert Internal Person's Email]

I would greatly appreciate your support in processing this request. Thank you for your assistance and consideration.

Best regards, [Your Name]

Example of a Detailed Request for Different Time Zones

Example for IST (India Standard Time)

Subject: Request to Reschedule Talend Certification Exam

Dear Support Team,

I hope this email finds you well.

I am writing to request a reschedule of my Talend Certification Exam. Below are the details:

• User Email ID: sam@xyz.com

• Exam Name: Talend Data Integration Certified Developer Exam

• Reason for Reschedule: I was unable to clear the exam on my first attempt and would like to reschedule to better prepare for my next attempt.

• Preferred Reschedule Date and Time: April 15th, 2025, at 10:00 AM IST (India Standard Time)

• Point of Contact (POC): Jane Doe (jane@xyz.com)

• Internal Person's Email ID: [Insert Internal Person's Email]

I would greatly appreciate your support in processing this request. Thank you for your assistance and consideration.

Best regards, Sam

Example for EMEA (Europe, Middle East, and Africa)

Subject: Request to Reschedule Talend Certification Exam

Dear Support Team,

I hope this email finds you well.

I am writing to request a reschedule of my Talend Certification Exam. Below are the details:

• User Email ID: alice@xyz.com

• Exam Name: Talend Data Integration Certified Developer Exam

• Reason for Reschedule: I was unable to clear the exam on my first attempt and would like to reschedule to better prepare for my next attempt.

• Preferred Reschedule Date and Time: April 15th, 2025, at 10:00 AM CET (Central European Time)

• Point of Contact (POC): John Smith (john@xyz.com)

• Internal Person's Email ID: [Insert Internal Person's Email]

I would greatly appreciate your support in processing this request. Thank you for your assistance and consideration.

Best regards, Alice

Example for PST (Pacific Standard Time)

Subject: Request to Reschedule Talend Certification Exam

Dear Support Team,

I hope this email finds you well.

I am writing to request a reschedule of my Talend Certification Exam. Below are the details:

• User Email ID: bob@xyz.com

• Exam Name: Talend Data Integration Certified Developer Exam

• Reason for Reschedule: I was unable to clear the exam on my first attempt and would like to reschedule to better prepare for my next attempt.

• Preferred Reschedule Date and Time: April 15th, 2025, at 10:00 AM PST (Pacific Standard Time)

• Point of Contact (POC): Lisa Brown (lisa@xyz.com)

• Internal Person's Email ID: [Insert Internal Person's Email]

I would greatly appreciate your support in processing this request. Thank you for your assistance and consideration.

Best regards, Bob

Example for EST (Eastern Standard Time)

Subject: Request to Reschedule Talend Certification Exam

Dear Support Team,

I hope this email finds you well.

I am writing to request a reschedule of my Talend Certification Exam. Below are the details:

• User Email ID: carol@xyz.com

• Exam Name: Talend Data Integration Certified Developer Exam

• Reason for Reschedule: I was unable to clear the exam on my first attempt and would like to reschedule to better prepare for my next attempt.

• Preferred Reschedule Date and Time: April 15th, 2025, at 10:00 AM EST (Eastern Standard Time)

• Point of Contact (POC): Michael Johnson (michael@xyz.com)

• Internal Person's Email ID: [Insert Internal Person's Email]

I would greatly appreciate your support in processing this request. Thank you for your assistance and consideration.

Best regards, Carol

Important Notes:

• Ensure all details are accurate and complete before sending your request.

• Use the appropriate email address based on your user type.

Following these steps will help you effectively request to reschedule your Talend Certification Exam. If you have any further questions, feel free to reach out to the respective teams.

Note: Once the email is sent to the Qlik Talend support team, a support request will be raised, and a support engineer will assist you further with the next steps. The rescheduling of the exam will depend on the validation of the request and the availability of the new exam date.

Error java.io.IOException: CreateProcess error=5, Access is denied error in jobserver.log.

The following error is shown when trying to execute a Job in Talend Administration Center:

Connection to JobServer Failed

Error in the JobServer.log:

Caused by: java.io.IOException: Cannot run program "\"C:\Program Files\Java\jre1.8.0_121\bin\java.exe\"" (in directory "C:\Talend\6.2.1\JobServer\repository\catalogueProduits_20170404_172121_XYZ\SAP_B2B_catalogueProduits"): CreateProcess error=5, Access is denied
at java.lang.ProcessBuilder.start(Unknown Source)
at java.lang.Runtime.exec(Unknown Source)
at org.talend.remote.jobserver.server.CommandServerSocket.runJob(CommandServerSocket.java:688)
... 9 more
Caused by: java.io.IOException: CreateProcess error=5, Access is denied
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.<init>(Unknown Source)
at java.lang.ProcessImpl.start(Unknown Source)
... 12 more

Cause

Executable path specified for org.talend.remote.jobserver.commons.config.JobServerConfiguration.JOB_LAUNCHER_PATH in the <JobServerInstallationDirectory>\agent\conf\TalendJobServer.properties file should not be set between double quotes ("").

Resolution

In TalendJobServer.properties, replace the line:

org.talend.remote.jobserver.commons.config.JobServerConfiguration.JOB_LAUNCHER_PATH="C:\Program Files\Java\jdk1.8.0_111\jre\bin\java.exe"

with

org.talend.remote.jobserver.commons.config.JobServerConfiguration.JOB_LAUNCHER_PATH=C:\Program Files\Java\jdk1.8.0_111\jre\bin\java.exe

No double quotes around the value.

The use of quotes is only necessary when your path contains spaces, Otherwise, enter the path without quotes.

Environment

Talend Administration Center 

Apache Airflow is a platform to programmatically author, schedule, and monitor workflows. Airflow uses Directed Acyclic Graph (DAG) to create workflows or tasks. For more information, see the Apache Airflow Documentation page.

This article shows you how to leverage Apache Airflow to orchestrate, schedule, and execute Talend Data Integration (DI) Jobs.

Environment

• Apache Airflow 1.10.2
• Nexus 3.9
• WinSCP 5.15
• PuTTY

Prerequisites

1. Apache Airflow installed on a server (follow the Installing Apache Airflow on Ubuntu/AWS installation instructions).
2. Python 2.7 installed on the Airflow server.
3. Java 1.8 installed on the Airflow server.
4. Access to the Nexus server from the Airflow server (in this example, both Nexus and Airflow are installed on the same server).
5. Talend 7.x Jobs published to the Nexus repository. (For more information on how to set up a CI/CD pipeline to publish Talend Jobs to Nexus, see Configuring Jenkins to build and deploy project items in the Talend Help Center.)
6. Access to the setup_files.zip file (attached to this article).

Process flow

1. Develop Talend DI Jobs using Talend Studio.
2. Publish the DI Jobs to the Nexus repository using Talend CI/CD module.
3. Execute the Directed Acyclic Graph (DAG) in Apache Airflow:
   • The first step in DAG is to download the Job executable from Nexus using the customized script.
   • The second step is to execute the downloaded Job.

   

Configuration and execution

1. Login to the Airflow server through SSH using WinSCP or PuTTY.

2. Create two folders named jobs and scripts under the AIRFLOW_HOME folder.

   

3. Extract the setup_files.zip, then copy the shell scripts (download_job.sh and delete_job.sh) to the scripts folder.

   

4. Copy the talend_job_dag_template.py file from the setup_files.zip to your local machine and update the following:

   • nexus_host
   • nexus_port
   • airflow_home
   • nexus_repo
   • job_group_id
   • job_name
   • job_version

   Also, update the default_args dictionary based on your requirements.

   

   For more information, see the Apache Airflow documentation: Default Arguments.

5. The DAG template provided is programmed to trigger the task externally. If you plan to schedule the task, update the schedule_interval parameter under the DAG for airflow task with values based on your scheduling requirements.

   

   For more information on values, see the Apache Airflow documentation: DAG Runs.

6. Rename the updated template file and place it in the dags folder under the AIRFLOW_HOME folder.

7. After the Airflow scheduler picks up the DAG file, a compiled file with the same name and with a .pyc extension is created.

   

8. Refresh the Airflow UI screen to see the DAG.

   Note: If the DAG is not visible on the User Interface under the DAGs tab, restart the Airflow webserver and the Airflow scheduler.

   

9. To schedule the task, toggle the button to On. You can also run the task manually.
10. Monitor the run status on the Airflow UI.

Conclusion

In this article, you learned how to author, schedule, and monitor workflows from the Airflow UI, and how to download and trigger Talend Jobs for execution.

Talend Cloud platform provides computational capabilities that allow organizations to securely run data integration processes natively from cloud to cloud, on-premises to cloud, or cloud to on-premises environments.

These capabilities are powered by compute resources, commonly known as Engines. This article covers the four basic types.

Content:

• Cloud Engine (CE)
• Remote Engine (RE)
• Remote Engine Gen2 (REG2)
• Cloud Engine for Design (CE4D)
• Cloud Engine versus Remote Engine
• Cloud Engine for Design versus Remote Engine Gen 2
• Need for additional engines
• Cloud Engine - usage considerations
• Remote Engine – recommendations
• Summary

Cloud Engine (CE)

A Cloud Engine is a compute resource managed by Talend in Talend Cloud that executes Job tasks.

• You can allocate Cloud Engines to environments in proportion to the number of concurrent task executions, workloads, and Job designs you plan to run.
• All environments can use unassigned Cloud Engines. If Cloud Engines are not allocated to specific environments, you may not be able to run certain tasks because other tasks might keep all the unassigned Cloud Engines occupied.
• Cloud Engines can handle parallel execution of three tasks. That means a maximum of three different tasks can run in parallel on a single Cloud Engine. (A task cannot run more than once concurrently on a single Cloud Engine.) So, if three different tasks are already running on a Cloud Engine or if the same task is already running on that engine, another Cloud Engine is selected to execute the task.
• If you run your task in Cloud Exclusive mode, you cannot execute other tasks on that Cloud Engine. You can only use Cloud Exclusive engines in environments that do not have Cloud Engines assigned to them.
• Cloud Engines have limited system resources – memory usage: 8 GB, disk usage: 200 GB.
• Only standard Data Integration Batch Jobs can run on Cloud Engines.
• You cannot group Cloud Engines together to form clusters.
• Cloud Engines are hosted on AWS or Azure Cloud.
• Talend manages Cloud Engines.
• Cloud Engines employ TCP communication.

Remote Engine (RE)

A capability in Talend Cloud platform that allows you to securely run data integration Jobs natively from cloud to cloud, on-premises to cloud, or cloud to on-premises environments completely within your environment for enhanced performance and security, without transferring the data through the Cloud Engines in Talend Cloud platform.

Java-based runtime (similar to a Cloud Engine) to execute Talend Jobs on-premises or on another cloud platform that you control.

• Remote Engines allow you to run Jobs, Routes, and Data Service tasks.
• Data Service and Route Microservice tasks can only be deployed on Remote Engines. OSGi type deployments require that Talend Runtime version 7.1.1 or higher is installed and running on the same machine as the Talend Remote Engine.
• Remote Engines support configurable max parallel execution: by default, a maximum of three different tasks can run in parallel on the same Remote Engine. However, this is a modifiable configuration.
• Remote Engines can be grouped to form clusters called Remote Engine Cluster. Remote Engines added to a cluster cannot be used to execute tasks directly from Talend Studio.
• Remote Engines are hosted on-premises or on the cloud.
• You manage Remote Engines.
• Remote Engines employ HTTPS communication.

Remote Engine Gen2 (REG2)

A Remote Engine Gen2 is a secure execution engine on which you can safely execute data pipelines (that is, data flows designed using Talend Pipeline Designer). It allows you to have control over your execution environment and resources because you can create and configure the engine in your own environment (Virtual Private Cloud or on-premises). Previously referred to as Remote Engines for Pipelines, this engine was renamed Remote Engine Gen2 during H1/2020. It is a Docker-based runtime to execute data pipelines on-premises or on another cloud platform that you control.

A Remote Engine Gen2 ensures:

• Data processing in a safe and secure environment, because Talend never has access to your pipelines' data and resources
• Optimal performance and security by increasing the data locality instead of moving large data to computation

Cloud Engine for Design (CE4D)

Cloud Engine for Design is a built-in runner that allows you to easily design pipelines without setting up any processing engines. With this engine you can run two pipelines in parallel. For advanced processing of data, Talend recommends installing the secure Remote Engine Gen2.

• CE4Ds have limited system resources – memory usage: 8 GB
• CE4Ds support a maximum of two pipelines that can run in parallel on a single CE4D
• CE4Ds should be used only for design purposes; that is, you shouldn’t use them to execute data pipelines in a Production environment

Cloud Engine versus Remote Engine

The following table lists a comparative perspective between the two engines:

Cloud Engine for Design versus Remote Engine Gen 2

Need for additional engines

Additional engines (CE or RE) may be required if you have one or more of the following use cases:

• Continuous delivery – for example, Dev and QA separate from UAT and Production environments
• Data access - data is in two different private locations where an engine is needed in each site (or a mix of Cloud and Remote Engines)
• Scalability - concurrent Job volume requires additional engines, Jobs are complex and require significant memory or CPU

These use cases depend on the deployment architecture in the specific customer environment and layout of the Remote Engine at the environment or workspace level configurations. This would need proper capacity planning and automatic horizontal and vertical scaling of the compute Engines.

Cloud Engine - usage considerations

Remote Engine – recommendations

If a source or target system is not accessible through the internet:

If one of the systems is not accessible using the internet, then a Remote Engine is needed.

When single flow requirements exceed the capacity of a Talend Cloud Engine:

If the Cloud Engine is too small (for example, the maximum memory of 5.25 GB, temporary space of 200 GB, two vCPU, or the maximum of 225 GB per hour) then, a Remote Engine is needed.

If a native driver is required:

If the solution requires a native driver, which is not part of the Talend action or Job generated code, a typical case for this is SAP with the JCO v3 Library, MS SQL Server Windows Authentication, then a Remote Engine is needed.

Data jurisdiction, security, or compliance reasons:

It may be desirable or required to retain data in a particular region or country for data privacy reasons. The data being processed may be subject to regulations such as PCI or HIPAA, or it may be more efficient to process the data within a single data center or public cloud location. These are all valid reasons to use a Remote Engine.

Summary

References

Talend Help Center documentation:

• REG2 and CE4D Architecture

• REG2 and CE4D

• REG2

• Execution engines in Talend Cloud

Talend Studio does not start and displays an error message:

An error has occurred. see the log file /studio/configuration/xxxxxxxx.log

The log files read:

!MESSAGE Application error !STACK 1 java.lang.IllegalStateException: Unable to acquire application service. Ensure that the org.eclipse.core.runtime bundle is resolved and started (see config.ini).     at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:81)     at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:400)     at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     at java.lang.reflect.Method.invoke(Method.java:498)     at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:659)     at org.eclipse.equinox.launcher.Main.basicRun(Main.java:595)     at org.eclipse.equinox.launcher.Main.run(Main.java:1501)

Cause

Cached values.

Resolution

To clear cached values and recreate certain configurations:

1. Close Studio and go to the Studio installation directory.
2. Open the configurations folder and delete the folders that begin with org. -- except org.talend.configurator.
3. Restart Studio.

If the issue persists, create a backup of your workspace, then switch to a new empty workspace and try to pull your projects again.

You can observe your Data Integration Jobs running on Talend Remote Engines if your Jobs are scheduled to run on Talend Remote Engine version 2.9.2 or later.

This is a step-by-step guide on how Talend Cloud Management Console can provide the data needed to build your own customized dashboards, with an example of how to ingest and consume data from Microsoft Azure Monitor.

Once you have set up the metric and log collection system in Talend Remote Engine and your Application Performance Monitoring (APM) tool, you can design and organize your dashboards thanks to the information sent from Talend Cloud Management Console to APM through the engine.

Content:

• Prerequisites
• Configuring and starting the remote engine
• Get the metric details from the API
• Push metric to Azure logs workspace
• Observable metrics
• Queries and dashboard example
• Report One: A sample report for the number of rows processed by a component within the overall run
• Report Two: A sample report to showcase the average time taken by each component grouped by Job
• Report Three: A sample report to showcase OS memory and file storage available in MB
• Report Four: A sample report to showcase all jvm_process_cpu_load events counted in the last two days, in15-minute intervals
• Azure dashboard
• Pin the reports to the Azure dashboard
• Sample Talend Data Integration Job
• Job structure
• Job details

Prerequisites

This document has been tested on the following products and versions running in a Talend Cloud environment:

• Remote Engine 2.9.2 #182 – downloaded and installed following the instructions in Installing Talend Remote Engine in the Talend Help Center.

Optional requirements for obtaining detailed Job statistics:

• Studio 7.3.1 R2020-07 – downloaded from the Talend Cloud Portal and updated with the appropriate monthly patch
• Republish your Jobs from the new version of Studio to Talend Cloud Management Console

Configuring and starting the remote engine

To configure the files and check that the Remote Engine is running, navigate to the Monitoring Job runs on Remote Engines section of the Talend Remote Engine User Guide for Linux. 

Get the metric details from the API

Use any REST client, such as Talend API Tester or Postman, and use the endpoint as explained below.

1. Endpoint:

   GET http://ip_where_RE_is_installed:8043/metrics/json

   8043 is the default http port of Remote Engines. Replace it with the port you used when installing the Remote Engine.
2. Add a header: Authorization Bearer {token}.
   This token is defined in the etc/org.talend.observability.http.security.cfg file as endpointToken={token}.
3. Example:

   GET http://localhost:8043/metrics/json
           Authorization Bearer F7VvcRAC6T7aArU

   
   

Push metric to Azure logs workspace

There are numerous ways to push the metric results to any analytics and visualization tool. This document shows how to use the Azure monitor HTTP data collector API to push the metrics to an Azure log workspace. Python code is also used to send the logs in batch mode at frequent intervals. Alternatively, you can create a Talend Job as a service for real-time metric extraction. For more information, see the attached Job and Python Code.zip file.

The logs are pushed to the Azure Log Analytics workspace as “custom logs”.

Observable metrics

Talend Cloud Management Console provides metrics through Talend Remote Engine. They can be integrated in your APM tool to observe your Jobs.  

For the list of available metrics, see Available metrics for monitoring in the Talend Remote Engine User Guide for Linux. 

Queries and dashboard example

Report One: A sample report for the number of rows processed by a component within the overall run

Query:

Remote_Engine_OBS_CL 
|where TimeGenerated > ago(2d)
|where name_s=='component_connection_rows_total' 
|summarize sum(value_d) by context_target_connector_type_s
|render piechart

Chart:

Report Two: A sample report to showcase the average time taken by each component grouped by Job

Query:

Remote_Engine_OBS_CL 
|where TimeGenerated > ago(2d)
|where name_s=='component_execution_duration_seconds'
|summarize count(), avg(value_d) by context_artifact_name_s,context_connector_label_s

Chart:

Report Three: A sample report to showcase OS memory and file storage available in MB

Query:

Remote_Engine_OBS_CL 
|where name_s=='os_memory_bytes_available' or name_s =='os_filestore_bytes_available'
|summarize sum(value_d)/1000000 by name_s

Chart:

Report Four: A sample report to showcase all jvm_process_cpu_load events counted in the last two days, in 15-minute intervals

Query: 

Remote_Engine_OBS_CL 
|where TimeGenerated > ago(2d)
|where name_s =='jvm_process_cpu_load'
|summarize events_count=sum(value_d) by bin(TimeGenerated, 15m), context_artifact_name_s
|render timechart

 Chart:

Azure dashboard

Pin the reports to the Azure dashboard

Sample Talend Data Integration Job

This section explains the sample Job used to send the metric logs to the Azure log workspace. This Job is available in the attached Job and Python Code.zip file.

Job structure

Job details

The components used and their detailed configurations are explained below.

tREST

Component to make a REST API Get call.

tJavaRow

The component used to print the response from the API call.

tFileOutputRaw

The component used to create a JSON file with the API response body.

tSystem

Component to call the Python code.

tJava

Related Content

Log management and monitoring in Talend Cloud.

Log4j, incorporated in Talend software, is an essential tool for discovering and solving problems. This article shows you some tips and tricks for using Log4j.

The examples in this article use Log4j v1, but Talend 7.3 uses Log4j v2. Although the syntax is different between the versions, anything you do in Log4j v1 should work, with some modification, in Log4j v2. For more information on Log4j v2, see Configuring Log4j, available in the Talend Help Center.

Content:

• Configuring Log4j in Talend Studio
• Emitting messages
• Routines
• Controlling Log4j message formats with patterns
• Logging levels
• Using Appenders
• Using filters
• Overriding default settings in Talend Administration Center

Configuring Log4j in Talend Studio

Configure the log4j.xml file in Talend Studio by navigating to File > Edit Project properties > Log4j.

You can also configure Log4j using properties files or built-in classes; however, that is not covered in this article.

Emitting messages

You can execute code in a tJava component to create Log4j messages, as shown in the example below:

log.info("Hello World");
log.warn("HELLO WORLD!!!");

This code results in the following messages:

[INFO ]: myproject.myjob - Hello World
[WARN ]: myproject.myjob - HELLO WORLD!!!

Routines

You can use Log4j to emit messages by creating a logger class in a routine, as shown in the example below:

public class logSample {
/*Pick 1 that fits*/
	private static org.apache.log4j.Logger log = org.apache.log4j.Logger.getLogger(logSample.class);
	private static org.apache.log4j.Logger log1 = org.apache.log4j.Logger.getLogger("from_routine_logSample");
 /*...*/
   public static void helloExample(String message) {
        if (message == null) {
            message = "World"; 
        }
        log.info("Hello " + message + " !");
        log1.info("Hello " + message + " !");
    }
}

To call this routine from Talend, use the following command in a tJava component:

logSample.helloExample("Talend");

The log results will look like this:

[INFO ]: routines.logSample - Hello Talend !
[INFO ]: from_routine_logSample - Hello Talend !

Using <routineName>.class includes the class name in the log results. Using free text with the logger includes the text itself in the log results. This is not really different than using System.out, but Log4j can be customized and fine-tuned.

Controlling Log4j message formats with patterns

You can use patterns to control the Log4j message format. Adding patterns to Appenders customizes their output. Patterns add extra information to the message itself. For example, when multiple threads are used, the default pattern doesn't provide information about the origin of the message. Use the %t variable to add a thread name to the logs. To easily identify new messages, it's helpful to use %d to add a timestamp to the log message.

To add thread names and timestamps, use the following pattern after the CONSOLE appender section in the Log4j template:

<param name="ConversionPattern"  		  		
       value= "%d{yyyy-MM-dd HH:mm:ss}  [%-5p] (%t): %c - %m%n" />

The pattern displays messages as follows:

ISO formatted date [log level] (thread name): class projectname.jobname - message contents

If the following Java code is executed in three parallel threads, using the sample pattern above helps distinguish between the threads.

java.util.Random rand = new java.util.Random();
log.info("Hello World");
Thread.sleep(rand.nextInt(1000));
log.warn("HELLO WORLD!!!");
logSample.helloExample("Talend");

This results in an output that shows which thread emitted the message and when:

2020-05-19 12:18:30  [INFO ] (tParallelize_1_e45bc79b-d61f-45a3-be8f-7089ab6d565d): myproject.myjob_0_1.myjob - Hello World
2020-05-19 12:18:30  [INFO ] (tParallelize_1_4064c9b8-0585-41e0-b9f0-95fb31e602b7): myproject.myjob_0_1.myjob - Hello World
2020-05-19 12:18:30  [INFO ] (tParallelize_1_a8ef1065-0106-4b45-8a60-d02a9cbe1f00): myproject.myjob_0_1.myjob - Hello World
2020-05-19 12:18:30  [WARN ] (tParallelize_1_e45bc79b-d61f-45a3-be8f-7089ab6d565d): myproject.myjob_0_1.myjob - HELLO WORLD!!!
2020-05-19 12:18:30  [INFO ] (tParallelize_1_e45bc79b-d61f-45a3-be8f-7089ab6d565d): routines.logSample - Hello Talend !
2020-05-19 12:18:30  [INFO ] (tParallelize_1_e45bc79b-d61f-45a3-be8f-7089ab6d565d): from_routine.logSample - Hello Talend !
2020-05-19 12:18:30  [WARN ] (tParallelize_1_a8ef1065-0106-4b45-8a60-d02a9cbe1f00): myproject.myjob_0_1.myjob - HELLO WORLD!!!
2020-05-19 12:18:30  [INFO ] (tParallelize_1_a8ef1065-0106-4b45-8a60-d02a9cbe1f00): routines.logSample - Hello Talend !
2020-05-19 12:18:30  [INFO ] (tParallelize_1_a8ef1065-0106-4b45-8a60-d02a9cbe1f00): from_routine.logSample - Hello Talend !
2020-05-19 12:18:31  [WARN ] (tParallelize_1_4064c9b8-0585-41e0-b9f0-95fb31e602b7): myproject.myjob_0_1.myjob - HELLO WORLD!!!
2020-05-19 12:18:31  [INFO ] (tParallelize_1_4064c9b8-0585-41e0-b9f0-95fb31e602b7): routines.logSample - Hello Talend !
2020-05-19 12:18:31  [INFO ] (tParallelize_1_4064c9b8-0585-41e0-b9f0-95fb31e602b7): from_routine.logSample - Hello Talend !

If you want to know which component belongs to which thread, you need to change the log level to add more information.

You can do this in Studio on the Run tab, in the Advanced settings tab of the Job execution.

In Talend Administration Center, you do this in Job Conductor.

Using DEBUG level adds a few extra lines to the log file, which can help you understand which parameters resulted in a certain output:

2020-05-19 12:51:50  [DEBUG] (tParallelize_1_c6de81be-1bbf-4f9b-9b7a-3d92bf345c40): myproject.myjob_0_1.myjob - tParallelize_1 - The subjob starting with the component 'tJava_1' starts.
2020-05-19 12:51:50  [DEBUG] (tParallelize_1_fa636a36-9f53-423f-abc6-b26c4c52c5b4): myproject.myjob_0_1.myjob - tParallelize_1 - The subjob starting with the component 'tJava_3' starts.
2020-05-19 12:51:50  [DEBUG] (tParallelize_1_d4da8ea0-4401-4229-82e9-86ff0ed67c3b): myproject.myjob_0_1.myjob - tParallelize_1 - The subjob starting with the component 'tJava_2' starts.

Keep in mind the following:

• Changing the default log pattern causes Studio to stop coloring the messages.
• The default log level in Studio is defined by the root logger's priority value (Warn, by default).
• Changing the log level changes the number of messages.
• Changing the pattern changes the message format.

Logging levels

The following table describes the Log4j logging levels you can use in Talend applications:

These levels offer high-level controls for messages. When changed from the outside they affect only the Appenders that did not specify a log level and rely on the level set by the root logger.

Using Appenders

Log4j messages are processed by Appenders, which route the messages to different outputs, such as to console, files, or logstash. Appenders can even send messages to databases, but for database logs, the built-in Stats & Logs might be a better solution.

Storing Log4j messages in files can be useful when working with standalone Jobs. Here is an example of a file Appender:

<appender name="ROLLINGFILE" class="org.apache.log4j.RollingFileAppender">
  <param name="file" value="rolling_error.log"/>
  <param name="Threshold" value="ERROR"/>
  <param name="MaxFileSize" value="10000KB"/>
  <param name="MaxBackupIndex" value="5"/>
  <layout class="org.apache.log4j.PatternLayout">
    <param name="ConversionPattern" 
           value="%d{yyyy-MM-dd HH:mm:ss}  [%-5p] (%t): %c - %m%n"/>
  </layout>
</appender>

You can use multiple Appenders to have multiple files with different log levels and formats. Use the parameters to control the content. The Threshold value of ERROR doesn't provide information about the Job execution, but a value of INFO makes errors harder to detect.

For more information on Appenders, see the Apache Interface Appender page.

Using filters

You can use filters with Appenders to keep messages that are not of interest out of the logs. Log4j v2 offers regular expression based filters too.

The following example filter omits any Log4j messages that contain the string " - Adding the record ".

<filter class="org.apache.log4j.varia.StringMatchFilter">
	<param name="StringToMatch" value=" - Adding the record " />
	<param name="AcceptOnMatch" value="false" />
</filter>

Overriding default settings in Talend Administration Center

When a Java program starts, it attempts to load its Log4j settings from the log4j.xml file. You can modify this file to change the default settings, or you can force Java to use a different file. For example, you can do this for Jobs deployed to Talend Administration Center by configuring the JVM parameters. This way, you can change the logging behavior for a Job without modifying the original Job, or you can revert back to the original logging behavior by clearing the Active check box.

When attempting to execute the automatic installer.exe for Remote Engine, on Windows Server 2019, it fails with the error:

Error running C:\TalendRemoteEngine/bin/client.bat -a 8104 -h localhost -u tadmin "feature:install wrapper"

When attempting to run the Remote Engine manually by executing the trun command in the bin directory of the Remote Engine installation, the following error occurs:

Cause

The installer.exe itself can cause the error during the automatic run of the Remote Engine.

The error caused during the manual run of Remote Engine occurs when the JAVA_HOME and PATH environment variables are not set up correctly on the machine where this is happening and can cause the batch files to fail when starting.

Resolution

The best way to avoid the error caused during the automatic run of the Remote Engine is to clear the existing Remote Engine installation and install it again manually with 7-Zip.

To avoid the error caused during the manual run of Remote Engine, set the JAVA_HOME and PATH environment variables according to the Setting up JAVA_HOME instructions available in Talend Cloud Installation Guide for Windows.