Talend Cloud Contextualization

TalendSolutionExpert — Tue, 23 Jan 2024 02:35:30 GMT

Overview

Talend Integration Cloud is a powerful integration platform that allows developers and system integrators to design, develop, and deploy Talend Data Integration Jobs. These Jobs can be administered centrally, and deployed and executed either on Talend Cloud or on a customer’s remote engine.

One of the key features of Talend Cloud is the concept of Contextualization, which effectively allows you to create a session-wide global variable that can be dynamically set and utilized during the execution of a Job. This provides these key benefits:

Job parameterization so values can be set externally
Separation of internal logic and the values that influence the logic
Ability to maintain key/value pairs internally and by separate role
Ability to version control context values
Ability to make context variables independent of the Jobs that are using them

This article explains the various implementation options that are available under the Talend Cloud platform.

Prerequisites

Understanding of Talend Data Integration
Familiarity with:
- Contextualization
- Talend Cloud configurations
- Talend Cloud concepts

Contextualization

In Talend Cloud, you have three implementation options:

Context group name as environment name

Connections

Resource file

Context group name as environment name

Log in to Talend Cloud and navigate to Integration Cloud Integration > Administration > Environments.
Create three environments.
Switch to Talend Studio. Open your Job, select the Contexts tab, and create context groups that have the same names as the environment names (as shown in the red box below).
1. Ensure that each context group contains each of the context parameter items (though the values for the context parameter items within each context group are different).
2. The name of each context parameter item must begin with parameter_.
  The screenshot shows connection values for a database instance for each environment.
Save and publish the Job, then add a Flow based on the published Job.
Switch to the first environment you created, in this example, the Development environment. Launch the Flow.
Notice that on the context selection step, the parameter values associated with the context group that matches the environment are shown as defaults.

Promoting the Flow to other environments, such as Test and Production, automatically substitutes the parameter item values of that context group. For example, when running in the Test environment, the Test context group values are shown as defaults.

When running in the Production environment, the Production Context Group Values are shown as defaults.

Connections

Although the previous methodology is a great way to automatically populate default values as a Job/Flow is migrated from each environment, the context parameter values are nevertheless hard-coded in the Job itself. Using Connection metadata separates the connectivity values from the Job, which reflects the modern reality of segregation of duties, where environmental connections are maintained by CloudOps while the Jobs are being developed by developers.

This section explains how to configure connections in Talend Cloud and have them picked up by the Job.

Log in to Talend Cloud and navigate to Integration Cloud.
Select the Development environment that you configured in the previous section.
From the navigation menu on the left, select Management, select Connections, then click Add Connection.
Select the Custom connection type from the list.
In the new Custom Connection pop-up window, provide a connection type. This must have a unique name with no spaces and should reflect the actual type of the connection. This example shows a connection to an Oracle DB named OracleDB.
Ensure that each parameter item name begins with connection_ConnectionType_parameterName.
Click Create to create the connection. Your connection configuration should be similar to this:
Switch to Talend Studio. Open your Job and create all the connection parameters with the same names you used in your connection (connection_ConnectionType_parameterName).
Once the context parameters are created, you can use them in the usual way by using the reference context.connection_.
Deploy the Job to Talend Cloud and create a Flow.
Once deployed, launch the Flow, then select the connection from the drop-down list. This passes the configured connections to the Flow.
Once the Job runs, it can be promoted to the next environment, which in this example is Test. Since there is no connection set up in the Test environment, when promoting from Development to Test, select the Override option. This copies the connection and creates it in the Test environment.
After the Job has been promoted to the Test environment, the dependent connection metadata is copied, and the values are set to <change-me>.
When the Flow is promoted to the target environment, navigate to the target environment, in this case, Test. From the left menu, select Management, then select Connections and click the newly promoted/created connection.
Edit the connection and change the values to match the Test environment.
Save the file and launch the newly promoted Flow, and the new connections are picked up.

Resource file

The last option is to use a resource file, which is the most flexible and widely used option. Here, create an external text file, such as .txt or .csv, containing key-value pairs separated with either a comma or any other valid character. These key values are then passed into the Flow at execution time, which in turn maps them to their corresponding context variables of the same names that were created at design time.

This methodology turns the creation and maintenance of context variables and their values into cloud operations tasks, and separates them from Job design and development. Resource files provide great flexibility in allowing not only credentials and connectivity values, but other variables that influence the execution of the flow.

Note: maintenance and version control of the resource file is a task that should be part of your configuration management processes and procedures, which are outside of Talend Cloud Platform.

Using a text editor, create a text file that contains key-value pairs for a typical DB connection.
Save the file and log in to Talend Cloud, then navigate to the target environment. From the menu on the left, click Management, then select Resources.
From the pane on the right, select Add Resource to create the new resource.
Browse and upload the resource file that you created in Step 1.
Once uploaded, the resource is then created.
In Talend Studio, using the Repository Navigator, create a new context group.
Talend recommends naming the context group the same as the Talend Cloud resource name.
Using the following guidelines, define the context variables:
1. Make the first context parameter the reference to the resource file. When defining this parameter, you must follow the naming convention of resource_file_.
2. The rest of the context parameter names in the group should exactly match the names that are defined as keys in the resource file uploaded.
3. You can also use default values in the group, for when the Job is tested locally in Studio.
Add the context group to the Job where the context variables will be utilized.
You can run the Job locally from Studio where the values from the default context group are used, as the resource file is stored centrally, and is deployed within the cloud.

Note: when the Job runs locally, there isn't a resource file to load, so it uses the default values. Thus, the error is expected and can be ignored.
When the Job is deployed and run in the cloud, it uses the resource file.
From the target environment where the Job was deployed, navigate to Flows, locate the newly deployed Job, and launch the Flow.
Since your Job contains a context parameter that starts with the reference resource_file_, Talend Cloud automatically links the predefined resource file to the context variable defined in the Job.
Select Go Live and launch the Flow against the desired engine.
The Job then uses the values that are defined in the resource file.
Examine the log file, and notice the resource file values.

Using implicit load instead of tFileInputDelimited

As an alternative to using the tFileInputDelimited component for loading the resource file, you can use the implicit load within the Job.

From your Job, select the Job tab, then select Extra.
Clear the Use Project Settings check box. Enable the Implicit tContextLoad check box. Select the From File radio button and in the From File text field, refer to the context parameter.

You do not need to use the contextLoad component explicitly, as this will load the values.
Publish the Job to Talend Cloud and launch the Flow.
Select the resource file, then select Go Live.
Upon the successful completion of the Job, examine the log file.

article Talend Cloud Contextualization in Official Support Articles

Talend Cloud Contextualization

Overview

Prerequisites

Contextualization

Context group name as environment name

Connections

Resource file

Using implicit load instead of tFileInputDelimited