Using Azure Functions to trigger Talend Cloud Jobs

[TalendSolutionExpert]

Overview

This article explains how to use Azure Functions to trigger a Job managed by the Talend Cloud platform. It explains how to use a trigger on Azure Blob to execute a Job published on Talend Cloud. The Job is designed in Talend Studio and published to the Talend Cloud platform where its execution on the cloud or a remote engine is managed.

It covers:

• Publishing a Job to Talend Cloud
• Creating an Azure function based on a Blob trigger using the JavaScript language
• Testing the integration end-to-end

Architecture

The diagram shows the technical implementation architecture of the solution described in this article.

Prerequisites

Microsoft Azure

• You need access to Microsoft Azure with a valid account (portal.azure.com).
• You need full access to Azure services.

Talend

• You need to be familiar with Talend Studio and Talend Cloud.
• You need access to Talend Cloud with a valid account. You can sign up for a free trial account here: https://iam.us.cloud.talend.com/idp/trial-registration.

Create a Job with Talend Studio

This section explains how to create a new Job and publish it to Talend Cloud.

Configure Talend Studio to connect to Talend Cloud

In Studio:

1. Navigate to Window > Preferences > Talend > Integration Cloud.

2. Fill in your:

   • Account Username

   • Account Password

     

3. Select the Advanced check box. This will display the Service URL drop-down list:

    

   Choose the correct Data Center for your location:

   • US Data Center
   • EMEA Data Center
   • <Custom>, add the correct URL for your environment

Create and publish a Job to Talend Cloud

This Job shows the logic of reading the file from Azure Storage, applying a transformation, and saving the file back to Azure Storage.

Since you retrieve the file locally onto the cloud or a remote engine where the Job is executed, any files from your previous runs must be removed. Thus, the Job starts by deleting any temp files from previous runs, retrieving and then deleting the file from Azure Storage, transforming it locally on the cloud or remote engine, and then uploading it into Azure Storage again. Your Job should look like this:

When you are ready to publish your Job, right-click the Job in the Repository and click Publish to Cloud.

If you have published the Job before, you may need to increment the version number of the Job before clicking the Finish button, as shown below:

Manage your deployment

Navigate to the login page for Talend Cloud: integrationcloud.talend.com. If you prefer to use the European instance, click the Go to European instance link.

1. Type your Email and Password, then click the LOGIN button to connect.

   

2. Click the Launch link to access Talend Cloud.

   

3. Verify that your Job was published successfully.

Design an Azure Function App

This section explains how to create an Azure Function App that will call or trigger the Job when a new file arrives in Azure Storage.

Create the Azure Function App

1. Open your web browser and type: portal.azure.com

   

2. Click Create a resource.

   

3. Configure your Azure Function App settings as follows:

   Field	Value
   App name	Define the name of your Azure Function App
   Subscription	Select your Subscription
   Resource Group	Select your Resource Group
   OS	Select Windows
   Hosting Plan	Select Consumption Plan
   Location	Select your location
   Storage	Select Azure Storage
   Application Insights Location	Select your location

4. Click Create.

   

Write the Azure Function Logic

This section explains to write the logic that your Azure Function will run when it is executed.

1. Navigate to your Azure Function App and click + (Create new).

   

2. On the Blob trigger pane, select JavaScript.
   

   

3. Configure your Function settings as follows:

   Field	Value
   Language	Leave JavaScript selected
   Name	Define the name of your Function
   Path	Define the name your Blob as: talend-in/{name}
   Storage account connection	Select your Azure Storage

   

4. Click Create.
5. Click your Function.

   

6. Replace the default code with the following, remembering to change the username and password and changing the body.executable parameter to match your specific configuration:

   function callback(error, response, body) {
   context.log(JSON.stringify(body));
   context.log(response);
   }
   
   module.exports = function (context, myBlob) {
   context.log("JavaScript blob trigger function processed blob \n Name:", context.bindingData.name, "\n Blob Size:", myBlob.length, "Bytes");
   
   // Body
   var parameters = new Object();
   parameters.fileNameIn = context.bindingData.name;
   var body = new Object();
   body.executable = "5b23c0d3b1187f0ebec41ee1";
   body.parameters = parameters;
   
   var jsonString= JSON.stringify(body);
   context.log(jsonString);
    
   // Call POST
   
   // Include the request library for Node.js
   var request = require('request');
   // Basic Authentication credentials
   var username = "*****@talend.com";
   var password = "*****";
   var authenticationHeader = "Basic " + new Buffer(username + ":" + password).toString("base64");
   
   //Request
   var options ={
   method: 'POST',
   url : "https://ipaas.us.cloud.talend.com/api/v1.1/executions",
   body: jsonString,
   headers : {
   "Content-Type": "application/json",
   "Accept": "application/json",
   "Authorization" : authenticationHeader }
   };
   
   request(options, callback);
   context.log("Done!")
   
   context.done();
   };

   

Create the package.json file

Create and place the following package.json file in the shared folder of your Azure Function.

{
     "name": "sample-cloud-storage",
     "version": "0.0.1",
     "dependencies": {
          "request": "2.87.0"
      }
}

The screen below shows the package.json file as it should be configured and located:

Configure the Platform Feature

1. Navigate to your Azure Function and click Platform features.

   

2. In the Platform features section, click Console.

   

3. Type dir in the console, then press Enter.

   

4. Confirm that the package.json file is in the directory.

   

5. Type npm install request, then press Enter.

   

You should see the output in the console as shown in the screenshot; if not, consult your Azure documentation and retrace your steps to make sure the function is properly configured. Talend recommends reading through the Azure Functions documentation, as configuration steps may change in the future.

End-to-end processing of the file

The integration you just developed is event-driven: as soon as a file is placed in Azure Storage, it will trigger the Azure Function, which will trigger the Job. Review the logs to see the execution of these event tasks. You should get an output file uploaded into Azure Storage.

INPUT

Place a file in the Blob Container (talend-in) as shown below:

Check the Azure Function Log

To confirm that the Azure Function has executed, you need to check its log.

Review Talend Job Execution

Click the Log link, to the right of the execution record, and confirm the Job executed successfully. If any errors are encountered, they are shown in the Log.

The Log window opens and displays the Job results, as shown below:

OUTPUT

Go to the Blob container (talend-out) and confirm that the output file was written by the Talend Job as expected.

Conclusion

This article explains how to build an end-to-end integration based on triggers with Azure Functions and a Talend Job, and shows a way to perform serverless computing. It is transparent for the IT team, and there is no need to maintain and host a server instance. Note that Talend Cloud provides Cloud Engines which are fully managed by Talend, and also provides Remote Engines if you want to host your own execution engine. However, this is not needed, unless you have security or other constraints about the data being processed that require you to host your own.

Microsoft Azure and Talend Cloud platforms make it easy for you to perform serverless trigger and event-based processing.