Automate file processing from Google Storage with Talend Cloud

TalendSolutionExpert — Tue, 23 Jan 2024 02:35:30 GMT

Overview

Automate file processing from cloud storage is one of the most common use cases. This article explains how to integrate Google Storage with Talend Cloud for this case.

Prerequisites

Talend Cloud account: You can sign up for a free trial account here, https://iam.us.cloud.talend.com/idp/trial-registration
Google Cloud account
Basic JavaScript knowledge

Scenario

A file is pushed to a Google Storage bucket.
A trigger calls a Cloud function.
The Cloud function calls a Talend Flow.
The Talend Flow retrieves the file to process it based on the parameters sent by the Cloud function.

Google Cloud Storage

Google Cloud Storage is an object storage system on the Google Cloud Platform (GCP), if you are familiar with Amazon, the equivalent is S3.

Creating a Bucket

Log in to your Google Cloud account, then go to the Storage page.
Click Create Bucket.
Configure your bucket.
- Name: Name your bucket.
- Default storage class: Define the type of storage:
  - Multi-Regional: This option replicates your bucket across multiple regions in an area (for example, Europe)
  - Regional: This option replicates your bucket across zones in a specific region
  - Nearline and Coldline: These two options are more of an archive option
- Location: Specify the location of your bucket.
- Labels: Set the metadata for your bucket.
- Encryption: Select the type of encryption you want to use.
Once your bucket is created, add a folder to classify your incoming files.
Create an access key for your application. In Google Cloud Storage, navigate to the Settings section, and click Create a new key.
Retrieve and secure the keys in a safe place.

Creating a Talend Job

Create a Talend Job to retrieve a file from Google Cloud Storage and display data in the console. Of course in a real life example, a Job is more complex. The demo Job is available in the GSRead.zip file attached to this article.

The Job is composed of the following steps:

Create a connection to Google Cloud Storage.
Retrieve the file from the Bucket.
Read the file and display to the console.
Close the connection to Google Cloud Storage.
Configure the context variables:
1. tempFolder: the folder used to store temporary data, on Talend Cloud Engine, it is /tmp.
2. fileKey: the file name from the GCP Bucket. It is composed of the folder and file name.
3. fileBucket: the Google bucket you created.
4. gcpAccessKey: the Google Storage access key you created.
5. gcpSecretKey: the Google Storage secret key you created.
Configure the tGSConnection component:
Configure the tGSGet component:

This component uses a split because the fileKey looks like cloud-function/sample.csv and you only need the sample.csv.
Configure the tFileInputRaw component:
Now your Job is configured. You can test it by adding valid parameters to the context.

Publishing a Job to Talend Cloud

For more information on publishing a Job to Talend Cloud, see Connecting Talend Studio to Talend Integration Cloud

Right-click your Job, and select Publish to Cloud.
Configure your export as needed, then click Finish.
Once the Job is published, click Open Job Flow.

If you are logged in Talend Cloud, you should see your configuration.
Configure the Job as appropriate.

Google Cloud Function

A Google Cloud Function allows you to implement the following logic:

Each time you create a file in your bucket, the function will call your Talend Job.

Google Cloud functions are written in JavaScript. The function and package code are available in the gcp-cloud-function.zip file attached to this article.

Find your Flow Id. In Talend Cloud, select your flow and check the Flow Id:
In your Google Cloud Platform console, go to Cloud Functions.

Create a function.

Name: Give your function a name.
Memory allocated: Select the memory needed for your function.
Trigger: In your case, use the Cloud Storage bucket.
Event Type: For each file Finalized/Created.
Bucket: The bucket you created.

You can use the inline editor to create the function.

/**
 * Triggered from a message on a Cloud Storage bucket.
 *
 * @param {!Object} event The Cloud Functions event.
 * @param {!Function} The callback function.
 */

function responseCall(error, response, body) {
    console.log(JSON.stringify(body));
    console.log(response);   
}


exports.processFile = (event, callback) => {
console.log('Processing file: ' + event.data.name);

// Body 
var parameters = new Object();
parameters.fileBucket = event.data.bucket;
parameters.fileKey  = event.data.name;
var body = new Object();
body.executable = "<Talend Flow ID>";
body.parameters = parameters;

var jsonString= JSON.stringify(body);
console.log(jsonString);

// Call Executions
// Include the request library for Node.js   
var request = require('request');
//  Basic Authentication credentials   
var username = "<Talend Cloud user>"; 
var password = "<Talend Cloud Password>";
var authenticationHeader = "Basic " + new Buffer(username + ":" + password).toString("base64");

//Request
var options ={
     method: 'POST',
     url : "https://ipaas.us.cloud.talend.com/api/v1.1/executions",
     body: jsonString,
     headers : { 
         "Content-Type": "application/json",
         "Accept": "application/json",
         "Authorization" : authenticationHeader }  
  };  

request(options, responseCall);
console.log("Done!")

callback();
};

This function is rather simple:

Create a JSON body to send to Talend Cloud, as shown below:

{
  "executable": "57f64991e4b0b689a64feed0",
  "parameters": {
    "fileKey": "cloud-function/sample.csv",
    "fileBucket": "mgainhao-demo"
  }
}

You can test your API on the API documentation page, Talend Cloud API-Executions, of Talend Cloud.
1. Add basic authentication for connection to Talend Cloud.
2. Create a request.
Because you are using the module request, you need to update the package.json to add dependencies.
Click Create. The function is created.
You can access the dashboard by clicking the name of the function.
Test your function. Create a new file in the Google Cloud bucket.
Your function is called.
In Talend Cloud, verify that there are new executions of your Job.
In the logs, you should be able to see the content of the file in the tLogRow_1 section.

Conclusion

Using Talend Cloud to process a file from cloud storage is easier with the ability to call Talend API within a Cloud Function.

article Automate file processing from Google Storage with Talend Cloud in Official Support Articles