Automate S3 file processing with Talend Cloud and AWS Lambda

[TalendSolutionExpert]

The integration between AWS S3 and Lambda is very common in the Amazon world, and many examples include executing the Lambda function upon S3 file arrival.

This article explains how to use AWS to execute a Talend Cloud Job.

Content:

• Overview
• Prerequisites
• Configuring Amazon S3
• Creating a Bucket
• Creating a Policy and a User
• Creating a Policy
• Giving a User Programmatic Access
• Creating and Publishing a Talend Job in Talend Cloud
• Creating a Job
• Publishing the Job in Talend Cloud
• AWS Lambda

Overview

1. A file is uploaded to an S3 bucket.

2. S3 triggers the Lambda function.

3. The Lambda function calls a Talend Flow.

4. The Talend Flow retrieves the S3 file to process it based on the parameters sent by the Lambda function.

Prerequisites

• A valid AWS account with access to the following:

• S3

• Lambda

• A Talend Cloud account or trial account

Configuring Amazon S3

Creating a Bucket

1. Sign in to your Amazon account and open the Amazon S3 page.

2. Click Create bucket.

   

3. Configure the following fields, then click Next:

   1. Bucket name: The bucket name must be unique across all AWS.

   2. Region: Select the region where your bucket resides, in this case, Ireland.

   

4. Keep the default settings. Click Next.

   

5. Keep the default permissions. Review the configuration, then click Create bucket.

   

Creating a Policy and a User

When accessing S3 with a remote Job, you need to give a user programmatic access (no access to the S3 console) and you need to create a policy limiting the user/application access to only this bucket.

Creating a Policy

1. In the AWS console, navigate to the IAM (Identity and Access Management) page.

2. Navigate to the Policies section, then click Create policy.

   

   

3. Using the visual editor, configure the policy as shown below:

   1. Service: Select S3.

   2. Action: Select GetObject and GetObjectVersion. GetObject allows you to retrieve the file in your Job.

   3. Resources: Point to your S3 bucket using ARN (Amazon Resource Name). The * at the end means all objects in your S3 bucket.

   4. Request conditions: Leave as is.

      

   5. Click JSON to see your policy in a JSON format, as shown below:

      

4. Review your policy, then click Create policy.

   

Giving a User Programmatic Access

1. In IAM, navigate to the Users section, then click Add user.

   

2. Select a User name, select the Programmatic access check box, then click Next: Permissions.

   

3. Select Attach existing policies directly, and choose the policy you created in the previous section.

   

4. Review your settings, then click Create user.

   

5. Well done, your user is created. Do not forget to download and save the Access and Secret keys.

   

Creating and Publishing a Talend Job in Talend Cloud

In this section, you learn how to create and publish a Talend Job in Talend Cloud.

Creating a Job

Create a Job that retrieves a file from S3, and displays the data in the console. Of course, a real Job will be more complex.

In Amazon S3, upload a file to test your Job.

1. Create a folder and name it connections.

   

2. Create a file, in this example connections_012018.csv, then upload the file to the connections folder.

   

3. In Studio, create a new context group called S3Parameters, then click Next.

   

4. Configure the following parameters using the information from your S3 bucket, then click Finish:

   • parameter_accessKey: the access key used by your application to connect to Amazon S3

   • parameter_secretKey: the secret key used by your application to connect to Amazon S3

   • parameter_bucketName: the bucket name on S3

   • parameter_bucketKey: the file key—on S3, there is no folder so the path is considered the file key

   • parameter_tempFolder: the temporary folder where you will store the file for processing—on Talend Cloud Engine, it is /tmp/

     

5. Create a new Job, and name it S3Read. The Job is composed of three stages:

   1. Create an object connection to S3.
   2. Get the file from S3.
   3. Read the file.
      
      

   

6. Configure the tS3Connection component to a specific region, and the context variables for Access and Secret keys.

   

7. Configure the tS3Get component to retrieve the file based on the context parameters, and store it in the temp folder.

   

8. Configure the tFileInputDelimited component to read the file stored in the temp folder.

   

9. Test the Job locally to see if it connects and reads the file correctly.

   

10. Next, upload the Job to Talend Cloud. Navigate to Window > Preferences > Talend > Integration Cloud and configure your access to Talend Cloud.

    

11. Once a connection is established, right-click the Job and select Publish to Cloud.

    

12. Click Finish.

    

13. When the Job has finished uploading, click Open Job Flow.

    

Publishing the Job in Talend Cloud

1. In Talend Cloud, you can see the required parameters.

   

2. Update the configuration based on your own bucket, then click Save.

   

3. Select your runtime, for this example, use a cloud engine.

   

4. Because you configured with an existing file, you can test your Job by clicking Run Now.

   

   

5. You will see the content of your file in the log.

   

6. Now, test your Job using a remote call with Talend Cloud API.

   

7. Confirm that you are using v1.1 API, then click Authorize.

   

8. Log in using your Talend Cloud account credentials.

   

9. Now, find the Flow Id. In Talend Cloud > Integration Cloud > Flows > the Flow Id is in the upper left corner of your flow.

   

   

10. For this example, use the POST /executions operations.

    

11. Create a body with:

    • executable: your Flow Id

    • parameters: all context variables you want to overwrite. In this example, specify the bucket name.

    

12. Scroll down, then click Try it out!

    

13. Review the results.

    

14. Check your flow and notice that a second execution appears.

    

AWS Lambda

At this stage, you have deployed your Job to Talend Cloud and tested a call with the API. Now, create the Lambda function, which is triggered through the API for each new file and call in your Job.

1. Connect to your AWS console, and in the Lambda section, select Create a function.

   

2. Give your function a name. Select the runtime Python 3.6. In the Role section, select Create custom role.

   

3. Create a new Role, it will create a role and a new role policy.

   

4. Review the configuration, then click Create function.

   

5. To create the trigger, select an S3 trigger on the left under Designer.

   

6. Configure the trigger with your bucket name and a prefix (in this example, the connections folder). Select Enable trigger, then click Add.

   

7. Verify that the new trigger was added.

   

8. Copy the code from the function in the lambda_function.py file attached to this article.

   

9. Configure the environment variables:

   • TCLOUD_API_ENDPOINT: URL to call the API

   • TCLOUD_USER: User that has the right to call the API

   • TCLOUD_PWD: the TCLOUD_USER password

   • TCLOUD_FLOWID: Talend Flow Id of the Job

   

10. Add tags to identify your function.

    

11. Save your function. Now you can add a new file to your folder in S3, and you will see an execution of the Lambda function.

    

12. In Talend Cloud, verify there is a third execution.

    

13. You will see the content of your file in the log.

    

For more information, see the AWS documentation, Using AWS Lambda with Amazon S3 page.