The integration between AWS S3 and Lambda is very common in the Amazon world, and many examples include executing the Lambda function upon S3 file arrival.

This article explains how to use AWS to execute a Talend Cloud Job.

Content:

• Overview
• Prerequisites
• Configuring Amazon S3
• Creating a Bucket
• Creating a Policy and a User
• Creating a Policy
• Giving a User Programmatic Access
• Creating and Publishing a Talend Job in Talend Cloud
• Creating a Job
• Publishing the Job in Talend Cloud
• AWS Lambda

Overview

1. A file is uploaded to an S3 bucket.

2. S3 triggers the Lambda function.

3. The Lambda function calls a Talend Flow.

4. The Talend Flow retrieves the S3 file to process it based on the parameters sent by the Lambda function.

Prerequisites

• A valid AWS account with access to the following:

• S3

• Lambda

• A Talend Cloud account or trial account

Configuring Amazon S3

Creating a Bucket

1. Sign in to your Amazon account and open the Amazon S3 page.

2. Click Create bucket.

   

    

3. Configure the following fields, then click Next:

   1. Bucket name: The bucket name must be unique across all AWS.

   2. Region: Select the region where your bucket resides, in this case, Ireland.

   

4. Keep the default settings. Click Next.

   

5. Keep the default permissions. Review the configuration, then click Create bucket.

   

Creating a Policy and a User

When accessing S3 with a remote Job, you need to give a user programmatic access (no access to the S3 console) and you need to create a policy limiting the user/application access to only this bucket.

Creating a Policy

1. In the AWS console, navigate to the IAM (Identity and Access Management) page.

2. Navigate to the Policies section, then click Create policy.

   

   

3. Using the visual editor, configure the policy as shown below:

   1. Service: Select S3.

   2. Action: Select GetObject and GetObjectVersion. GetObject allows you to retrieve the file in your Job.

   3. Resources: Point to your S3 bucket using ARN (Amazon Resource Name). The * at the end means all objects in your S3 bucket.

   4. Request conditions: Leave as is.

      

   5. Click JSON to see your policy in a JSON format, as shown below:

      

4. Review your policy, then click Create policy.

   

Giving a User Programmatic Access

1. In IAM, navigate to the Users section, then click Add user.

   

2. Select a User name, select the Programmatic access check box, then click Next: Permissions.

   

3. Select Attach existing policies directly, and choose the policy you created in the previous section.

   

4. Review your settings, then click Create user.

   

5. Well done, your user is created. Do not forget to download and save the Access and Secret keys.

   

Creating and Publishing a Talend Job in Talend Cloud

In this section, you learn how to create and publish a Talend Job in Talend Cloud.

Creating a Job

Create a Job that retrieves a file from S3, and displays the data in the console. Of course, a real Job will be more complex.

In Amazon S3, upload a file to test your Job.

1. Create a folder and name it connections.

   

2. Create a file, in this example connections_012018.csv, then upload the file to the connections folder.

   

3. In Studio, create a new context group called S3Parameters, then click Next.

   

4. Configure the following parameters using the information from your S3 bucket, then click Finish:

   • parameter_accessKey: the access key used by your application to connect to Amazon S3

   • parameter_secretKey: the secret key used by your application to connect to Amazon S3

   • parameter_bucketName: the bucket name on S3

   • parameter_bucketKey: the file key—on S3, there is no folder so the path is considered the file key

   • parameter_tempFolder: the temporary folder where you will store the file for processing—on Talend Cloud Engine, it is /tmp/

     

5. Create a new Job, and name it S3Read. The Job is composed of three stages:

   1. Create an object connection to S3.
   2. Get the file from S3.
   3. Read the file.
      
      

   

6. Configure the tS3Connection component to a specific region, and the context variables for Access and Secret keys.

   

7. Configure the tS3Get component to retrieve the file based on the context parameters, and store it in the temp folder.

   

8. Configure the tFileInputDelimited component to read the file stored in the temp folder.

   

9. Test the Job locally to see if it connects and reads the file correctly.

   

10. Next, upload the Job to Talend Cloud. Navigate to Window > Preferences > Talend > Integration Cloud and configure your access to Talend Cloud.

    

11. Once a connection is established, right-click the Job and select Publish to Cloud.

    

12. Click Finish.

    

13. When the Job has finished uploading, click Open Job Flow.

    

Publishing the Job in Talend Cloud

1. In Talend Cloud, you can see the required parameters.

   

2. Update the configuration based on your own bucket, then click Save.

   

3. Select your runtime, for this example, use a cloud engine.

   

4. Because you configured with an existing file, you can test your Job by clicking Run Now.

   

   

5. You will see the content of your file in the log.

   

6. Now, test your Job using a remote call with Talend Cloud API.

   

7. Confirm that you are using v1.1 API, then click Authorize.

   

8. Log in using your Talend Cloud account credentials.

   

9. Now, find the Flow Id. In Talend Cloud > Integration Cloud > Flows > the Flow Id is in the upper left corner of your flow.

   

   

10. For this example, use the POST /executions operations.

    

11. Create a body with:

    • executable: your Flow Id

    • parameters: all context variables you want to overwrite. In this example, specify the bucket name.

    

12. Scroll down, then click Try it out!

    

13. Review the results.

    

14. Check your flow and notice that a second execution appears.

    

AWS Lambda

At this stage, you have deployed your Job to Talend Cloud and tested a call with the API. Now, create the Lambda function, which is triggered through the API for each new file and call in your Job.

1. Connect to your AWS console, and in the Lambda section, select Create a function.

   

2. Give your function a name. Select the runtime Python 3.6. In the Role section, select Create custom role.

   

3. Create a new Role, it will create a role and a new role policy.

   

4. Review the configuration, then click Create function.

   

5. To create the trigger, select an S3 trigger on the left under Designer.

   

6. Configure the trigger with your bucket name and a prefix (in this example, the connections folder). Select Enable trigger, then click Add.

   

7. Verify that the new trigger was added.

   

8. Copy the code from the function in the lambda_function.py file attached to this article.

   

9. Configure the environment variables:

   • TCLOUD_API_ENDPOINT: URL to call the API

   • TCLOUD_USER: User that has the right to call the API

   • TCLOUD_PWD: the TCLOUD_USER password

   • TCLOUD_FLOWID: Talend Flow Id of the Job

   

10. Add tags to identify your function.

    

11. Save your function. Now you can add a new file to your folder in S3, and you will see an execution of the Lambda function.

    

12. In Talend Cloud, verify there is a third execution.

    

13. You will see the content of your file in the log.

    

     

For more information, see the AWS documentation, Using AWS Lambda with Amazon S3 page.

When trying to execute a Task, the execution fails with the following error:

Exceeded the limit of deployment attempts: you have reached the limit of Flow deployments on the engine.

Cause

You have reached the maximum number of allowable concurrent flows (running Jobs). The limit is set by license, or where the Remote Engine is being used to run the flows—your configuration.
A Cloud Engine can only run up to three flows at the same time (that is, three concurrent flows).
A Remote Engine can only run up to three flows at the same time by default.
So, when you look at your flow execution history in Talend Cloud, and you see this error message for some flows that have attempted to execute, it means that you already have three flows running. Thus, there are no open execution slots for the new flow to run. The flow that is not able to run is rescheduled and remains in the queue. When one of the currently-running Jobs finishes, the next flow in the queue is run.

If a flow is not able to be executed after several rescheduling attempts (that is, no open execution slots have opened up, or other flows were first/earlier in the queue), the flow moves into an error state and will no longer attempt to run.

Resolution

You can modify or remove the limit by following the instructions and modifying the configuration of your Remote Engine.

1. Install the Remote Engine, but do not enter the pairing key in the install wizard interface—leave this field blank. Do enter/select values for all other fields/drop-downs.
2. Modify the following files, as shown below:
   • REMOTE_ENGINE_INSTALL_DIR/etc/org.talend.ipaas.rt.deployment.agent.cfg
     
     Change this value to 0 (unlimited):
     

     max.deployed.flows=3

   • REMOTE_ENGINE_INSTALL_DIR/etc/preauthorized.key.cfg
     Fill in these three fields:
     

     remote.engine.pre.authorized.key = 
     remote.engine.name = dev_remote_engine_1
     remote.engine.description = Cool remote engine for dev 1

3. Restart the Remote Engine.
4. Wait a few minutes, and the Remote Engine shows Available in Talend Cloud.

Environment

Talend Cloud 

Be more agile, get more value from data, foster greater collaboration, and enable data users to become more effective by moving to Talend Data Fabric in the cloud.

As a Talend user, you recognize the value of data to your business. Organizations that use data and analytics to drive business strategy adapt to change quickly and develop insights that generate new value. They harvest data to improve productivity, make faster and more accurate decisions, and reduce costs. They become more innovative and competitive, discover and deploy new business models more effectively, and foster better engagement with customers, employees, and partners.

Accomplishing all this, however, is not easy. The rising pace of business and the increasing complexity of the data landscape — more data, more users, more applications, more environments (on premises, cloud, and hybrid), and more regulation — make it harder for organizations to have complete, clean, and trusted data they can rely on. No wonder that 60% of companies have unreliable data health. Data workers in some organizations spend two-thirds of their time searching and preparing data rather than using it for making decisions and running the business.

You’re already ahead of the pack, because you rely on Talend to help you find trust amidst this data chaos and deliver data that is complete, clean, uncompromised, and readily available across the organization.

You can do even more by moving to Talend in the cloud.

Download the full document from this article.