How to get started with the Amazon Lambda connector using Qlik Application Automation

[Jayarams]

The Amazon Lambda connector allows Qlik Application Automation in Qlik Cloud Services to easily launch Lambda functions, which simplifies data-driven automation.

Amazon Lambda runs code without maintaining servers, is triggered by events, and is charged solely for the time it takes.

This article explains how the Amazon Lambda connector in Qlik Application Automation can be used within Qlik Cloud.

Content:

• Authentication
• AWS Setup
• Qlik Cloud
• Available blocks
• Use Cases
• Writeback to the DB restricted to the outside world
• Extract and transform data within pipelines
• Migrating Apps between tenants
• Executing complex custom code for data manipulations
• Limitations

Authentication

AWS Setup

By default, users and roles don't have permission to create or modify Lambda resources. They cannot perform tasks using the AWS Management Console, AWS Command Line Interface (AWS CLI), or AWS API. To grant users permission to perform actions on the resources that they need, an IAM administrator can create IAM policies. The administrator can then add the IAM policies to roles, and users can assume the roles.

To learn how to create an IAM identity-based policy by using these sample JSON policy documents, see Creating IAM policies in the IAM User Guide.

1. Open the AWS console
2. Navigate to Identity and Access Management (IAM) 
3. Create a new policy with basic access to run the Lambda function.
   1. Click "Create policy."
   2. Select "JSON" as the policy language.
   3. Enter a policy name (e.g., "MyOrderAPIPolicy").
   4. In the policy editor, paste your policy document in JSON format. This document defines the resources, actions, and conditions that determine the user's permissions. for example:
      

      {
      	"Version": "2012-10-17",
      	"Statement": [
      		{
      			"Sid": "InvokeFunctions",
      			"Effect": "Allow",
      			"Action": [
      				"lambda:InvokeFunction",
      				"lambda:InvokeAsync"
      			],
      			"Resource": "*"
      		},
      		{
      			"Sid": "ListFunctions",
      			"Effect": "Allow",
      			"Action": "lambda:ListFunctions",
      			"Resource": "*"
      		}
      	]
      }

   5. Review and validate your policy using the JSON validator.
   6. Click "Create policy." 
4. Navigate back to Identity and Access Management (IAM) 
5. Click Create User and enter the new username. 
   
   Optionally, you can also give separate console access to the user.
   

   
   

6. In the Create User editor, you can select one of the three options.
   
   For our example, let’s select the highlighted option Attach policies directly.
   
   
   
   
7. Scroll down. In the Permissions Policies text box, type and search for the policy created earlier. 
8. (Optional) Expand Set permissions boundary - optional and set Use permissions boundary to control the maximum permissions 
   
   Review the on-screen text for details.
   
   
    
9. Review and create user.
10. Navigate to the Security Credentials tab.
11. Click Create Access Key.
    
    
    
    
12. For Use Case, click Third-party service and follow the recommendations. A description tag is optional.
13. Click Create Access Key
14. Copy the Access Key and Secret Access Key values.
    
    Store them safely. The Secret Access Key will only be shown once. See Access key best practices onscreen. 
    
    
    

Qlik Cloud

1. Open Qlik Cloud
2. Navigate to My Automations
   
   
   
   
3. Switch to the Connections tab
   
   
   
   
4. Click Add new connection.
5. Search for Amazon Lambda and click Add.
   
   
   
6. Provide the access key and secret access key and specify where your AWS region is located.
   
   
   

Available blocks

The Amazon Lambda connector consists of the following blocks:

• Invoke Function

Use Cases

The Amazon Lambda connectors allow calling Lambda functions, programmable for diverse tasks including:

• Writeback to the DB restricted to the outside world
• Extract and transform data within pipelines
• Migrating Apps between tenants
• Executing complex custom code for data manipulations

Writeback to the DB restricted to the outside world

The Amazon Lambda connector allows you to write back App data to the source database. If clients do not have access to their firm's database, the company may offer an Amazon Lambda function. We can utilize the Amazon Lambda connector to call this function, which will then update the database.

Refer to How to build a write back solution with native Qlik Sense components and Qlik Application Automation for instructions on how to configure the writeback. In the automation, instead of using the JIRA connector.

Use the Amazon Lambda connector's Invoke Function block, as illustrated:

Extract and transform data within pipelines

1. In the data pipeline, there can be scenarios similar to the one above where we need to extract the data from the database and use the Lambda function to transform it.
2. The client can create a Lambda function to extract and manipulate data from the source in situations where we don't have a Qlik Application Automation connector.
   
   With the aforementioned functionality, we can use Automation to call the Lambda function block.
   
   The data may be stored on Amazon S3 after it has been extracted and transformed. The Do Reload block can be used to run the Load script and obtain data from Amazon S3.
   
   
   

Migrating Apps between tenants

Using Github and Github Actions, see Move hydrated apps between tenants with third-party tools on how to migrate apps between tenants. Lambda functions allow us to accomplish comparable functionality.

Replace the "Github" blocks in the automation detailed in the aforementioned post with the "Invoke Function" block.

Include the following logic in the Lambda function:

1. Retrieve OAuth token for source tenant.
2. Run export command on source tenant.
3. Download the exported app from the source tenant
4. Retrieve OAuth token for target tenant.
5. Upload app to target tenant.
6. Import app to shared space.
7. Send callback to automation.
   
   

Executing complex custom code for data manipulations

External libraries are not supported by automation; however, you can use the Lambda function if a sophisticated transformation needs to be performed during the automation process and external libraries are needed. 

You can adhere to the following in Automation:

• You can transmit data straight through the "Invoke Function" block in the payload parameter for modest amounts of data.
  
  
• For larger data,
  
  
  1. Store the Data in Amazon S3.
  2. Trigger Lambda function using the "Invoke Function" block.
  3. Get Data from the Amazon S3.

Limitations

• While writing this article, the "Invoke Function" API allows File as input; however, sending data through File is not supported in Automation. Alternatively, upload the file to another platform (such as Amazon S3) and access it directly in Lambda Function.
• Larger datasets cannot be transmitted directly in the "Invoke Function" block; instead, we advise storing the data on other platforms, such as Amazon S3, and then utilizing the Lambda Function to access it.
• Because it is a POST API and not a GET request with pagination, there is a limit on the amount of data that can be received in the "Invoke Function" block following the successful execution of the Lambda function. We advise using alternative platforms, such as Amazon S3, to store the data and access it later using the Amazon S3 connector.
• "Invoke Function" blocks theoretically support the asynchronous function. Because the Amazon SNS connector does not presently allow Lambda function webhook events. Users may initiate trigger automation from within the asynchronous function to finish the remaining steps in the automation, depending on the Lambda function's current state of execution.
• Make sure the Synchronous function doesn't take longer than 55 seconds to execute, as this will trigger a timeout error in the "Invoke Function" block. Use the asynchronous function if the function takes longer. See Qlik Application Automation limitations for more information on limitations.