Databricks - How to get started with Databricks in Qlik Application Automation

[Ionut_Dumitrascu]

This article gives an overview of the available blocks in the Databricks connector in Qlik Application Automation. It will also go over a basic example of triggering a model transition and catching the information using a webhook for alerting purposes.

Databricks is a cloud-based collaborative data science, data engineering, and data analytics platform that combines the best of data warehouses and data lakes into a lakehouse architecture.​

This connector consists of:

• CRUD support for jobs and job runs API
• CRUD support for pipelines API
• Support for model transition actions
• Support for webhooks
• other helper blocks: List clusters, List pools.

For authenticating, you will need to provide an API Token. To get an API Token: click on Settings in the lower left corner of your Databricks workspace, then click User Settings, then go to the Access Tokens tab, then click the Generate New Token button.

Let's now go over a basic example of how to trigger a model transition and how to catch this information using a webhook using the Databricks connector.

For this use case, we will need two automations. The first one will serve to trigger the model transition, and the second one will have a webhook configured for the same model, to trigger when a transition is created. We will use the Slack connector for alerting.

Steps to configure the first automation:

• Create an automation;
• From the left menu, select the Databricks connector;
  
• Use the 'Search for blocks' input to search for the 'Create Model Transition Request' block.
• Drag and drop the block into the automation and link it to the start block;
  
• On the right side of the screen, fill in the necessary information to create a transition. Use the 'do-lookup' functionality to search for a model, select the stage and specify the version.
  
• Use the 'Search for blocks' input to search for the 'Approve Model Transition Request' block, drag it into the automation and link it to the 'Create Model Transition Request' block. Fill in the information, which is similar to the other block.
  

Steps to configure the second automation:

• Create an automation;
• Click on the 'Start' block, and from the right side of the screen, click on 'Run mode' and select 'Webhook'. Choose the 'Databricks' connector and the 'On transition request created' event. Use the 'do-lookup' functionality to choose the same model as in the first automation.
  
• Save the automation. This will create the webhook that you just configured and link it to the automation.
• From the left menu, select the Slack connector;
• Use the 'Search for blocks' input to search for the 'Send message' block. Drag it into the automation and link it to the webhook block.
  
• On the right side of the screen, fill in the necessary information to send a message. Use the 'do-lookup' functionality to select a channel, and then click on the 'text' input field and map it with the 'text' field coming from the webhook.
• Save the automation.

Now, by running the first automation, you will create and approve a model transition. If you check the second automation, you will notice that it was automatically triggered and the Slack message containing the transition information was sent.

The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.