Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
When Bobby McFerrin wrote his little song “Don't Worry, Be Happy” in 1988, he may not have been thinking about Application Programming Interfaces (APIs), but APIs can certainly help data operations or DataOps become worry-free.
An API is a handy method to access certain operations, features, or data within an application. They are useful in automating repetitive and time-consuming tasks. This leads in nicely with DataOps, which is essentially a set of best practices using processes and technologies to improve quality, speed, and collaboration on data with automation.
At Qlik, automation lies at the heart of our solutions across our entire end-to-end portfolio, from Data Integration to Data Analytics, which not only delivers actionable insights but also makes it possible to automate actions.
For example, let’s examine our cloud-native solutions, Qlik Cloud Data Integration and Qlik Application Automation.
Qlik Cloud Data Integration offers a pro-code/low-code approach to help you build automated replication pipelines and data transformation pipelines that deliver the freshest data. Our best-in-class change data capture (CDC) technology automatically delivers data whenever changes occur at the source.
Qlik Application Automation is our low-code solution for building integrated workflows between the most popular SaaS applications. You can choose from a host of blocks that automatically respond to business events and trigger from a myriad of SaaS applications and cloud-based platforms. This saves you from having to write hundreds of complex lines of code for all the APIs that you may want to use.
The blocks are all built on the public APIs for Qlik Cloud which are also published on the Qlik developer portal.
- Get Data Task Details
- List Projects on a given data space
- List data tasks within a given project
- Start and Stop Tasks
- Start data tasks and wait for completion (released mid May 2024)
As and when we release further APIs for Qlik Cloud, we will continue to improve and add to these blocks over time to help build out further use cases. A point in case is the 'Start data tasks and wait for completion' block, which was recently made available.
These automation blocks provide an additional layer of control and flexibility over your data integration payloads, allowing you to construct more complex orchestration workflows on top of your data pipelines.
For example, you can use them to orchestrate jobs you have running in Qlik Cloud and Talend Cloud jobs, Databricks notebooks or even DBT cloud.
There are many use cases for Qlik App Automation in Qlik Cloud Data Integration; here are just a few examples.
Data Gateway Management
You could automate gateway updates, which would stop a CDC task, update your data movement gateway and then restart the CDC task after the gateway has been successfully updated.
Automating Support Ticket Processes
Switch from reactive to proactive, for example, in the event a task fails, such as a network failure or a source dataset not being available. You could set up an automation workflow that would send a ServiceNow message should a failure occur.
Optimize Workflows with Parallel Tasks
This is the latest capability we have added for Qlik Cloud Data Integration: the ability to start several tasks in parallel and wait for all of the tasks to be completed before the workflow continues to the next step.
To bring this last example to life, imagine we are an online retail fashion store that receives orders from our customers every day.
We’ve got some hot shoes in stock, and we are receiving orders hand over feet 😊 so every day we are getting more and more orders and in parallel to that we need to collect their customer details such as name, address credit status and so on.
In our system of records we have two different operational databases, one for sales, one for customer data.
We have created a project in Qlik Cloud Data Integration, which has a series of tasks set up for each of these databases, and the project is set to run on a fixed scheduled basis. (It's actually the project in the first screenshot)
The way it's currently handled in Qlik Cloud Data Integration by default is that any time either one of these storage tasks is completed, the data transformation process will then start to run, however it is possible that both of our operational databases are updated at different times in the day, at different frequencies, in addition the sales order table is a much larger dataset than the other, resulting in the storage task for that table taking much longer to complete than the customer dataset.
This is where Qlik Application Automation comes into play. It helps us get finer control over that orchestration and ensure that both tasks are completed before the transform can be run.
We can now set up a workflow in Qlik Application Automation and use the new block “Start Data Tasks And Wait For Completion,” which will update both of our storage tasks to ensure we have fresh data from both sales and customer data sources for our transformation tasks to work on.
All automations are set up with the 'Start' block and can be run on a schedule, as required. In this example, we have made a very simple automation that starts all storage tasks at the same time. The first step is to list out the data tasks in the project that we want to run. We can then utilize the best practice of creating variables for each of the storage tasks that we want the workflow to orchestrate, to cater for new storage tasks that may come up in the future.
The tasks are then used as inputs to the 'Start Data Tasks and Wait for Completion' blocks, which poll the tasks until they are finished and return their status results, such as failed, cancelled, or complete.
This status is then used in the 'Condition' block, which, depending on the status returned, will either start the TRANSFORM task if both STORAGE tasks are completed successfully or send a message via Slack to the DEV team notifying them to investigate.
There are also plenty of monitor capabilities within Qlik Cloud to help check status, like the 'History view' in Qlik Application Automation, which shows how your automations are operating.
We will add a new template to Qlik Application Automation that reproduces the above use case. It can be customized to your needs and helps you get started in minutes.
Check out this support article for the new Template, which is coming soon: "Start Data Integration tasks and wait for completion"
This is just one of many use cases for Qlik Application Automation, which brings an extra level of control to your Qlik Cloud Data Integration workflows and adds huge value to our customers.
If you have a Qlik Cloud Data Integration tenant, you can start using Qlik Application Automation today.
We would love to hear your use cases and what value it brings to your organization.