Search or browse our knowledge base to find answers to your questions ranging from account questions to troubleshooting error messages. The content is curated and updated by our global Support team
This article explains how a reload task chain can be built that will run longer than the maximum automation run limit of 4 hours.
This is an advanced article, if you’re new to automations or task chaining, I suggest you first read this basic article.
The maximum run time of an automation is 4 hours. This can be a problem when you want to build a task chain of multiple reloads of big apps that could take longer than that. To solve this, we'll build a system of two automations that run together:
Automation 1: Task Chain Initiator, this automation will run once and start the task chain. It will trigger the first app reloads.
Automation 2: Task Chain Manager, this automation will be scheduled to run every 5 minutes. It will verify if any reloads have finished, whether they were successful and if any new reloads should be triggered.
The Task Chain Manager automation will need to know which reloads were triggered by the Task Chain Initiator and will need a way to store which reloads it already processed in previous runs. In other words: these automations need some sort of state to track the progress of the reload task chain. We’ll use Google Sheets for this, but of course, feel free to use any other cloud storage tool.
The state will be stored in one file and divided over 4 sheets. Go ahead and create a new file in Google Sheets.
Sheet 1: Apps for initial reload, this sheet contains the ids and names of the apps that should be reloaded in the first automation. They will be executed by the Reload Initializer automation and run in parallel. Add a new sheet to the file you created, give it the name “Apps for initial reload” and add each app's id and name as in the below example. The order of the apps in the sheet is ignored.
Sheet 2: Queued reloads, this sheet will keep track of which apps were already triggered for reloading.
Add the sheet to the file and name it “Queued reloads” but don’t fill in any fields, the content of the sheet is autogenerated by the automations.
Sheet 3: Finished reloads, this sheet keeps track of all reloads that have finished, it contains their id, name, the status of the reload, and the reload log. Add the sheet to the file and name it “Finished reloads” but don’t fill in any fields, the content of the sheet is autogenerated by the Task Chain Manager automation.
Sheet 4: Chained reloads, this sheet dictates which apps should be reloaded after certain apps finished reloading. Add a new sheet to the file and name it “Chained reloads” specify the order in which apps should be executed.
The order of the rows in the sheet is ignored, the sheet is processed as follows:
If a reload has finished for an app from column “First_id”, then trigger a new reload for the app that’s specified in the column “Next_id” on the same row.
Add this sheet to the file and specify the apps as in the example below:
This automation will only run once at the beginning of the full task chain process, it will execute the following steps:
Note: an export of this automation is attached to this article as a JSON file named "Task chain initiator.json".
This automation will run on a schedule and will be the engine of our task chain process. Feel free to adjust the schedule according to your needs. On each run of the automation, the following steps will be executed:
Note: an export of this automation is attached to this article as a JSON file named "Task chain manager.json".
The reloads in the Finished reloads sheet will look like this:
When scheduled every 5 minutes, the "Task Chain Manager" automation will generate 288 daily runs. This bonus paragraph explains how the number of total runs can be reduced. Instead of having an automation that has many executions that run short, we'll build an automation that has only a few executions that run for a longer time.
The information in this article is provided as-is and to be used at your own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.
How to task chaining with Qlik Application Automation
How to import and export automations
Hi Emile, Thanks for this article ! Very Helpfull.
I have some questions and comments:
1) The case block will have a list of the apps that are in the queued sheet and not in the finished sheet in order to see if they are pending to reload.. and then will see if the status of those apps is succeeded or failed. What what happens if the app is still reloading ? It won't have a failed or succeded status... I don't understand very well this part. It will go again throuh the loop until the app has any of those 2 status?
2) The original automation of the task chain manager will be running 288 times a day which is 8,928 times a month, which is way more than the 5,000 limit with the original SaaS subscription. So definitely not scalable..
With the Bonus, there is a loop that will be running for 30 times (I assume you put 30 as a random number), The purpose is to reload 30 apps, and then end the loop and start again reloading more apps, so it never reachs the 1 hour limit ? Assuming that 30 apps take less than one hour..
Anyway, if all the apps are reloaded, when the automation see that there are none records that exist in queued sheet but doesn't exist in finished sheet, That means that the task chain has finished, but I don't want that after one minute the automation starts again, it will be running the loop (30 times), without doing anything because all the apps are reloaded, and then again, and again... So I will take less than 288 runs a day, but it will take quite a lot... and I don't want to run out of the 5,000 jobs (automation runs) that I have, especially beacuse I want to use more automations for other purposes.
Maybe there is a way that when all the apps of the task chain are reloaded, the automation doesn't run again.. I will also explore but in order to make this scalable, I think is important to figure out how to do it without have the automation running all the time, every day.
3) What's the purpose of the sleep block ? I didn't understand it.
4) Would be great to trigger the Task Chain Manager Automation, from the Task Chain Initiator Automation, maybe when the apps from the task chain initiator are completely reloaded or when they start reloading, call the Task Chain Manager Automation.
5 ) Do you know if Qlik is planning to include something like the tasks chain in the QMC of Qlik Sense Enterprise, in Qlik Sense SaaS? It would be way easier to reload chained apps, which is very common when you are creating BI proyects, with a lot of big apps, with tons of information, and different stages of development (apps to load data from DB and storing tables as qvds, other apps that read those qvds and transform data, other apps to join/concatenate/applymaps, etc to create the data model, and at the end the final app with visualizations that only read qvds already transformed in order to have a better performance).
Thanks again for the thorough feedback, this is much appreciated!
I did my best to answer everything below:
Indeed, if the app is reloading, the Case block will ignore the app. In the next execution of the loop, the case block will check that app's reload status again. This will repeat until the status is either FAILED or SUCCEEDED.
(and 3) No, the purpose of the loop block is not to reload an app 30 times, it is to check 30 times if any app has completed reloading and to take action accordingly.
That automation does the following in a loop:
4. Indeed, that’s a possibility but then the Task Chain Manager Automation could only run for one hour max.
5. I have no news on the roadmap for the QMC, is this already requested through Ideation?
FYI, we've increased the max run time of automations from 1 hour to 4 hours.
is there any chance Qlik will increase this so that reloads can run longer than 4 hours?
This is a lot of work to do something that worked flawlessly in enterprise without any extra work....
I'd recommend posting this in our ideas forum. Previously, it was set to one hour, and an idea was logged to provide feedback to our product team (which is why it's now at four hours). Let us know the business requirements for longer-running automations through an idea, and our product team will review it.
All the best,