I have just over 60 apps which need to be reloaded via the Scheduler as regularly as possible. This number will continue to grow so I need some tips on how to manage task scheduling.
At the moment most of the apps run on an hourly basis. We have created chains to link the apps which depend on same QVD files or are linked in some other way. However, at the moment, we are restricted to running 4 concurrent tasks. We are expecting some issues which something leads to failed tasks and server issues if we don't keep an eye on the scheduler.
If you have any tips on how to create an efficient schedule or set-up chains in a good way please let me know!
Thanks in advance!
P.S. I am using Qlik Sense Enterprise on a private cloud set-up
The only tip I can offer here is to use something like the FileTime() function to check whether or not your QVDs have updated and then stopping the load if they haven't using exit script. If you do that, you can schedule more often than once per hour but you won't use up any resources for the actual reload unless the underlying QVD has changed.
The concurrent tasks is limited by the number of cores on your server, so increasing the cores will allow you to increase from 4. Make your load scripts as fast/efficient as possible, by limiting the number of fields loaded and calculations performed. Binary loads might help too. Be aware of the "max retries" property; it might be better to leave at 0 so you don't risk overlapping tasks.
I like this idea! However, if I would use exit script then the data will not be saved. I have looked into partial reloads (as it is suppose to save data before starting script) but am unsure how to use especially via the scheduler.
In the meantime, I am PoC-ing the concept where I create another app & tasks with the sole purpose of checking if the QVD has been updated. If this task passed (i.e. QVD has updated) then it will kick off my original app to reload. If not, then it will fail avoiding the redundant reload of the original app.
Indeed, exiting directly will not save the previous data (or rather, i twill save the new blank data). The quick-and-dirty workaround for this, if you're so inclined, is to insert any statement that would make the script fail (and as a result, not save). However, if you want to avoid that, then you are indeed correct in using a second app to test the QVD's update status.
My logic for using the separate is that I want to be able to quickly identify the tasks on the scheduler UI which I expect to fail from those which I don't (i.e. from a genuine error).
