Concurrent Reload Settings in Qlik Sense Enterprise
The max concurrent reloads can be configured in the Qlik Sense Management Console.
Open the Qlik Sense Management Console
Navigate to Schedulers
Enable the Advanced section (if not already on by default)
Set the required max concurrent reloads
IMPORTANT TO REMEMBER
A reload node could run NumberOfCores - 2
If for example, the reload node has 8 CPU Cores then it could possibly run 6 concurrent reloads at its maximum.
A reload task typically consumes at least one core, sometimes more
With more concurrent tasks running than there are number of cores, the total time to complete a number of tasks that is larger than the number of cores does not improve with more parallel tasks running due to starvation of the server
Each reload running is loaded into RAM. This means that the average RAM footprint grows with more parallel reloads
With more parallel reloads than cores the probability for total starvation increases since the QIX engine is CPU hungry and other processes (for example, the operating system) might not get enough CPU cycles to perform their duties
Use the guideline, NumberOfCPUCores - 2 in performing capacity planning on the possible amount of concurrent scheduled reloads.
Scheduled reloads are queued, however, if not enough resources in the system
To validate that environment needs proper capacity planning, go to C:\ProgramData\Qlik\Sense\Log\Scheduler\Trace\<Server>_System_Scheduler.txt, an example log shows below:
<ServerName>_System_Scheduler.txt Domain\qvservice Engine connection released. 5 of 4 used Domain\qvservice Engine connection 6 of 4 established Domain\qvservice Request for engine-connection dequeued. Total in queue: 25
Use the "Max concurrent reloads" to limit the maximal concurrent tasks can be run at same time on current node. By default, it's set to 4, which means only 4 tasks can be run at same time on this node. When the 5th task comes in:
It will be queued by sequence.
The queue has a time setting, which will eliminate (erase) a queued task if the timeout limit reaches. By default, it's set to 30 mins, in this case, if none of the first 4 tasks finishes in 30 minutes after the 5th task comes in, then the 5th task will be cancelled due to timeout. It has to be triggered again (manually or next scheduled time slot).
Once one of the 4 running tasks finishes, the 5th task will be executed on this node if it's not timed out.
If you set timeout to 0, this will make the queue never timeout.
It's suggested not to run more than 10 concurrent reloads at once, unless the schedular node has an extreme amount of resources dedicated to it.
If there is low memory or cores available on the node, then some tasks may kick off however take several hours to complete. Adding resources to the node running the reloads is suggested.
On a multi-node deployment, tasks will be balanced from the master node to any node(s) designated as slaves.
It's highly advised to check if the central node in the QMC in the Schedulers section is set to "Master and Slave" or "Master". When set to "Master", it will send all reload jobs to the reload/scheduler nodes, as it should. However if a central node is set to Master and Slave, this means the Central node will also be involved in performing reloads, and is not suggested.
When there's a 3 nodes deployment (one master and two slaves) and each node is set to "Max concurrent reloads" = 4 and "Engine timeout" = 30:
Master node receives a new task execution request.
Master node checks the resource availability on each of the slave nodes.
Master node assigns this task to the node which is least busy at the moment of triggering*.
*The balance check only involves CPU occupation and memory usage but NOT each Scheduler node queuefor now. So even if the queue on a certain node has free space but the system usage is high (such as CPU 95% used, 2 tasks running), the new coming in task will be assigned to another fully queued node but with lower system usage (let's say CPU 60% used, but 4 tasks running)
For chained tasks, there is no load balancing that happens for tasks. All chained tasks are sent to the same Scheduler Node as the original task. This original assignment is based on overall availability based on metrics. This means having a large number of Chained Tasks is going to send all of those to the same node. This should be avoided with Chained Tasks being used for 'dependencies' for the current job.