Concurrent Reload Settings in Qlik Sense Enterprise

[Sonja_Bauernfeind]

The max concurrent reloads can be configured in the Qlik Sense Management Console.

1. Open the Qlik Sense Management Console
2. Navigate to Schedulers 
3. Enable the Advanced section (if not already on by default)
4. Set the required max concurrent reloads
   
   
   

    

IMPORTANT TO REMEMBER

• A reload node can run NumberOfCores - 2 
  
  This means: If your reload node has 8 CPU Cores, it can run 6 concurrent reloads at its maximum.
  
  Use the guideline (NumberOfCPUCores - 2) when performing capacity planning on the possible amount of concurrent scheduled reloads.
  
  
• The cores can be physical cores or virtual cores; any CPU cores Qlik Sense can "see".
  
  
• While a reload task typically consumes one core, this is not a hard rule. It can require multiple cores.
  
  
• Allowing more parallel reloads than CPU cores available does not improve the runtime of concurrent tasks. The reload node's resources will deplete. 
  
  This also means: With more parallel reloads than cores the probability for total starvation increases since the QIX engine is CPU hungry and other processes (for example, the operating system) might not get enough CPU cycles to perform their duties.
  
  
• Each running reload is loaded into RAM. This means that the average RAM footprint grows with more parallel reloads.
  
  
• Qlik Sense will queue scheduled reloads when system resources reach their threshold 
  
  
• To validate that environment needs proper capacity planning, go to C:\ProgramData\Qlik\Sense\Log\Scheduler\Trace\<Server>_System_Scheduler.txt, an example log shows below:
  
  

  <ServerName>_System_Scheduler.txt
  Domain\qvservice    Engine connection released. 5 of 4 used 
  Domain\qvservice    Engine connection 6 of 4 established 
  Domain\qvservice    Request for engine-connection dequeued. Total in queue: 25

Concurrent settings

Use the "Max concurrent reloads" to limit the maximal concurrent tasks can be run at same time on current node. By default, it's set to 4, which means only 4 tasks can be run at same time on this node.

When the 5th task comes in:

1. It will be queued by sequence. 
2. The queue has a time setting, which will eliminate (erase) a queued task if the timeout limit reaches. By default, it's set to 30 mins, in this case, if none of the first 4 tasks finishes in 30 minutes after the 5th task comes in, then the 5th task will be cancelled due to timeout. It has to be triggered again (manually or next scheduled time slot).
3. Once one of the 4 running tasks finishes, the 5th task will be executed on this node if it's not timed out.
4. If you set timeout to 0, this will make the queue never timeout.
5. It's suggested not to run more than 10 concurrent reloads at once, unless the schedular node has an extreme amount of resources dedicated to it.
6. If there is low memory or cores available on the node, then some tasks may kick off however take several hours to complete. Adding resources to the node running the reloads is suggested.

Multi-node deployment

On a multi-node deployment, tasks will be balanced from the manager node to any node(s) designated as workers.

It's highly advised to check if the central node configured is set to Manager and Worker or Manager. When set to Manager, it will send all reload jobs to the reload/scheduler nodes, as it should. However if a central node is set to Manager and Worker, this means the Central node will also be involved in performing reloads. This is not recommended. 

The work flow looks as follows:

1. Manager node receives a new task execution request.
2. Manager node checks the resource availability on each of the worker nodes. 
3. Manager node assigns this task to the node with the lowest number of running tasks per "Max concurrent reloads" setting.

The improvement to track the Max concurrent reloads can, if desired, be disabled. This reverts Sense to an older load balancing method that relies only on CPU usage. 

To disable the setting:

1. Open the Scheduler.exe.config, which by default is located in: C:\Program Files\Qlik\Sense\Scheduler\Scheduler.exe.config
2. Set DisableLegacyLoadBalancingBehavior setting to false
3. Restart Qlik Sense Scheduler Service
4. Repeat these actions on each node of the cluster running the Qlik Sense Scheduler Service

Example

In our example, we allow one concurrent reload, but we assume that two reloads are executed at the same time.

• If Task A is executed first, Task B is queued.
• If Task A is executed first and then fails, Task B is executed after the failure.
• After Task A's failure, Task A is queued and will execute after Task B has finished.

[jchoucq]

Hi @Sonja_Bauernfeind 

thank you very much for this article

does this rule still apply?

"For chained tasks, there is no load balancing that happens for tasks. All chained tasks are sent to the same Scheduler Node as the original task. "

Indeed, i've made a test on 2 different multi node architectures (August 2021 SR3 and May 2022 Patch 4), and in both, in chained tasks, the two tasks were executed by 2 differents scheduler nodes.

One last question, is it possible to "determine" which node is concerned in an execution of an external task, not linked to an app ?

Have a good day.

Johann

[Sonja_Bauernfeind]

Hello @jchoucq 

Let me review this for you.

All the best,
Sonja 

[Sonja_Bauernfeind]

Hello @jchoucq I've verified this with our Subject Matter experts and can confirm that the information in the article was outdated. 

Thank you for raising this!

The outdated information was removed and the article cleaned up a little.

All the best,
Sonja 

[Kunal]

@Sonja_Bauernfeind Just to confirm, the latest Qlik Sense Enterprise on Windows  version (say May 2023) does not have this limitation with multiple scheduler nodes?

"For chained tasks, there is no load balancing that happens for tasks. All chained tasks are sent to the same Scheduler Node as the original task. "

Thanks

[korsikov]

we have a client with dedicated reload node. 28 vCPU and 26 concurent reload task set.

When we try to start  reload task with 25 similar app we see that at
fact only 10 tasks works, all another wait in "running" status w/o any action. what we can to do in this situation?

[Sonja_Bauernfeind]

Hello @korsikov 

The max concurrent cannot be matched 100% to your cores. Other factors will come into play, such as how some reloads consume more cores and/or will consume resources which signal the Qlik Sense server to begin queueing.

In addition, note:

It's suggested not to run more than 10 concurrent reloads at once, unless the schedular node has an extreme amount of resources dedicated to it.

You may have reached the point where a singular reload node will not be sufficient to carry out the tasks you need, or you will need to review overall performance and reload behaviour.

To investigate why your reloads are queued, we recommend monitoring resources as well as reviewing the log files to locate the reason why a job was queued. If you need more direct assistance with this, please post about your requirement and challenge in the appropriate forum (Deployment & Management), or, if you require direct involvement with Qlik, our consulting services can be contacted to assist you with sizing questions.

All the best,
Sonja 

[korsikov]

Hi Sonja,

Thank you for fast responce.

there is come kind of hardcoded limitation or it can customize it? QlikView has Heap size limit, but in Qlik Sense it's working in one Engine.exe process.

[BTIZAG_OA]

Hello @Sonja_Bauernfeind  i have a question for this,

According to 

1. Manager node assigns this task to the node with the lowest number of running tasks per "Max concurrent reloads" setting.

When working on multinode-deployment and one of server becomes unresponsive for any reason (high cpu usage or memory leak etc.) all tasks that need to start are hangs at trigger status. Could this be caused by the setting above? Because of the server(any of workernode) becomes unresponsive, central node might not get answer of current running task number from that node, and might waiting response from it to distribute task for available node. Untill central node gets answer from unresponsive node, it keeps waiting tasks in trigger status. How can we overcome this? Its making a interruption on tasks in our site, untill you restart the slave tasknode that becomes unresponsive. 

[christianborg]

Good morning @Sonja_Bauernfeind in which cases do you recommend to disable the Max concurrent reloads config for schedulers and by default use the older load balancing method that relies only on CPU usage ?

Example does it make sense to use the older balancing rule when the task performance (due to different amounts of data loaded) are quite different from on another i.e. one task can take 5 minutes and another task 40 min ?

[rolandvanparidon]

When you mention cores, are these virtual or physical cores?