Tasks queued despite enough resources on Qlik reload nodes

[ArifShah]

Hi,

We have three schedular nodes (centeral and two  slave nodes). They are quite big machines with 500 gb ram and 20 core processor. I have set the concurrent tasks to 6 on each node

I tried to run the operation monitor and it is being queued. Currently there are only 6 tasks running on centeral node 1 task running on one slave node. The other slave node is not running anything

I can see that the QMC is sending the operation monitor reload to the central node despite it running 6 tasks already. One of the slave nodes is running only one task while the second slave node is not loading anything

Can someone please help me understand what is wrong with the load balancer. Why is it not sending these tasks to the slave nodes. 

P.S: Both the slave nodes are fine and it has run many tasks during the nighly data reloads

Shah

[ArifShah]

I have investigated a little more. Monitoring apps are supposed to be run only on the central node. Since central node is currently busy, the tasks are being queued

The question still remains the same though. How come central node is running 6 tasks, one slave node is running only one task while the other slave node is not running anything

[gandalfgray]

Yes, we are having this situation also.

The task balancing is quite rudimentary.

You can read the details in this knowledge article: https://support.qlik.com/articles/000008047

Quoting from the doc:

1. Master node receives a new task execution request.
2. Master node checks the resource availability on each of the slave nodes. 
3. Master node assigns this task to the node which is least busy at the moment of triggering*.

*The balance check only involves CPU occupation and memory usage but NOT each Scheduler node queue for now. So even if the queue on a certain node has free space but the system usage is high (such as CPU 95% used, 2 tasks running), the new coming in task will be assigned to another fully queued node but with lower system usage (let's say CPU 60% used, but 4 tasks running)

[gaidamichal]

Hello,

This algorithm has been changed in newest patches. Exert from Release notes:

And this will be further improved I believe as it's not perfect now ;).

Regardless, operations monitor apps are by default load balanced on the central node. Check if the load balancing rules you have for apps puts the operations monitor on rim nodes as well. App can only be reloaded on a node that it's balanced to.