Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Jul 2, 2024 4:54:32 AM
Jul 23, 2015 3:34:17 AM
The max concurrent reloads can be configured in the Qlik Sense Management Console.
<ServerName>_System_Scheduler.txt
Domain\qvservice Engine connection released. 5 of 4 used
Domain\qvservice Engine connection 6 of 4 established
Domain\qvservice Request for engine-connection dequeued. Total in queue: 25
Use the "Max concurrent reloads" to limit the maximal concurrent tasks can be run at same time on current node. By default, it's set to 4, which means only 4 tasks can be run at same time on this node.
When the 5th task comes in:
On a multi-node deployment, tasks will be balanced from the manager node to any node(s) designated as workers.
It's highly advised to check if the central node configured is set to Manager and Worker or Manager. When set to Manager, it will send all reload jobs to the reload/scheduler nodes, as it should. However if a central node is set to Manager and Worker, this means the Central node will also be involved in performing reloads. This is not recommended.
The work flow looks as follows:
The improvement to track the Max concurrent reloads can, if desired, be disabled. This reverts Sense to an older load balancing method that relies only on CPU usage.
To disable the setting:
In our example, we allow one concurrent reload, but we assume that two reloads are executed at the same time.
Hello @rolandvanparidon
Thank you for your comment!
The cores can be physical cores or virtual cores; any CPU cores Qlik Sense can see.
I have updated the article accordingly.
All the best,
Sonja
@Sonja_Bauernfeind thank you for the response!
From this article, I'm gathering that it is still recommended to take physical cores into account. Am I correct in making that assumption? (https://community.qlik.com/t5/Official-Support-Articles/Virtualization-Best-Practices-In-QlikView-An...)
Currently our r6a.2xlarge server with 16 virtual cores is struggling with 10 max concurrent tasks and is running at max cpu. As a result we have failing tasks with no error log.
I understand this is slightly more specific then the topic. So if I should address it elsewhere, I'm happy to move it.
Hi @Sonja_Bauernfeind any insights on my previous comment please ?
----------------------------------------------------------------------------------------------------------------------------------------
Good morning @Sonja_Bauernfeind in which cases do you recommend to disable the Max concurrent reloads config for schedulers and by default use the older load balancing method that relies only on CPU usage ?
Example does it make sense to use the older balancing rule when the task performance (due to different amounts of data loaded) are quite different from on another i.e. one task can take 5 minutes and another task 40 min ?
Hello @christianborg
Please post your query directly in the appropriate forum (Qlik Sense Management and Deployment), where your knowledgeable Qlik peers can assist you. If you require more involved assistance, our Professional Services can assist you.
Hello @rolandvanparidon
The article you mentioned highlights the importance of how Qlik will use all resources it can get. Meaning if you do have a virtual environment, it is important the resources are not pooled/shared.
For a more detailed answer though for your specific requirement I'd need to refer you to either our forums (Qlik Sense Management and Deployment) or to our professional services (Professional Services).
All the best,
Sonja
Hi @Sonja_Bauernfeind ,
Would you be able to dissect this bullet point a bit more?
“With more concurrent tasks running than there are number of cores, the total time to complete a number of tasks that is larger than the number of cores does not improve with more parallel tasks running due to starvation of the server.”
Am I correct in rephrasing it into:
“When more concurrent tasks are running than there are CPU cores, the total time to complete these tasks does not improve by adding more parallel tasks because the server’s resources become insufficient.”
Hello @rolandvanparidon
That is correct!
I will look into rephrasing this section (and generally review the article while I am at it).
All the best,
Sonja