article Concurrent Reload Settings in Qlik Sense Enterprise in Official Support Articles

Concurrent Reload Settings in Qlik Sense Enterprise

Sonja_Bauernfeind — Tue, 02 Jul 2024 08:54:32 GMT

The max concurrent reloads can be configured in the Qlik Sense Management Console.

Open the Qlik Sense Management Console
Navigate to Schedulers
Enable the Advanced section (if not already on by default)
Set the required max concurrent reloads

IMPORTANT TO REMEMBER

A reload node can run NumberOfCores - 2

This means: If your reload node has 8 CPU Cores, it can run 6 concurrent reloads at its maximum.

Use the guideline (NumberOfCPUCores - 2) when performing capacity planning on the possible amount of concurrent scheduled reloads.
The cores can be physical cores or virtual cores; any CPU cores Qlik Sense can "see".
While a reload task typically consumes one core, this is not a hard rule. It can require multiple cores.
Allowing more parallel reloads than CPU cores available does not improve the runtime of concurrent tasks. The reload node's resources will deplete.

This also means: With more parallel reloads than cores the probability for total starvation increases since the QIX engine is CPU hungry and other processes (for example, the operating system) might not get enough CPU cycles to perform their duties.
Each running reload is loaded into RAM. This means that the average RAM footprint grows with more parallel reloads.
Qlik Sense will queue scheduled reloads when system resources reach their threshold
To validate that environment needs proper capacity planning, go to C:\ProgramData\Qlik\Sense\Log\Scheduler\Trace\<Server>_System_Scheduler.txt, an example log shows below:

<ServerName>_System_Scheduler.txt
Domain\qvservice Engine connection released. 5 of 4 used
Domain\qvservice Engine connection 6 of 4 established
Domain\qvservice Request for engine-connection dequeued. Total in queue: 25

Concurrent settings

Use the "Max concurrent reloads" to limit the maximal concurrent tasks can be run at same time on current node. By default, it's set to 4, which means only 4 tasks can be run at same time on this node.

When the 5th task comes in:

It will be queued by sequence.
The queue has a time setting, which will eliminate (erase) a queued task if the timeout limit reaches. By default, it's set to 30 mins, in this case, if none of the first 4 tasks finishes in 30 minutes after the 5th task comes in, then the 5th task will be cancelled due to timeout. It has to be triggered again (manually or next scheduled time slot).
Once one of the 4 running tasks finishes, the 5th task will be executed on this node if it's not timed out.
If you set timeout to 0, this will make the queue never timeout.
It's suggested not to run more than 10 concurrent reloads at once, unless the schedular node has an extreme amount of resources dedicated to it.
If there is low memory or cores available on the node, then some tasks may kick off however take several hours to complete. Adding resources to the node running the reloads is suggested.

Multi-node deployment

On a multi-node deployment, tasks will be balanced from the manager node to any node(s) designated as workers.

It's highly advised to check if the central node configured is set to Manager and Worker or Manager. When set to Manager, it will send all reload jobs to the reload/scheduler nodes, as it should. However if a central node is set to Manager and Worker, this means the Central node will also be involved in performing reloads. This is not recommended.

The work flow looks as follows:

Manager node receives a new task execution request.
Manager node checks the resource availability on each of the worker nodes.
Manager node assigns this task to the node with the lowest number of running tasks per "Max concurrent reloads" setting.

The improvement to track the Max concurrent reloads can, if desired, be disabled. This reverts Sense to an older load balancing method that relies only on CPU usage.

To disable the setting:

Open the Scheduler.exe.config, which by default is located in: C:\Program Files\Qlik\Sense\Scheduler\Scheduler.exe.config
Set DisableLegacyLoadBalancingBehavior setting to false
Restart Qlik Sense Scheduler Service
Repeat these actions on each node of the cluster running the Qlik Sense Scheduler Service

Example

In our example, we allow one concurrent reload, but we assume that two reloads are executed at the same time.

If Task A is executed first, Task B is queued.
If Task A is executed first and then fails, Task B is executed after the failure.
After Task A's failure, Task A is queued and will execute after Task B has finished.

Re: Concurrent Reload Settings in Qlik Sense Enterprise

Emmanuelle__Bustos — Wed, 23 Dec 2020 23:47:38 GMT

Thank you!!!

Re: Concurrent Reload Settings in Qlik Sense Enterprise

TusharDev_Qlik — Thu, 03 Mar 2022 10:32:12 GMT

@Sonja_Bauernfeind

do we have similar document for Qlik Nprinting to check how many concurrent task executions are possible?

Re: Concurrent Reload Settings in Qlik Sense Enterprise

Sonja_Bauernfeind — Fri, 11 Mar 2022 13:35:08 GMT

Hello @TusharDev_Qlik

Qlik NPrinting does not have functionality like this in the product as of right now. If you are interested in raising a request with our developers, I would suggest logging an idea in our ideations section.

All the best,
Sonja

Re: Concurrent Reload Settings in Qlik Sense Enterprise

simobatt — Mon, 11 Jul 2022 12:59:28 GMT

Hi,

Qlik Sense can support concurrent reloads of the same reload task (and so of the same app)?

If so, which is the behavior? Each reload is queued and wait untile the previous reload finished?

Trying to execute the same task (2 concurrent triggers/invocations) or trying to reload the same app with 2 different task in the same moment, makes any difference?

Best
Simone

Re: Concurrent Reload Settings in Qlik Sense Enterprise

Sonja_Bauernfeind — Fri, 22 Jul 2022 11:01:18 GMT

Hello @simobatt

Could you clarify the requirement for this? Reloading the same app at the same time would lead to access issues on the file.

All the best,
Sonja

Re: Concurrent Reload Settings in Qlik Sense Enterprise

Bartosz — Tue, 23 Aug 2022 14:10:42 GMT

Is it necessary to restart services after changing "max concurrent reloads", does the change require reboots?

Re: Concurrent Reload Settings in Qlik Sense Enterprise

Sonja_Bauernfeind — Wed, 24 Aug 2022 11:16:20 GMT

Hello @Bartosz A restart is not required, no 🙂

All the best,
Sonja

Re: Concurrent Reload Settings in Qlik Sense Enterprise

Bartosz — Wed, 24 Aug 2022 11:18:53 GMT

Thx

Re: Concurrent Reload Settings in Qlik Sense Enterprise

simobatt — Wed, 24 Aug 2022 11:34:17 GMT

Hi @Sonja_Bauernfeind

I don't have a specific requirement, it is just a question about the standard behaviour.

We are developing an orchestration and reload service outside Qlik using the Sense APIs.

And I would like to know what happen if a task already running is started again.

Maybe an error? Or the reload execution is queued? Or we have concurrent execution of the 2 reloads?

Thanks

Simone

Re: Concurrent Reload Settings in Qlik Sense Enterprise

Sonja_Bauernfeind — Thu, 01 Sep 2022 09:56:38 GMT

Hello @simobatt

While there is no theoretical limit on how many tasks (overall) you can trigger, hardware and bandwidth and file locking limitations may get in the way.

As for your specific question on triggering one task twice, this will fail with one of the below errors:

Cannot start task, already active

Could not reserve an executor for task: TaskExecutionSession already exist for App

All the best,
Sonja

Re: Concurrent Reload Settings in Qlik Sense Enterprise

jchoucq — Tue, 31 Jan 2023 10:49:05 GMT

Hi @Sonja_Bauernfeind

thank you very much for this article

does this rule still apply?

"For chained tasks, there is no load balancing that happens for tasks. All chained tasks are sent to the same Scheduler Node as the original task. "

Indeed, i've made a test on 2 different multi node architectures (August 2021 SR3 and May 2022 Patch 4), and in both, in chained tasks, the two tasks were executed by 2 differents scheduler nodes.

One last question, is it possible to "determine" which node is concerned in an execution of an external task, not linked to an app ?

Have a good day.

Johann

Re: Concurrent Reload Settings in Qlik Sense Enterprise

Sonja_Bauernfeind — Fri, 10 Feb 2023 08:41:59 GMT

Hello @jchoucq

Let me review this for you.

All the best,
Sonja

Re: Concurrent Reload Settings in Qlik Sense Enterprise

Sonja_Bauernfeind — Fri, 10 Feb 2023 11:27:01 GMT

Hello @jchoucq I've verified this with our Subject Matter experts and can confirm that the information in the article was outdated.

Thank you for raising this!

The outdated information was removed and the article cleaned up a little.

All the best,
Sonja

Re: Concurrent Reload Settings in Qlik Sense Enterprise

Kunal — Tue, 03 Oct 2023 21:28:42 GMT

@Sonja_Bauernfeind Just to confirm, the latest version (say May 2023) does not have this limitation with multiple scheduler nodes?

"For chained tasks, there is no load balancing that happens for tasks. All chained tasks are sent to the same Scheduler Node as the original task. "

Thanks

Re: Concurrent Reload Settings in Qlik Sense Enterprise

korsikov — Tue, 24 Oct 2023 08:46:38 GMT

we have a client with dedicated reload node. 28 vCPU and 26 concurent reload task set.

When we try to start reload task with 25 similar app we see that at
fact only 10 tasks works, all another wait in "running" status w/o any action. what we can to do in this situation?

Re: Concurrent Reload Settings in Qlik Sense Enterprise

Sonja_Bauernfeind — Tue, 24 Oct 2023 08:56:36 GMT

Hello @korsikov

The max concurrent cannot be matched 100% to your cores. Other factors will come into play, such as how some reloads consume more cores and/or will consume resources which signal the Qlik Sense server to begin queueing.

In addition, note:

It's suggested not to run more than 10 concurrent reloads at once, unless the schedular node has an extreme amount of resources dedicated to it.

You may have reached the point where a singular reload node will not be sufficient to carry out the tasks you need, or you will need to review overall performance and reload behaviour.

To investigate why your reloads are queued, we recommend monitoring resources as well as reviewing the log files to locate the reason why a job was queued. If you need more direct assistance with this, please post about your requirement and challenge in the appropriate forum (Deployment & Management), or, if you require direct involvement with Qlik, our consulting services can be contacted to assist you with sizing questions.

All the best,
Sonja

Re: Concurrent Reload Settings in Qlik Sense Enterprise

korsikov — Tue, 24 Oct 2023 11:30:03 GMT

Hi Sonja,

Thank you for fast responce.

there is come kind of hardcoded limitation or it can customize it? QlikView has Heap size limit, but in Qlik Sense it's working in one Engine.exe process.

Re: Concurrent Reload Settings in Qlik Sense Enterprise

BTIZAG_OA — Thu, 14 Dec 2023 08:00:33 GMT

Hello @Sonja_Bauernfeind i have a question for this,

According to

Manager node assigns this task to the node with the lowest number of running tasks per "Max concurrent reloads" setting.

When working on multinode-deployment and one of server becomes unresponsive for any reason (high cpu usage or memory leak etc.) all tasks that need to start are hangs at trigger status. Could this be caused by the setting above? Because of the server(any of workernode) becomes unresponsive, central node might not get answer of current running task number from that node, and might waiting response from it to distribute task for available node. Untill central node gets answer from unresponsive node, it keeps waiting tasks in trigger status. How can we overcome this? Its making a interruption on tasks in our site, untill you restart the slave tasknode that becomes unresponsive.

Re: Concurrent Reload Settings in Qlik Sense Enterprise

christianborg — Wed, 15 May 2024 08:55:42 GMT

Good morning @Sonja_Bauernfeind in which cases do you recommend to disable the Max concurrent reloads config for schedulers and by default use the older load balancing method that relies only on CPU usage ?

Example does it make sense to use the older balancing rule when the task performance (due to different amounts of data loaded) are quite different from on another i.e. one task can take 5 minutes and another task 40 min ?