topic parallel qvw reloads and processors in QlikView

parallel qvw reloads and processors

Mon, 26 Jan 2026 18:19:17 GMT

I swear I've seen this post before but cannot find it. What is the recommended number of parallel reloads (qv.exe) per server processor ?

I'm using powershell to loop through a list of qvd builders and reloading them with the /r switch.

Our server has 4 dual core processors. Currently, I'm limiting it to 4 simultaneous sessions of qv.exe.

Can I really run 8 ?

Any advice is appreciated.

parallel qvw reloads and processors

Sat, 27 Nov 2010 13:55:38 GMT

no thoughts eh ? am I that far off the mark ?

I notice that QVB.exe is actually the process running when these reloads are called by qlikview (QEMC).

Any process count probably needs to count those as well.

Again, any input is appreciated.

parallel qvw reloads and processors

rwunderlich — Mon, 29 Nov 2010 06:16:36 GMT

The recommendation is that you limit the max number of concurrent reloads to the number of CPUs. For example, 4 CPUs, limit to 4 reloads. If you are also using the same server for user sessions at the same time as reloads, you should consider fewer reloads to reserve some cycles for user activity.

As you've noted, QVB.exe is the reload task for a server initiated reload. The max number of QVB tasks is set on the QEMC Server, Performance tab. Reload tasks that are triggered when the max tasks have been reached will wait for a "reload engine" to become available for the time limit specified in QlikviewDistributionService.exe.config.

Instead of using qv.exe for your reloads, here are a couple of options that will honor the max reloads settings by performing reloads in the server context. May make your life easier (or not).

1. Set your qvd loader tasks triggers to be "On external event" and use the external event call (http) to trigger these tasks from your external script.

2. Use QlikviewDistributionService.exe command line to run the reloads as server reloads. This is covered in the QV Server Ref Guide.

-Rob

parallel qvw reloads and processors

Mon, 29 Nov 2010 12:27:34 GMT

Thanks Rob.

For now, I'll leave it at 4 and monitor. Most of the reloads are completed before users begin accessing the system.

I'll check #2 in the ref guide.

parallel qvw reloads and processors

Mon, 29 Nov 2010 14:43:28 GMT

Hi,

Number of processor cores = number of QVB.exe is not necessarily true. For example loading huge data volumes from database is a "network bound" task , barely uses any CPU. You could have lots of them in parallel, as many as license allows.

And since version 9, QVB.exe can use more more CPU cores for operations like GROUP BY in script, taking all available CPU for minutes . This is quite nasty, ex when there are two tasks in this phase , locking up all 24 cores in a huge server.

It would be nice to configure the maximum number of threads that QVB is using, so that the server does not become CPU bound ==> failing random tasks with timeout because of this (file shares, directory lookups, publishing to QVS)

-Alex

parallel qvw reloads and processors

rwunderlich — Tue, 30 Nov 2010 08:58:05 GMT


Alexandru Toth wrote:Number of processor cores = number of QVB.exe is not necessarily true. For example loading huge data volumes from database is a "network bound" task , barely uses any CPU. You could have lots of them in parallel, as many as license allows

That has been the general recommendation as a starting point. But you may find the need to adjust it up or down for your particular circumstances.


Alexandru Toth wrote:
It would be nice to configure the maximum number of threads that QVB is using, so that the server does not become CPU bound ==> failing random tasks with timeout because of this (file shares, directory lookups, publishing to QVS)

I had not noticed the multithreading in V9, although I aware it's used in V10 and was beginning to ponder its impact on configuration. You can set CPU affinities for Reload on the QEMC Performance tab, That may not limit the total number of threads spawned, but hopefully will limit the number of CPUs the threads use.Please let us know if you test that option that and how it works,

-Rob

parallel qvw reloads and processors

Tue, 30 Nov 2010 09:41:46 GMT

Affinity is great for QVS process, because there is only one of it.

But I see 20 QVB running same time. I don't want those to all run in same 2 cores => affinity. I want each using 2 different cores => number of threads.

-Alex

parallel qvw reloads and processors

Thu, 02 Dec 2010 14:30:14 GMT

Hi,

Is this proof about the multi-threading in QVB ? There are 24 cores, 256 GB memory, server is not paging, but CPU is overloaded . It is quite difficult to make screenshots since the Remote Desktop is totally unresponsive ..

-Alex

parallel qvw reloads and processors

Fri, 07 Jan 2011 11:47:23 GMT

Update.

The script I use logs the max cpu and mem used by each process while reloading. It bares out what Alex said about low CPU usage on reloads. Memory usage per reload is also around the same as the the qvd file size. this makes sense.

If a memory intensive QVB kicks off while your reloads are running, the reloads can (and do) fail with an out of memory error. Somehow the QVB's are immune to this failure. They go merrily along while others around it crash.

So - If I can ensure that no (or few) publisher tasks run while reloading, it appears that 6 or 8 can run at the same time as long as I have enough memory.

Alex: This thread discusses the limitation of concurrent publisher tasks which may explain what you are seeing in your last post. http://community.qlik.com/forums/t/26649.aspx

parallel qvw reloads and processors

Tue, 11 Jan 2011 10:18:07 GMT

Have fought long time with this. The partial solution is to kill the QVB (and QVS and QV developer too) whenever memory goes above a threshold. Exactly like the Unix ulimit.

The complete solution would be to use Windows System Resource Manager . In addition to killing tasks when using too much memory, it can limit CPU per process . It means never ever QVB is allowed to use 80% CPU on a 24 core machine --> no more timeouts in other programs (Directory Connector, slow QVB reloads, QEMC)

--------

If QlikTech will implement an option for limiting number of threads in QVB.exe, that will be just great. That will enable running 10 concurrent processes, none using more than x% of CPU. That is different than setting the affinity = original problem in less CPU cores.

-Alex

Re: parallel qvw reloads and processors

pablolabbe — Fri, 06 Jul 2012 13:38:27 GMT

Hi Alex,

Post it as an Idea, then we could vote on.

Best,

Pablo Labbe

Qlikview Consultant

www.visiongi.com.br

Re: parallel qvw reloads and processors

Tue, 07 Apr 2015 12:12:37 GMT

Hi Tod,

I was facing task failures as we have set up more than 15 reload tasks to run parallely. So i had reach out to support to find out what is the agreeable/suggested no of parallel reloads here is the reply from support:

Hope this helps

1) How many cores do you all have installed? Because having an odd number of cores isn't something that I have experienced.
2) The rule of thumb is that the maximum limit should be number of cores - 1. The thought behind this is that you should always leave at least one core available for the operation system.
3) Keep in mind that each task uses a core. So if you haven't configured your registry to allow more than 9 cores for QlikView, then you will experience task failures when you are trying to run more than 9 tasks. But that additional task will run fine when it is not trying to run with 9 other tasks.
4) From the case description, it looks like you all are running 11.0 SR2 which was released in November 2012. Upgrading may not be possible due to your environment, but there have been quite a few releases since then with many, many bug fixes. So if it is possible, I would recommend considering upgrading to a newer service release. Additionally, this version would not be patchable should you encounter an issue that requires a patch.

Let me know about your availability and I can schedule a Webex to talk through this case with you.

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Attachment mentioned in the reply has the following content:

Description

In Version 11 Release Notes, we say Support can provide the information on how to run more engines than 9.

Version 10 Release Notes
Due to a limitation associated with using Microsoft’s COM objects we recommend that you limit the number of QlikView Engines (QlikView Enterprise Management Console>> Setup>> Distribution Services>> Advanced tab) to a maximum of 9 or the number of processor cores available on the host server, whichever is lower.

Version 11 Release Notes
Due to a limitation associated with using Microsoft’s COM objects we recommend that you limit the number of QlikView Engines (QlikView Management Console>> System>> Setup>> Distribution Services>> Advanced tab) to a maximum of 9 or the number of processor cores available on the host server -1, whichever is lower.
If you have more than 9 processor cores, and wish to run more Engines, contact Support for information regarding a registry change to the Desktop Memory Heap settings on the server.

Resolution

Note: Please backup your registry before you make any changes.

Desktop memory Heap Registry change:

Change the registry setting:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SubSystems\Windows

Change the desktop heap setting in the registry to '1024,20480,2048'

%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows
SharedSection=1024,20480,2048 Windows=On SubSystemType=Windows
ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3
ServerDll=winsrv:ConServerDllInitialization,2 ProfileControl=Off
MaxRequestThreads=16

Default is 1024,3072,512 in x86 or 1024,20480,768 in x64

http://blogs.msdn.com/ntdebugging/archive/2007/01/04/desktop-heap-overview.aspx