Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
Not applicable

parallel qvw reloads and processors

I swear I've seen this post before but cannot find it. What is the recommended number of parallel reloads (qv.exe) per server processor ?

I'm using powershell to loop through a list of qvd builders and reloading them with the /r switch.

Our server has 4 dual core processors. Currently, I'm limiting it to 4 simultaneous sessions of qv.exe.

Can I really run 8 ?

Any advice is appreciated.

TD

1 Solution

Accepted Solutions
rwunderlich
Partner Ambassador/MVP
Partner Ambassador/MVP

The recommendation is that you limit the max number of concurrent reloads to the number of CPUs. For example, 4 CPUs, limit to 4 reloads. If you are also using the same server for user sessions at the same time as reloads, you should consider fewer reloads to reserve some cycles for user activity.

As you've noted, QVB.exe is the reload task for a server initiated reload. The max number of QVB tasks is set on the QEMC Server, Performance tab. Reload tasks that are triggered when the max tasks have been reached will wait for a "reload engine" to become available for the time limit specified in QlikviewDistributionService.exe.config.

Instead of using qv.exe for your reloads, here are a couple of options that will honor the max reloads settings by performing reloads in the server context. May make your life easier (or not).

1. Set your qvd loader tasks triggers to be "On external event" and use the external event call (http) to trigger these tasks from your external script.

2. Use QlikviewDistributionService.exe command line to run the reloads as server reloads. This is covered in the QV Server Ref Guide.

-Rob

View solution in original post

11 Replies
Not applicable
Author

no thoughts eh ? am I that far off the mark ?

I notice that QVB.exe is actually the process running when these reloads are called by qlikview (QEMC).

Any process count probably needs to count those as well.

Again, any input is appreciated.

rwunderlich
Partner Ambassador/MVP
Partner Ambassador/MVP

The recommendation is that you limit the max number of concurrent reloads to the number of CPUs. For example, 4 CPUs, limit to 4 reloads. If you are also using the same server for user sessions at the same time as reloads, you should consider fewer reloads to reserve some cycles for user activity.

As you've noted, QVB.exe is the reload task for a server initiated reload. The max number of QVB tasks is set on the QEMC Server, Performance tab. Reload tasks that are triggered when the max tasks have been reached will wait for a "reload engine" to become available for the time limit specified in QlikviewDistributionService.exe.config.

Instead of using qv.exe for your reloads, here are a couple of options that will honor the max reloads settings by performing reloads in the server context. May make your life easier (or not).

1. Set your qvd loader tasks triggers to be "On external event" and use the external event call (http) to trigger these tasks from your external script.

2. Use QlikviewDistributionService.exe command line to run the reloads as server reloads. This is covered in the QV Server Ref Guide.

-Rob

Not applicable
Author

Thanks Rob.

For now, I'll leave it at 4 and monitor. Most of the reloads are completed before users begin accessing the system.

I'll check #2 in the ref guide.

Not applicable
Author

Hi,

Number of processor cores = number of QVB.exe is not necessarily true. For example loading huge data volumes from database is a "network bound" task , barely uses any CPU. You could have lots of them in parallel, as many as license allows.

And since version 9, QVB.exe can use more more CPU cores for operations like GROUP BY in script, taking all available CPU for minutes . This is quite nasty, ex when there are two tasks in this phase , locking up all 24 cores in a huge server.

It would be nice to configure the maximum number of threads that QVB is using, so that the server does not become CPU bound ==> failing random tasks with timeout because of this (file shares, directory lookups, publishing to QVS)

-Alex

rwunderlich
Partner Ambassador/MVP
Partner Ambassador/MVP


Alexandru Toth wrote:Number of processor cores = number of QVB.exe is not necessarily true. For example loading huge data volumes from database is a "network bound" task , barely uses any CPU. You could have lots of them in parallel, as many as license allows


That has been the general recommendation as a starting point. But you may find the need to adjust it up or down for your particular circumstances.


Alexandru Toth wrote:
It would be nice to configure the maximum number of threads that QVB is using, so that the server does not become CPU bound ==> failing random tasks with timeout because of this (file shares, directory lookups, publishing to QVS)


I had not noticed the multithreading in V9, although I aware it's used in V10 and was beginning to ponder its impact on configuration. You can set CPU affinities for Reload on the QEMC Performance tab, That may not limit the total number of threads spawned, but hopefully will limit the number of CPUs the threads use.Please let us know if you test that option that and how it works,

-Rob

Not applicable
Author

Affinity is great for QVS process, because there is only one of it.

But I see 20 QVB running same time. I don't want those to all run in same 2 cores => affinity. I want each using 2 different cores => number of threads.

-Alex

Not applicable
Author

Hi,

Is this proof about the multi-threading in QVB ? There are 24 cores, 256 GB memory, server is not paging, but CPU is overloaded . It is quite difficult to make screenshots since the Remote Desktop is totally unresponsive ..

-Alex

Not applicable
Author

Update.

The script I use logs the max cpu and mem used by each process while reloading. It bares out what Alex said about low CPU usage on reloads. Memory usage per reload is also around the same as the the qvd file size. this makes sense.

If a memory intensive QVB kicks off while your reloads are running, the reloads can (and do) fail with an out of memory error. Somehow the QVB's are immune to this failure. They go merrily along while others around it crash.

So - If I can ensure that no (or few) publisher tasks run while reloading, it appears that 6 or 8 can run at the same time as long as I have enough memory.

Alex: This thread discusses the limitation of concurrent publisher tasks which may explain what you are seeing in your last post. http://community.qlik.com/forums/t/26649.aspx

Not applicable
Author

Have fought long time with this. The partial solution is to kill the QVB (and QVS and QV developer too) whenever memory goes above a threshold. Exactly like the Unix ulimit.

The complete solution would be to use Windows System Resource Manager . In addition to killing tasks when using too much memory, it can limit CPU per process . It means never ever QVB is allowed to use 80% CPU on a 24 core machine --> no more timeouts in other programs (Directory Connector, slow QVB reloads, QEMC)

--------

If QlikTech will implement an option for limiting number of threads in QVB.exe, that will be just great. That will enable running 10 concurrent processes, none using more than x% of CPU. That is different than setting the affinity = original problem in less CPU cores.

-Alex