Skip to main content

Official Support Articles

Search or browse our knowledge base to find answers to your questions ranging from account questions to troubleshooting error messages. The content is curated and updated by our global Support team

Announcements
Welcome to Qlik Community! Check out our new navigation! FIND OUT MORE

How to increase and how many reload engines (QVB) to allow in an environment

cancel
Showing results for 
Search instead for 
Did you mean: 
Sonja_Bauernfeind
Digital Support
Digital Support

How to increase and how many reload engines (QVB) to allow in an environment

By default, QlikView Publisher sets the number of "Max number of QlikView engines to use for distribution" to 4. This value determines how many QlikView Publisher tasks can be run in parallel and can be freely adjusted to suit any environment. Sometimes you see the error/warning Failed to allocate engine before you hit the limit you have set.  

A more detailed best practice guide is available in Scaling the QlikView Publisher.
 

The image below depicts where to increase the number of engines for distribution or administration. (default 4 and 20 respectively)

Max Number of Engines.png

 

When determining how many Reload Engines to allow per QlikView Publisher, there are a number of factors to take into consideration. 

 

Fully MultiThreaded

The first and fundamental factor that affects how many Reloads that can safely take place simultaneously, is that the Reload Engine, the QVB.exe process, is a fully MultiThreaded process. This means that a single executing task using a Reload Engine WILL try to use as much resources from the hardware as it can, without limit. The limiting factors are only Script Efficiency and Data transfer rates. A reload script that does lots of data aggregation at script level will have a big impact on the CPU utilization for that Reload Task for example. Tasks that run in parallel on a server will compete for the same pool of resources containing CPU cycles and RAM. 

With this basic fundamental factor in mind, we can start discussing the possible settings

The first consideration is that you should never allow for more Engines than the Server has CPU cores -1. So for a Server with 8 CPU cores, we should never sat a value higher than 7. 

Note also, that the QlikView Distribution Service has been equipped with a load protection mechanism since the 12.10 release. See Tasks are not executed as expected and are shown in status Queued for details. 
 

System bottlenecks

The second consideration is that even on big servers with lots of CPU cores (16+) it rarely pays to go higher than approximately 13 simultaneous reload engines allowed. This is partly due to the first fundamental factor (every running task will try to consume all the resources) and partly due to other bottlenecks, where for example multiple data connections using the same drive will start becoming slower as the concurrent connections goes up. Going for a high number also normally increases the failure rate significantly. Qlik recommends starting with a low number and then increasing gradually until you hit the sweet-spot for your own environment, where you are getting a high throughput without introducing a high failure rate. If your reload schedule calls for a high number of simultaneous tasks to execute, you will be better off clustering the Publisher role over more servers than increasing the number of engines on a single node. 
 

Microsoft Windows limitation

A third consideration is Windows settings that affect the number of concurrent processes of the same kind. If you enable a high number of reload engines and start seeing windows errors related to Windows running out of Desktop Heap Size. This may be visible in the Windows System Event log of the server running Qlikview Distribution Service, with errors saying "Desktop Heap exhausted". If so, you will need to adjust Windows Desktop Heap Size to allow for a higher server load. Allowing for a bigger Desktop Heap may also allow a highly loaded server to run for longer periods of time without the need for server restarts.
Messages like this,
Failed to create QlikView Engine.. Exception=System.Runtime.InteropServices.COMException (0x80080005)
Shows that there is an issue with desktop heap size.

IMPORTANT: Windows registry changes are made at your own risk and are not supported by Qlik Support. Please make a backup of the Windows registry before applying changes to enable rollback if the setting change fails or introduces Windows instability.

Desktop Memory Heap size is control in the following Windows registry key on the server running Qlikview Distribution Service:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SubSystems\Windows

 

Change the desktop heap setting by setting the SharedSection value to 1024,20480,2048. After change the registry key should look something like the following (all on one line):

%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,2048 ...


Notice that the default value is 1024,3072,512 in x86 or 1024,20480,768 in x64 environments. 
Also, please note that the skipped part (marked as "..." ) might differ according to the operating system version and should not be modified.

Depending on size of deployment and complexity of tasks/scripts you might need to increase the Non-interactive Desktop Heap even further, the following list gives the next possible setting for the value. 

%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,4096 ...


OR

%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,6144 ...


OR

%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,8192 ...

 

Microsoft reference regarding Windows heap and Server 2012: https://support.microsoft.com/en-us/kb/947246
 

A more detailed best practice guide is available in Scaling the QlikView Publisher.

Labels (2)
Comments
Mika_2000
Contributor III
Contributor III

Is it true that customers with a QlikView Publisher license can change the Max number of simultaneous QlikView engines for distribution and customers with a QlikView Server license cannot change this setting?

We have two environments one test environment and one production environment. On the test environment we have QlikView publisher and we can change the Max number of simultaneous QlikView engines for distribution.

On the production environment we have a QlikView server license and we cannot change the Max number of simultaneous QlikView engines for distribution. This textbox where we can change the engines is not aviable and it's just a number.

Sonja_Bauernfeind
Digital Support
Digital Support

Hello @Mika_2000 

You are able to change the max concurrent reloads that can run, rather than distribution/management as with the Publisher.

The QlikView Server without a Publisher license cannot distribute and will only reload documents as per a simplified schedule configured on the user document itself.

concurrent reloads.png

 

Hope this helps!

All the best,
Sonja 

Mika_2000
Contributor III
Contributor III

Hi @Sonja_Bauernfeind,

Thank you very much for the fast reply! 

This works.

 

Kind regards,

Mika_2000
Contributor III
Contributor III

Hi @Sonja_Bauernfeind,

I was a little to quick with confirming that the solution you provided worked 🙃.

 

Mika_2000_0-1659962369048.png

Our test environment

 

Mika_2000_2-1659962455125.png

Our production environment.

 

We only have the option to change the Max number of simultaneous Cloud Distrubutions.

It is however worth noting that the test environment has a different license then our production environment.

Test environment: NUMBER_OF_CLUSER_NODES;2;;

Production environment: NUMBER_OF_CLUSER_NODES;1;;

Can this be the issue?

Thank you in advance 😄

 

 

 

Sonja_Bauernfeind
Digital Support
Digital Support

Hello @Mika_2000 

I am basing my reply on the assumption that you have a Publisher license in the Prod environment:

The actual "reloads/distributions" should be right above where you have cut off. If they aren't there, then I would suggest logging a ticket with support as I do not know why those could possibly be cut off.

This is what it should look like:

qlikview engine for reloads and distribution.png

 

 

Mika_2000
Contributor III
Contributor III

Unfortunately the option isn't there either, thanks for your reply!

I will submit a ticket at support, if I find anything in the meantime or support is able to help me with this issue I'll let you know the solution!

Thanks a lot for your help 😀

Sonja_Bauernfeind
Digital Support
Digital Support

Hello @Mika_2000 

Thank you for the collaboration!  I'm curious to see where the issue is coming from and will be documenting it once we know.

All the best,
Sonja 

Version history
Last update:
3 weeks ago
Updated by: