Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
By default, QlikView Publisher sets the number of "Max number of QlikView engines to use for distribution" to 4. This value determines how many QlikView Publisher tasks can be run in parallel and can be freely adjusted to suit any environment. Sometimes you see the error/warning Failed to allocate engine before you hit the limit you have set.
A more detailed best practice guide is available in Scaling the QlikView Publisher.
The image below depicts where to increase the number of engines for distribution or administration. (default 4 and 20 respectively)
When determining how many Reload Engines to allow per QlikView Publisher, there are a number of factors to take into consideration.
Fully MultiThreaded
The first and fundamental factor that affects how many Reloads that can safely take place simultaneously, is that the Reload Engine, the QVB.exe process, is a fully MultiThreaded process. This means that a single executing task using a Reload Engine WILL try to use as much resources from the hardware as it can, without limit. The limiting factors are only Script Efficiency and Data transfer rates. A reload script that does lots of data aggregation at script level will have a big impact on the CPU utilization for that Reload Task for example. Tasks that run in parallel on a server will compete for the same pool of resources containing CPU cycles and RAM.
With this basic fundamental factor in mind, we can start discussing the possible settings
The first consideration is that you should never allow for more Engines than the Server has CPU cores -1. So for a Server with 8 CPU cores, we should never sat a value higher than 7.
Note also, that the QlikView Distribution Service has been equipped with a load protection mechanism since the 12.10 release. See Tasks are not executed as expected and are shown in status Queued for details.
System bottlenecks
The second consideration is that even on big servers with lots of CPU cores (16+) it rarely pays to go higher than approximately 13 simultaneous reload engines allowed. This is partly due to the first fundamental factor (every running task will try to consume all the resources) and partly due to other bottlenecks, where for example multiple data connections using the same drive will start becoming slower as the concurrent connections goes up. Going for a high number also normally increases the failure rate significantly. Qlik recommends starting with a low number and then increasing gradually until you hit the sweet-spot for your own environment, where you are getting a high throughput without introducing a high failure rate. If your reload schedule calls for a high number of simultaneous tasks to execute, you will be better off clustering the Publisher role over more servers than increasing the number of engines on a single node.
Microsoft Windows limitation
A third consideration is Windows settings that affect the number of concurrent processes of the same kind. If you enable a high number of reload engines and start seeing windows errors related to Windows running out of Desktop Heap Size. This may be visible in the Windows System Event log of the server running Qlikview Distribution Service, with errors saying "Desktop Heap exhausted". If so, you will need to adjust Windows Desktop Heap Size to allow for a higher server load. Allowing for a bigger Desktop Heap may also allow a highly loaded server to run for longer periods of time without the need for server restarts.
Messages like this,
Failed to create QlikView Engine.. Exception=System.Runtime.InteropServices.COMException (0x80080005)
Shows that there is an issue with desktop heap size.
IMPORTANT: Windows registry changes are made at your own risk and are not supported by Qlik Support. Please make a backup of the Windows registry before applying changes to enable rollback if the setting change fails or introduces Windows instability.
Desktop Memory Heap size is control in the following Windows registry key on the server running Qlikview Distribution Service:
Change the desktop heap setting by setting the SharedSection value to 1024,20480,2048. After change the registry key should look something like the following (all on one line):
%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,2048 ...
Notice that the default value is 1024,3072,512 in x86 or 1024,20480,768 in x64 environments.
Also, please note that the skipped part (marked as "..." ) might differ according to the operating system version and should not be modified.
Depending on size of deployment and complexity of tasks/scripts you might need to increase the Non-interactive Desktop Heap even further, the following list gives the next possible setting for the value.
%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,4096 ...
OR
%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,6144 ...
OR
%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,8192 ...
Microsoft reference regarding Windows heap and Server 2012: https://support.microsoft.com/en-us/kb/947246
A more detailed best practice guide is available in Scaling the QlikView Publisher.
Hello @Mika_2000
You are able to change the max concurrent reloads that can run, rather than distribution/management as with the Publisher.
The QlikView Server without a Publisher license cannot distribute and will only reload documents as per a simplified schedule configured on the user document itself.
Hope this helps!
All the best,
Sonja
I was a little to quick with confirming that the solution you provided worked 🙃.
Our test environment
Our production environment.
We only have the option to change the Max number of simultaneous Cloud Distrubutions.
It is however worth noting that the test environment has a different license then our production environment.
Test environment: NUMBER_OF_CLUSER_NODES;2;;
Production environment: NUMBER_OF_CLUSER_NODES;1;;
Can this be the issue?
Thank you in advance 😄
Hello @Mika_2000
I am basing my reply on the assumption that you have a Publisher license in the Prod environment:
The actual "reloads/distributions" should be right above where you have cut off. If they aren't there, then I would suggest logging a ticket with support as I do not know why those could possibly be cut off.
This is what it should look like:
