How to increase and how many reload engines (QVB) to allow in an environment
By default, QlikView Publisher sets the number of "Max number of QlikView engines to use for distribution" to 4. This value determines how many QlikView Publisher tasks can be run in parallel and can be freely adjusted to suit any environment. Sometimes you see the error/warning Failed to allocate engine before you hit the limit you have set.
The image below depicts where to increase the number of engines for distribution or administration. (default 4 and 20 respectively)
When determining how many Reload Engines to allow per QlikView Publisher, there are a number of factors to take into consideration.
The first and fundamental factor that affects how many Reloads that can safely take place simultaneously, is that the Reload Engine, the QVB.exe process, is a fully MultiThreaded process. This means that a single executing task using a Reload Engine WILL try to use as much resources from the hardware as it can, without limit. The limiting factors are only Script Efficiency and Data transfer rates. A reload script that does lots of data aggregation at script level will have a big impact on the CPU utilization for that Reload Task for example. Tasks that run in parallel on a server will compete for the same pool of resources containing CPU cycles and RAM.
With this basic fundamental factor in mind, we can start discussing the possible settings
The first consideration is that you should never allow for more Engines than the Server has CPU cores -1. So for a Server with 8 CPU cores, we should never sat a value higher than 7.
The second consideration is that even on big servers with lots of CPU cores (16+) it rarely pays to go higher than approximately 13 simultaneous reload engines allowed. This is partly due to the first fundamental factor (every running task will try to consume all the resources) and partly due to other bottlenecks, where for example multiple data connections using the same drive will start becoming slower as the concurrent connections goes up. Going for a high number also normally increases the failure rate significantly. Qlik recommends starting with a low number and then increasing gradually until you hit the sweet-spot for your own environment, where you are getting a high throughput without introducing a high failure rate. If your reload schedule calls for a high number of simultaneous tasks to execute, you will be better off clustering the Publisher role over more servers than increasing the number of engines on a single node.
Microsoft Windows limitation
A third consideration is Windows settings that affect the number of concurrent processes of the same kind. If you enable a high number of reload engines and start seeing windows errors related to Windows running out of Desktop Heap Size. This may be visible in the Windows System Event log of the server running Qlikview Distribution Service, with errors saying "Desktop Heap exhausted". If so, you will need to adjust Windows Desktop Heap Size to allow for a higher server load. Allowing for a bigger Desktop Heap may also allow a highly loaded server to run for longer periods of time without the need for server restarts. Messages like this, Failed to create QlikView Engine.. Exception=System.Runtime.InteropServices.COMException (0x80080005) Shows that there is an issue with desktop heap size.
IMPORTANT: Windows registry changes are made at your own risk and are not supported by Qlik Support. Please make a backup of the Windows registry before applying changes to enable rollback if the setting change fails or introduces Windows instability.
Desktop Memory Heap size is control in the following Windows registry key on the server running Qlikview Distribution Service:
Notice that the default value is 1024,3072,512 in x86 or 1024,20480,768 in x64 environments. Also, please note that the skipped part (marked as "..." ) might differ according to the operating system version and should not be modified.
Depending on size of deployment and complexity of tasks/scripts you might need to increase the Non-interactive Desktop Heap even further, the following list gives the next possible setting for the value.