Qlik Community

Ask a Question

Knowledge Base

Search or browse our knowledge base to find answers to your questions ranging from account questions to troubleshooting error messages. The content is curated and updated by our global Support team

Announcements
Join us at the Cloud Data and Analytics Tour! REGISTER TODAY

Qlik Engine Memory Management

Sonja_Bauernfeind
Digital Support
Digital Support

Qlik Engine Memory Management

For more details, see QIX engine memory management and CPU utilization.

The below applies to QlikView and Qlik Sense installations hosted on a Microsoft Windows operating system. 
 

Working Set

This control sets the minimum and maximum of the physical amount of RAM that can be used by the Qlik engine. This way it is possible to control if an application can be swapped out of physical memory or not. However, there are no guarantees that the operating system can serve the process with the amount of memory set here.

Using too high settings will degrade the performance of other processes on the computer; this may, however, be desirable if the computer is dedicated for the Qlik engine. Do not change these settings unless you are well acquainted with Windows Virtual Memory Manager! Read more about working sets in the Microsoft Windows documentation.

The settings are:

Low

Sets the minimum amount of memory, in percentage, to be allocated to the application/process. If the use of RAM goes above this limit, Windows is allowed to swap the memory QlikView Server is using to disk.

High

Sets the maximum amount of memory, in percentage, to be allocated to the application/process. If the use of RAM goes above this limit Windows should swap the memory Qlik uses to disk.
Note! The Qlik engine assumes that it has reserved physical memory up to the Low limit.

 

Resolution:

 

Windows, Virtual Memory, Working Sets, and the Page File

 

Qlik products run on the Windows family of operating systems which run on one of two main families of chipsets: x86 (32bit) and x64 (64bit). This combination is very well understood and documented.
Windows and the chipset collaborate to present a memory model to process called “Virtual Memory”. In this model a process does not address physical memory directly, but instead uses IDs to address chunks of “Virtual Memory” called “Pages”.  When an application wants to read or write to a Page of memory it uses the ID of the page and the address of memory within the Page rather than an address in physical memory.
This separation of “Virtual Memory” from physical memory allows Windows to move Pages from physical memory to the disk and back from the disk to physical memory, without the application being aware of where its memory is at any one time. The effect is that the set of applications running on Windows can address much more memory than physically exists on the machine. The downside of this system is that the “Page File” is on disk and is very slow compared to physical memory. It is typically up to Windows to reduce this lost time by intelligently deciding which Pages stay in memory and which are moved to the “Page File”.
The “Working Set” is a term invented by Microsoft that refers to the set of Pages an application has in physical memory at any one time. The Working Set varies over time because Windows can swap Pages onto the disk or load Pages into memory.
 

Page File Settings

 

The Qlik engine can use the Page File, but due to the impact on performance, this usage should be reduced.
We suggest to allow Windows to manage the page file size, though this can be modified.  If memory settings for the QlikView or Qlik Sense system are correct, and the memory is sufficient for the requirements, then the Page File should rarely be used by qvs.exe or the Sense engine. 
The Page File should not be removed or reduced to 0 because this can introduce instabilities into Windows itself and also into qvs.exe if it does need to use the Page File.
Microsoft also suggests that a Page File be moved to a dedicated disk that does not have redundancy checks (RAID for example is not needed).
 

Allocated, Committed, and Free Virtual Memory

 

In the Performance log of qvs.exe there are 4 fields that relate to Virtual Memory:

  • Allocated – Amount of memory in Pages that have been requested by qvs.exe.
  • Committed – Amount of memory in Pages that have been used by qvs.exe.
  • Free – Amount of Virtual Memory theoretically available.
  • Largest Free Block – The largest block of contiguous memory that is free.

 
A page is allocated to a process by Windows when the process requests the memory. However, it is only Committed when the process begins to use it (first write). This allows certain Page IDs to be valid but to not actually have the Page at all – neither in physical memory nor in the Page File – because the Page contains nothing. As a result Committed <= Allocated.
 
Free only functions on 32bit systems. On 64bit systems Windows returns the theoretical amount of memory it can address rather than the theoretical maximum given the real amount of physical RAM and Page File. So on 64bit it almost always returns 8912GB.

 
The Largest Free Block is typically not very useful. In 32bit mode QlikView may ask for one set of contiguous pages of memory that is larger than what can be returned by Windows. This can cause problems for QlikView. In 64bit architectures though, this situation rarely occurs and this information should simply be ignored.
 
In short, the key measure is the Allocated field. Other information that can be useful but must be obtained from Windows monitoring tools include the amount of physical memory allocated to the process and the amount of Page File allocated to the process. Unfortunately, how to do this is out of the scope of this document.
 

Qlik's Memory Model

 

Qlik's memory usage can be broken up into two main categories:

Necessary memory – internal data structures, each document’s database, definition of GUI, every user’s session state etc.
Cache – results of calculations previously carried out to avoid recalculating. This memory can theoretically be lost and recalculated each time.
 
By default, each QlikView/Sense Server is set to a Low Working Set Limit of 70% and a High limit of 90%. The QlikView/Sense Server Service will try to use physical memory exclusively and not use the Page File at all up until the Low value is reached.

Once the Low value is reached, the Qlik engine will begin discarding old cache data and replace it with new where possible. It will also signal Windows that it can begin using the pagefile if necessary. When the High working set limit is reached, the engine will try to stop caching data altogether. Note that at this point issues with the services will begin to arise (performance/stability problems). The engine event log will usually shed light on that and help identify whether or not you might require more memory or not.

 

Recommendations

 

Because usage of the Page File will slow QlikView or Qlik Sense Server down dramatically, it is important to monitor memory usage and adjust the Low setting correctly. If memory grows beyond the Low setting and this setting can’t be increased without impacting the rest of the server, then more memory is needed. Memory can grow beyond the High setting, but this will dramatically impact the performance of the platform. In short, there is no way to fix a hard limit on how much memory QlikView Server can use and have it refuse to load documents or connect users for example.

 

Labels (1)
Version history
Revision #:
4 of 4
Last update:
‎2020-09-17 04:11 AM
Updated by:
 
Contributors