Qlik Sense Engine Enterprise on Windows: How the memory hard max limit works

[Bastien_Laugiero]

The Qlik Sense Engine allows for a Hard Max Limit to be set on memory consumption. This setting requires that the Operating System is configured to support this, as described in the SetProcessWorkingSetSizeEx documentation (QUOTA_LIMITS_HARDWS_MAX_ENABLE parameter).

Before using the Hard Max Limit, familiarize yourself with Microsoft's memory management:

Source: learn.microsoft.com

By default, using the SetProcessWorkingSetSize function to set an application's minimum and maximum working set sizes does not guarantee that the requested memory will be reserved, or that it will remain resident at all times. When an application is idle, or a low-memory situation causes a demand for memory, the operating system can reduce the application's working set below its minimum working set limit. If memory is abundant, the system might allow an application to exceed its maximum working set limit.

The QUOTA_LIMITS_HARDWS_MIN_ENABLE and QUOTA_LIMITS_HARDWS_MAX_ENABLE flags enable you to ensure that limits are enforced.

When you increase the working set size of an application, you are taking away physical memory from the rest of the system. This can degrade the performance of other applications and the system as a whole. It can also lead to failures of operations that require physical memory to be present (for example, creating processes, threads, and kernel pool). Thus, you must use the SetProcessWorkingSetSize function carefully. You must always consider the performance of the whole system when you are designing an application.

How to review your memory settings in Qlik Sense or configure the Hard Max Limit:

After enabling QUOTA_LIMITS_HARDWS_MAX_ENABLE as per Microsoft's guidelines:

1. Open the Qlik Sense Management Console in a supported browser
2. Navigate to Engines in the Management Console and select your engine
   
   
   
   
3. Expand the Advanced settings from the menu to the right
   
   
   
   
4. Review your memory settings and verify that Max Max Limit is selected
   
   

See Editing an engine - Qlik Sense for administrators for details.

To note:

Even with the hard limit set, it may still be possible for the host operating system to report memory spikes above the Max memory usage (%).

This is down to how the Qlik Sense Engine memory limit will be defined based on the total memory available.

Example:

• A server has 10 GB of memory in total, and the Max memory usage (%) is set to 90%.
• This will allow the process engine.exe to use 9 GB of memory. 
• Another application or process may at that point already be consuming 5 GB memory, and this will cause an overload of the system if the engine is set to use 9 GB. 
• The engine.exe will not be able to respect other services

The memory working setting limit is not a hard limit to set on the engine. This is a setting which determines how much we allocate and how far we are allowed to go before we start alarming on the working set beyond parameters. 

Internal Investigation IDs:

QLIK-96872

[QFabian]

Excellent @Bastien_Laugiero , thank you

[aadil_madarveet]

Thanks for the article.

Could you please explain what happens if the max memory usage is reached in a multi-node cluster. If the memory usage one node 1 is at max usage, will the resources of node 2 be used to maintain user sessions and performance. My question is basically, does Qlik manage resources spill over to next available node for the same app.

Thanks,
Aadil

[AdamBS]

Hi

Does the help indicate that this Hard max limit is basically void unless the O/S adjustment is done?

https://help.qlik.com/en-US/sense-admin/February2024/Subsystems/DeployAdministerQSE/Content/Sense_De...

As per @aadil_madarveet when the limit is reached, what is the behavior of the engine?  Does it just restart?

[Sonja_Bauernfeind]

Hello @AdamBS 

Thank you so much for the feedback, this helps us keep our articles up to date. I've added the note regarding the Windows setting, as, yes, it is required. 

As for the second question: If an app is opened at 90% usage on this node and hasn't otherwise been load balanced previously, what will happen is that the app will open while Qlik attempts to aggressively clean memory out to get back to the minimum value. This can still result in a memory overload, but does not have to. 

All the best,
Sonja 

[AdamBS]

Interesting.  Thanks for the update, Adam

[rwunderlich]

@Bastien_Laugiero The referenced doc for configuring the OS seems to show the setting is made using code. How does a customer go about enabling this OS modification in their QS Server?

-Rob