Memory Maxed & not being released/Working Set not Working?

[MalcolmCICWF]

Today we had more open reports in Access Point than we usually see, which isn't bad if they are needed. A couple of these reports are large such as the Inventory Report that you see. The problem is I had users complain they couldn't get into a report through Access Point and when I looked the Memory was at 99% (60GB or RAM assigned ). It seemed like with the default working set of 70/90 on the server, some of these "0" sessions should have been freed up from the memory when new people were trying to access reporting. Am I wrong? It's almost like there was some sort of memory leak... Anyone experienced this? been running this environment for many years, nothing has changed recently.

I reset the server quickly and it cleared up. They opened 4-5 reports and memory is hovering around 10% now.  

Sessions open

[Miguel_Angel_Baeyens]

The fact that QlikView does not release any memory until a service restart is by design and expected. This is not an issue and will not cause any performance problems, or slowness or server hangs.

If however it does, it basically means the server lacks resources to process the load that QlikView is being requested to process: usually too-demanding apps, undersized hardware or a too large concurrency peak.

The reason is that QlikView will add to memory every click the user does and every application the user opens until it reaches the Working Set Low (which, oversimplifying is like saying to Windows "QlikView can use 70% of the available RAM no matter what". This is the value by default, and it can be changed in the QMC.

If more memory is needed due to more users accessing or too many apps open, QlikView will start purging to avoid the Working Set High to be reached, after that, most likely, the operating system will start paging to disk and slowing down the entire server (and creating a huge file in the hard disk).

When that happens often, it's time to see if a new server or increasing resources in the case of virtual instances.

I would first of all check the Windows Event Viewer to confirm that it is the QlikView Server service what is causing the resource exhaustion, or if any other process running in the background (such as antivirus or backup) could be the culprit.

Check also the QlikView Events, Sessions and Performance logs, to identify potential error messages.

Then, see which of those QVWs is using the most memory, and dig deeper until you can find a very expensive object which could be trigger that excess of memory requests.

See how to check which object is the most resource consuming by enabling the QIX Performance logs (https://support.qlik.com/articles/000062587 and https://help.qlik.com/en-US/qlikview/April2019/Subsystems/Server/Content/QV_Server/QlikView-Server/Q...) and check the Scalability Tools (https://community.qlik.com/t5/Qlik-Scalability/QV-Scalability-Tools-QV10-QV11-QV12/gpm-p/1486942) to benchmark your environment and make sure it will support the load it should be supporting.

[MalcolmCICWF]

yes, I understand that, but what I am saying is that QlikView is not releasing memory at it's working load. The zero sessions are examples of reports kept in memory but nobody actively looking at them. Then there are sessions of 1 and 2 where users are looking at reports, but my memory was at 99%, and QlikView was not releasing the 0 sessions which are help in memory in case they are called upon again. When QlikView hit it's 70% working load, it should have purged these zero sessions and it wasn't. The reports that are open and active sessions should not have made up the 99% of the memory being used.

Once I restarted the server to attempt to solve it, all sessions were purged and the users who were in the active sessions you see, once again opened those reports, but this time memory was around 30%. It's as if QlikView was not releasing memory it no longer needed.

I was trying to see if anyone else had experienced something along these lines. This is the first I have seen of it in 8 or so years

[Brett_Bleess]

Jeremy, what you can use to free up the 'document' space are the QMC settings:

Global document timeout setting 

Document level timeout setting 

So you can control things globally or per document such that some documents can have longer/shorter timeouts than others.  Keep in mind this setting only begins counting down when there are no user sessions in the app...  Hopefully this is the piece you were looking to find.  If you have Publisher, you would likely want to set things in QMC here:

Publisher Source Document timeout setting 

That should cover things, just note this will only unload the QVW from memory, the cached views etc. fall under the working set limits, so the memory will only drop by the actual in-memory usage of the QVW file.

Regards,
Brett

[MalcolmCICWF]

First off, thanks for the help, It may be time to revisit our document timeout times as we continue to grow.

But, we still a couple months later seem to be running into more issues. We recently went from 8 to 12 CPU cores and from 40 to 60 RAM in a virtual environment.  I see many posts throughout the community of environments at  256-512 RAM & 32-64 CPUs. I am wondering then, has our needs outgrown our resources and is it normal to have larger setups such as this? 

I ask because we are getting the "WorkingSet: Virtual Memory is growing critically beyond parameters 68.333 (42.000) GB" which is over our max 60GB RAM. It's only for a couple of our reports though that have 30+ million records in them. The one has 266 fields, 3 sheets, 154 objects, 78 expressions, and is over 2GB. The users swear by needing everything that is in there and I'm not sure if a few of the objects or expressions are triggering this large usage, nor am I sure how to identify a specific one causing it..... is there a way to do this easily?

Or, is it reasonable to say we may need to expand our resources as we grow to accommodate larger reports such as this? Is there a limit within Qlik or can you just keep throwing resources at it? Seems like that is the wrong way to go about it long term but I'm not sure right now.

[Brett_Bleess]

Probably the best advice I could give you here would be to get in touch with your Qlik Account Manager and discuss a health check/architecture review with them, it could be done by our consultants or through one of our partners, but I think that would likely ensure you are good for the current as well as provide you a roadmap to follow going forward.  We do have some customers with some very large servers and a lot of folks have gone virtual.  Cannot recall if I mentioned that in virtual environments, it is key that the resources are dedicated upfront when the server boots up, dynamic allocation will not work, and the host server should NOT be oversubscribed, as that can cause issues too.  

Cannot recall if I gave you the following link either, but posting again just in case, this should help explain how things work more thoroughly:

Qix Engine memory management 

Regards,
Brett

[MalcolmCICWF]

We are aware of most of that yes, thank you. Isn't it true that because how QLikview is designed, it really isn't supported by QLik if you move to a virtual environment? We are currently running ours on a VM setup with 60GB RAM and 12 CPU. It is only 1-2 reports that have 30 million record... by 262 fields... by 150 objects..... by 78 expressions... that cause the spikes when they start clicking around. But, when asking if we can split the report or reduce any of this, they cannot live without anything they have loaded in. Then our only option to them is a larger environment, which is not a great scalable or sustainable route.

[Brett_Bleess]

We have many customer using VM's at this  point, the trick is to be sure to treat the VM as a physical server, all resources allocated on boot of the VM, no dynamic allocations, and be sure the Host machine is not oversubscribed, our VM should be able to have full access to all the CPU/Memory allocated to it without being throttled by the host, that should work pretty well, will be a small performance hit, but not too bad from what we have seen.  The way someone told me to think about it recently was have the VM team think of us as a transactional SQL instance, that should put them in the right frame of mind when setting things up in the VM for you.  Hope this helps.

Regards,
Brett

[MalcolmCICWF]

Are you aware of a way to identify specific problem Expressions or Objects in a report? Governance Dashboard shows a report's complexity and all of that, but nothing what specifics they are clicking that is pulling the large memory usage. Is there a specific "Limit" in Qlikview such as something like Excel which has a record limit... or are we only limited by the resources we have available? We've never ran into this issue, but now this larger report is causing a large memory pull and they can't remove anything from the report...

[Brett_Bleess · Accepted solution]

The only one of which I am aware is the following, everything is based upon resources in the environment...

https://help.qlik.com/en-US/qlikview/April2019/Content/QV_HelpSites/limitations.htm

I am not aware of any other tools to track down poorly performing expressions either.  You could try Rob Wunderlich's Cookbook site perhaps, as he has some unique things out there:

https://qlikviewcookbook.com/

Regards,
Brett