QVS Memory usage -> Daily restarts of QVS service

[klaus_feldam]

QV 11.20.12235.0

Windows Server 2008 R2 SP1

X64

Memory: 28GB

3x Xeon EON E5530 2.4GHz

10 users

10 QVW documents

<500,000 records

------------------------

We are daily running through the memory ceiling of our QlikView 11 server, with the QVS consuming 70-80% of available memory. At that point, users are kicked off and unable to reconnect as memory utilization keeps growing. It's not until we stop and restart the QVS service that normal operation is restored. This happens every 18-24 hours.

We only have 10 users and 10 QVW documents. I have previously worked in a 250-user QlikView environment with 8 quad core processors and 128GB memory - same problem.

Does anyone have any recommendations to resolve this problem?

If there isn't a solution to this unsustainable daily operational issue and memory 'leak' maybe it is time to finally look at a competitor solution.

After 4 years of using QlikView, I love the product, but can't believe Qlik hasn't found a sustainable solution to this memory issue.

[#2]

Klaus,

I would like to point you to Henrics document The QlikView Cache  there is a mention in there of how Qlikview manages the cache, and particularly regarding how it caches expressions.  Some of my Applicaitons have similar expressions on charts in various locations, some returning massive  datasets, I think this may be the cause of our issues.  I am going to streamline the expressions in an example app, and see if the memory utilisation will drop in a test environment.  I do not know yet how we will implement this, but I am thinking perhaps devine expressions as variables, and use the variable in the chart/table.

Also the cached expressions co exist across applications if the applications share the save qvd and expressions.

Richard

[#3]

The fact that 70-80% of your memory is consumed by the QVS is absolutely normal. QVS keeps its cache, even if there are no users.

We run an environment with about 250 users and 16 (operational) documents (disk space sizes between 50 Mb and 1.2 Gb) and during the day memory consumption is always around 70-75%. We do, however, not reload during the day, all reloads are done in the morning. Are users typically getting disconnect during the reloads? If so, a useful test would be to lower the working set limits from e.g. 70-90% to 60-80% leaving enough room for the distribution service. Given the high frequency of your reloads, moving the distribution service to another server is also worthwhile considering.

I noticed also you are still on SR2. We also had issues with users getting disconnected in the past (QV9,QV10 and QV11). When analyzing the memory consumption back then, we typically saw high peaks of memory usage, up to 95+ percent during most of these disconnects. For some reason, either there were issues with QVS memory consumption (leaks?) or with the working set management (QV not releasing cache fast enough?). We have had a ticket with QT on this for about 2 years and they never really found a true cause. For some reason, not even clear to the QT-people, the disconnect issues disappeared completely when we upgraded to QV11 SR4. So, you might also consider an upgrade.

[shane_spencer]

Hi Klaus,

We have similar issues, Qlik have been unable to help and haven't even been able to recreate the issue.

The problem is a "Memory Contention". When you get to the Working Set Min (default value of 75%) QVS will start to flush cached results. If you try to do something that requires a lot of memory at this point QVS will try to flush memory at the same time as allocating memory to the new task, cpu will run up to 100% causing the server to effectively hang. There are a few things you can do to avoid this.

Rather than the thrice daily recycle you could implement a cache flush. Update the settings.ini file with

ClearCacheTimesPerDay=1

in the the [Settings 7] configuration section. Or 2 for twice a day, 3 for three times a day etc.

QlikView IS prone to memory leakages, no matter what Qlik would like you to believe, and this works most of the time but sometimes fails to clear the cache.

Adjust the working set min. At 28GB RAM there's not room for change however with 128GB RAM you can put this higher, maybe 85%. Essentially you need to allow enough memory for all other processes on the server when QVS is at Working Set Min. Use Perfmon to monitor this and pick a suitable level.

Set all your documents to pre-loaded as this will prevent a document being loaded in to Memory when Memory is at Working Set Min.

Understand your "document memory footprint". With all you documents pre-loaded restart QVS and see how much memory they use. The rest will be used for cache so it's important to understand how much of your overall memory is used by documents and how much is cached results.

Understand your reload impact. Use Perfmon to monitor Memory Utilisation during a reload. Isolate the system from users so you can see the impact of just the reload. See how much memory grows during the reload, how long it takes and whether memory drops back to the previous level after. It should drop back but we've experienced "leakages" where after the reload when a document is updated in Memory the old version does not release it's memory. Qlik are still working to fix this but have at least managed to replicate. It may be necessary to move the Publisher to a seperate server as Mark suggested above.

[rwunderlich]

Shane Spencer

"Set all your documents to pre-loaded as this will prevent a document being loaded in to Memory when Memory is at Working Set Min."

I disagree with this recommendation. Instead of responding to user behavior, this artificially drives the RAM usage up. I think it's better to reduce the Document Timeout setting and let unused documents get unloaded sooner.

-Rob

[rwunderlich]

Klaus,

Looking at your logs, I think you server is mis-configured or under resourced for how you are trying to use it.

Here's your memory usage pattern for May 14. Notice how the Avg Loaded Docs keeps increasing. This indicates that you don't have "Allow only one copy of Document in memory" checked on in the server settings (as someone previously recommended). Note that the Doc Settings and Ref Docs to not track the Loaded Docs trend. That's because almost all the Doc Sessions -- 1400 in this case -- are from a user PATHDASH being used from multiple machines. These are short sessions that do not close properly and leave the session and document behind. Is this some kind of automation?

Some recommendations:

1. Set "Allow only one copy of Document in memory" ON. That should have a lot of positive impact.

2. You say your server is 28MB, but the event logs indicate that min is 16.8 which would be 60%. Was there a reason it was turned down from the default of 70%? I would turn it back up to 70%.

3. Reduce the QEMC "Document Timeout".

4. Examine what's going on with the PATHDASH user. This is not typical user behavior, and may require some adjustment or cleanup.

-Rob

[shane_spencer]

If QlikView was stable / reliable at high memory utilisation you would be right, but it's not. Pre-loading will prevent the documents being loaded in to memory when memory utilisation is already at working set min; from my experience it is this type of activity that requires a large amount of memory that's not readily available that causes QlikView a lot of problems. Therefore it's best to have the documents pre-loaded so you can control when they are loaded in to memory.

Rob Wunderlich wrote:

1. Set "Allow only one copy of Document in memory" ON. That should have a lot of positive impact.

With regards to this setting we've experienced an issues with it not working correctly - a memory leak. What happens is when a document is reloaded a new version will be loaded in to memory but the memory from the old document is not released. We've currently got a case open with Qlik, they've been able to recreate the problem and they're working on a solution but in the short term we've had to implement daily QVS restarts.

[#8]

Rob,

I think it depends on the circumstance.

This setting can be useful when you have a few large documents, rather than many small documents.  For example :- If you expect to run with 3 large documents and you only have 2 loaded, then loading the third when the first 2 have consumed a lot of cache and the system memory utilisation is near working-set min, this can cause problems on some implementations, as the document does not manage to load before it times out the user, since its busy finding memory. (and possibly has much to find)

I do agree though if you have 20-30 smaller documents that are infrequently used then allowing unused docs to time out quicker is better.

Perhaps the advice fromShane Spencer should have been

"if you have large documents that need to be used by many users then preload them to ensure they have memory allocated, if you have small infrequently used documents then do not pre-load"

Richard

[rwunderlich]

Excellent refinement Richard. Thanks.

-Rob

[klaus_feldam]

Rob,

Thank you very much for your input and suggestions. You are a great resource on this community.

From the May 14 log, you can see the effects of my QlikView Server Service restarts at 8am and 4pm. The dips are the service restarts.

1. "Allow only one copy of Document in memory" is already set to ON.

2. I turned the working set down to 60%. I have now set it back to 70%.

3. Reduced the QEMC "Document Timeout" to 120 minutes.

4. The PATHDASH user is used for 4 dashboard monitors that we are using in our office. These monitors have 6-8 Qlikview apps (dashboards) open in a Firefox browser, which rotates through the dashboards every couple of minutes and performs a refresh of the page every time a tab is opened. This explains the number of times PATHDASH opens a document. I haven't found any better way of refreshing my QlikView data on our business dashboards.

[Anonymous]

Klaus

Older versions of Microsoft .NET had a memory leak that caused problems like you describe.

Is your .NET a fairly recent version with fairly recent patches applied ?

The actual joker is descibed in support.microsoft.com/kb/2628838 & support.microsoft.com/kb/2600217