Memory usage problems in qlikview server

[#1]

Hi everybody!

I am having 70% of memory usage in qlikview server (without automatic loading operations executing). We have to restart the server every week in order to deallocate memory, that's why I have some questions about this:

1. Is the amount of memory usage normal?

2. I would like to know which operations are using the memory? user sessions?

3. I know that we have the possibility to configure the minimum and maximum of the physical amount of RAM that is allocated to. The default value is 70%. Could I change it to be less than that, for example: 50%?

4. Is there a way to improve this behaviour in order to avoid restart the server every week?

Hope you can help me. Any idea will be appreciate. Thank you!

Images:

[marcus_sommer]

That's behaviour is normal and deliberately designed in this way. The server cached applications and user-sessions within the working set limits and normaly there is no need to restart the services on a regular basis - if the server gets new demands on RAM it will release enough of old cache-parts. A quite important requirement for it is to have an only for the QlikView server dediciated machine (respectively nodes by a larger environment). A sharing of the resources will always lead to more or less problems and/or to a manual monitoring/controlling of the different tools.

- Marcus

[#3]

Thank you Marcus!

The server is dedicated to Qlikview. We have 150Gb of RAM. The problem is that when the automatic loads are executing, the %memory usage increases and some loadings fail. In the logs we can see that the problem is memory.

What should we do to avoid this problem?

[marcus_sommer]

The updates/distributions of applications controlled by the qmc aren't a direct part of the qlikview server else of the publisher engine (qvb.exe) and if they are running on the same machine they will be compete on the available RAM and this might be sometimes too small for the publisher and therefore some the tasks may fail.

If you have only this machine you will need to decrease the working set limits for the server and/or you need a (better) strategy/timeline when to do which qmc-tasks (different schedules, extending of the timeframes, avoiding parallel running tasks ...). Helpful could be also some improvements within your datamodels and your data-architecture by implementing a more granular layer-structure and therefore splitting the various tasks into smaller slices.

Best would of course be to add one or several machines more within a cluster-installation to get each server-services its own dedicated server and/or to use even several machines for the server/pulbisher-services.

- Marcus

[#5]

Hello @Linda

For your questions amount of memory usage normal?

Minimize memory usage

Large amounts of data require better control over memory usage. QlikView helps a lot by automatically detecting data types in source data. However QlikView not always does it correctly and sometimes it allocates more memory for a field value than it really is necessary. On a smaller data sets it's not critical, however such sub-optimal data allocation might have significant impact on a large data volume. This behavior comes from the fact that QlikView basically has 3 data types:

• Number
• Text
• Dual

Denormalize data model

Besides memory consumption, another challenge that comes with big data volume is increased calculation time. QlikView users and developers are a bit spoiled in this regard as they used to sub-second responses and often get annoyed when calculation times exceed 7-10 seconds which would still be considered as acceptable response time with other BI tools.

Thanks.

[#6]

Marcus I have been reading information about memory usage and now I understand better how it is managed.

We have the working set limited as follow: 70% - 95% (Total: 150 GB RAM)

We can see sometimes 99% of memory usage (without automatic loading operations executing) and this causes performance issues in the server. I have some conclusions about this behaviour:

1. We have left little memory space for OS. (5%), perhaps OS need more than that.

2. Our documents need more RAM, perhaps we have poor design in some of them.

Posible solutions:

1. Decrease the low limit of the working set, perhaps 60% and decrease the high limit to 90%, leaving 10% for OS.  is this a posible solution?

2. Tunning the documents. How can I know which documents are candidates to be tunning?

I will aprecciate your comments. Thank you very much!

[#7]

Thank you Madison!

[marcus_sommer]

Yes, an adjusting of the working set limitations might be helpful whereby the OS didn't need very much of the RAM. I'm not a specialist on this point but normally 1 - 2 GB RAM should be enough RAM for it (maybe by executing system updates it needs a bit more).

Therefore I think that it will be the server and the publisher which are competing for the available RAM to the same time and the best would be if you could separate their resources-peaks into different intervals. The best way to get an overview when which processes and applications consume which resources is the use of the QlikView Governance Dashboard 2.0.4 released!‌.

Based on this you could check your most consuming applications with the Document Analyzer v3.6 is available and especially reducing the max. number of distinct field-values will decrease the RAM consumption of your applications: The Importance Of Being Distinct.

- Marcus

[#9]

Hi Marcus!

Thank you very much for your response. I have two more questions please!!

1. Do you know if publisher is allowed to use memory from the working set? 

2. Is likely that Publisher use memory when there are no reloads? if so, what other processes Publisher do that need RAM?

Thank you very much!

[marcus_sommer]

The publisher will always take all available RAM but it does not include the reserved and occupied RAM from the server. This meant if the server consumed RAM within the limits of the working sets it won't release it for any other processes.

AFAIK the publisher consumed only RAM if there are any tasks running which didn't need mandatory reload-tasks, it could be also any distributions.

Are you really sure that there no other processes running on the machine? Maybe the windows event- and performance-logs could give hints whats happening on the machine - whereby I would at first check and monitor all qlik related processes with the governance dashboard before searching on the OS side.

- Marcus