Improve performance of QV

[Anonymous]

I have the following configuration in QMC

System properties

size of QVW is 15 GO

My problem is the QVW is slowly how coul I improve it ?

[#2]

Hello There,

15GB is quite a large application.  If you and/or your team are QV savvy, you should first attempt to reduce the memory stamp and increase the performance of the QVW.  This would include such activities as:

1. Get rid of unused fields and make text fields -> num when applicable

2. Optimize the data model and get rid of unused rows (dimensions without facts and vice versa).

3. Optimize the performance of particularly strenuous charts and tables.

Search for Rob Wunderlich's Document Analyzer - this is very helpful for performance tuning.

The specs on the server are decent - you may want to consider one with more processors as QV takes advantage of multi-threaded machines. QV is also very dependent on memory.. Check to see what the average RAM utilization on the server is by QVS.

I'd be happy to make introductions if you;d like to speak to a consultant who specializes in this type of thing.

Best,

David

[srchilukoori]

Hi,

A 15 GB app empirically takes up 150 GB or more RAM on the server for the first user, and then some more for the subsequent users.

Your best shot to optimize the application is to drop all the unused fields from the application towards the end of the script. This can be done by a simple statement as below for each unused field.

drop field <field name>;

[#4]

I respectfully disagree, srchilukoori

The drop field function does not relieve the application of the 'state space' created by the initial load of a field. Thus, on an application of said magnitude, you can leave quite a memory stamp for fields you aren't even pulling in..

You can simply comment out the fields you don't want if you have a preceding load.  What I would do is drop fields from your loaders (or transformers) and load them into a completely new QVW <-- this will ensure that you get rid of any remnants of unnecessary RAM.  You can then copy and paste objects.  The only downside here is you lose any variables that are not defined in the script - but I find that many of those wind up going unused.

I do not believe the 15GB to 150GB on server is correct.  Several factors affect the size of the application on the server, including # of rows, the data model, possible selections.  Also - the server they are using is only 128GB, so it couldn't possibly exceed that. 

[srchilukoori]

Hi David,

You are right about the state space, however the drop fields is test to confirm if in fact the amount of data in application is the culprit. The following link explains the memory usage by a Qlik app very well.

https://community.qlik.com/blogs/qlikviewdesignblog/2013/01/28/memory-statistics

Qlikview app takes up to 10 times of the qvw size in the RAM, on an average. The n is however dependent on the type of data model, # of fields and rows, type of data as you mentioned. When the app needs more RAM than the server has, OS starts using the paging file. This makes the entire server run really slow. of the 128 GB close to 30GB is used by the Windows it self, which leaves about 100 GB for the rest of the application.

https://community.qlik.com/thread/84367

Thanks.

[#6]

That is a good way to test, I agree.  You can also get a very good look at the fields that take up a good deal of memory by using Document Analyzer.

I'm not convinced on the QVW size.  I was just working with an application last week that is about 8GB off, and only 12GB on the server.  Check out this whitepaper - it is an interesting read.

QlikView Scalability Overview

HOWEVER - I think you are very much on the right line of thought.. Even if it is not 10 times as large in this case, they are likely running close to or above the working set minimum on a daily basis.  You mentioned that QVS relies on the OS, which is extremely slow.. QVS also starts to unload memory, which means that you can't take advantage of the cached calculations!

Wouldn't be surprised if you start seeing server crashes if they aren't already happening.