QVD Optimization

[chhavi376]

Hi All,

I have 3 Microsoft Excel Comma Separated Values Files(3.5GB each)

all the three files have exactly same columns, but different data.

I want to append the data into a QVD.

the resultant QVD is 4.8GB, and is making the application very heavy(2.3GB).

I am facing a lot of performance issues due to this.

any suggestions?

Thanks

[rwunderlich · Accepted solution]

Examine the qvw with

Qlikview Cookbook: QV Document Analyzer http://qlikviewcookbook.com/recipes/download-info/document-analyzer/

You may have some fields that can be removed or optimized.

-Rob

http://masterssummit.com

http://qlikviewcookbook.com

[chhavi376]

Hi Rob,

I tried this,

In recommendations, I am just getting this for all the fields. I don't this is going to help much

[avinashelite]

how many records you have ?

Do you need all the data to be displayed while login in to app ?

You could force the user to select the filters and then display the data . EG if you 10 years of data then customer doesn't need all 10 years data at same time rite

What's the scheme your following ? START is proffered with the huge data set

[chhavi376]

around 1.5cr

Yes, We would need all the data

[Anil_Babu_Samineni]

This is the case, I learned few from stevedark‌. He can help you on this.

[sarvesh]

Hi..

Please refer below mentioned link..hope it will help !!

https://community.qlik.com/thread/160713

[stevedark]

Thanks Anil for the mention.

If it is not possible to have different apps with one having only recent data (that is used by most people most of the time) and another with all data (which is used rarely by a few people) then you need to look at the granularity of the data.

A column with fewer distinct values will take much less memory and generally perform better.  One simple way to achieve this is to round numbers to a lower number of decimal places (limiting the number of possible values).  Splitting fields is another good way of achieving this (e.g. if you have account codes with 8 digits you could split into two and each field would only have up to 9,999 possible values, rather than one field with up to 99,999,999 possible values).

My blog has a number of performance optimisation posts that you may find helpful.

Where are you seeing the most problems with performance, on loading data or using the app once it is loaded?

What does your data model look like when all data is loaded - fewer tables are better.

Steve

[rwunderlich]

It sounds like you have many "full dual" fields Dual Storage vs Dual Behavior | Qlikview Cookbook

How many bytes of savings is Document Analyzer suggesting you will save if you correct these?  The savings column is not shown in the screenshot you posted.

-Rob

http://masterssummit.com

http://qlikviewcookbook.com

[chhavi376]

Hi Rob,

Yes, that did help.

Thanks