topic Re: QVD Optimization in QlikView

QVD Optimization

chhavi376 — Mon, 03 Apr 2017 07:29:42 GMT

Hi All,

I have 3 Microsoft Excel Comma Separated Values Files(3.5GB each)

all the three files have exactly same columns, but different data.

I want to append the data into a QVD.

the resultant QVD is 4.8GB, and is making the application very heavy(2.3GB).

I am facing a lot of performance issues due to this.

any suggestions?

Thanks

Re: QVD Optimization

rwunderlich — Mon, 03 Apr 2017 07:46:23 GMT

Examine the qvw with

Qlikview Cookbook: QV Document Analyzer http://qlikviewcookbook.com/recipes/download-info/document-analyzer/

You may have some fields that can be removed or optimized.

-Rob

http://masterssummit.com

http://qlikviewcookbook.com

Re: QVD Optimization

chhavi376 — Mon, 03 Apr 2017 08:57:42 GMT

Hi Rob,

I tried this,

In recommendations, I am just getting this for all the fields. I don't this is going to help much

Re: QVD Optimization

avinashelite — Mon, 03 Apr 2017 09:03:38 GMT

how many records you have ?

Do you need all the data to be displayed while login in to app ?

You could force the user to select the filters and then display the data . EG if you 10 years of data then customer doesn't need all 10 years data at same time rite

What's the scheme your following ? START is proffered with the huge data set

Re: QVD Optimization

chhavi376 — Mon, 03 Apr 2017 09:37:53 GMT

around 1.5cr

Yes, We would need all the data

Re: QVD Optimization

Anil_Babu_Samineni — Mon, 03 Apr 2017 10:58:34 GMT

This is the case, I learned few from stevedark‌. He can help you on this.

Re: QVD Optimization

sarvesh — Mon, 03 Apr 2017 11:41:07 GMT

Hi..

Please refer below mentioned link..hope it will help !!

https://community.qlik.com/thread/160713

Re: QVD Optimization

stevedark — Mon, 03 Apr 2017 12:20:47 GMT

Thanks Anil for the mention.

If it is not possible to have different apps with one having only recent data (that is used by most people most of the time) and another with all data (which is used rarely by a few people) then you need to look at the granularity of the data.

A column with fewer distinct values will take much less memory and generally perform better. One simple way to achieve this is to round numbers to a lower number of decimal places (limiting the number of possible values). Splitting fields is another good way of achieving this (e.g. if you have account codes with 8 digits you could split into two and each field would only have up to 9,999 possible values, rather than one field with up to 99,999,999 possible values).

My blog has a number of performance optimisation posts that you may find helpful.

Where are you seeing the most problems with performance, on loading data or using the app once it is loaded?

What does your data model look like when all data is loaded - fewer tables are better.

Steve

Re: QVD Optimization

rwunderlich — Thu, 06 Apr 2017 10:02:16 GMT

It sounds like you have many "full dual" fields Dual Storage vs Dual Behavior | Qlikview Cookbook

How many bytes of savings is Document Analyzer suggesting you will save if you correct these? The savings column is not shown in the screenshot you posted.

-Rob

http://masterssummit.com

http://qlikviewcookbook.com

Re: QVD Optimization

chhavi376 — Mon, 10 Apr 2017 08:50:11 GMT

Hi Rob,

Yes, that did help.

Thanks

Re: QVD Optimization

rwunderlich — Mon, 10 Apr 2017 09:26:21 GMT

Good to hear. Please mark one of the answers as correct to close the thread.

-Rob