Qlik Community

QlikView Scripting

Discussion Board for collaboration on QlikView Scripting.

Announcements

Breathe easy -- you now have more time to plan your next steps with Qlik!
QlikView 11.2 Extended Support is now valid through December 31, 2020. Click here for more information.

sasikanth
Not applicable

QVD Compression Technique


HI All,

Can anybody please explain the technique behind QVD compression,

How it is fast when fetching the data from QVD than database?

Thanks

9 Replies
Clever_Anjos
Not applicable

Re: QVD Compression Technique

First of all, QlikView will remove any duplicates of each field creating a symbol table.

Then numeric values are stored into binary form.

The answer to your second question is depends how fast your database can be.

I´ve seen 100x qvd x database ratio

sundarakumar
Not applicable

Re: QVD Compression Technique

If u create a qvd the original data size will be compressed upto 10 times...

This is done by associative property.

The data is actually saved only once in all othyer places where it should be present is denoted by a pointer...

-Sundar


sasikanth
Not applicable

Re: QVD Compression Technique

hi sundar

Can you please explain me how that is done by Associative property?

sundarakumar
Not applicable

Re: Re: QVD Compression Technique

PFA,

Please see page 8 and 9, they have explaine hoe data is compressed in qlikview.

sundarakumar
Not applicable

Re: Re: QVD Compression Technique

Please go thro the below link for more info

http://www.quickintelligence.co.uk/qlikview-qvd-files/

Qvd file stores data in the format how data is handled in qlikview, so it is the best way to store and to be used.

hope this helps..

-Sundar

sundarakumar
Not applicable

Re: QVD Compression Technique

neetha_p
Not applicable

Re: QVD Compression Technique

Hi Sasikanth,

The way QlikView stores  the data internally that makes difference,fetching data from QVD than Database.

Below example gives accurate picture of how it is done:

If I load a very simple dataset:

Fruit:
LOAD *
INLINE [
Fruit, Color
Banana, Yellow
Blueberry, Blue
Tomato, Red
Tomato, Green
];
And then add both fields as list boxes, I see that there are only 3 fruit listed - even though I have loaded 4 entries. "Tomato" is only stored once by QlikView. Internally, there are binary keys keeping track of the association between "Tomato" along with "Red" and "Green".

In a large data set with a lot of repeating values (especially text values) the fact that each field value is only stored once means that there is a huge saving in the amount of memory needed to store the information. This is why QlikView can store so much data in memory.

Let's add another table of data:Colors:
LOAD * INLINE [
Color, Red, Green, Blue
Yellow, 255, 255, 0
Blue, 0, 0, 255
Red, 255, 0, 0
Green, 0, 255, 0
Amber, 255, 121, 0
];

I now might think that I have 2 tables of data - both of which have a field called "Color". But I don't! There is only one field called "Color" in our dataset and this field will have 5 values. Internally, the association between each of them and their associated values are maintained with binary keys. It just so happens that one of the values - "Amber" - is not associated with any of the values in the "Fruit" field.

Note that there is probably no direct association between "Fruit" and "Red". The association is indirect. However, because of the smaller data set in memory and the binary keys, the association is very fast.

chiru_thota
Not applicable

Re: QVD Compression Technique

Hi,

(1) Attached PPT will clearly explain how data is stored in QVD

(2) It fast because QVD is in Qlikview native format unlike database table

Thanks,

Chiru

Storage1.jpg

neetha_p
Not applicable

Re: QVD Compression Technique

Hi Sasikanth,

Please check below link for document compression:

http://qlikviewnotes.blogspot.co.uk/2011/03/qlikview-compression.html