Skip to main content
Announcements
Join us at Qlik Connect for 3 magical days of learning, networking,and inspiration! REGISTER TODAY and save!
cancel
Showing results for 
Search instead for 
Did you mean: 
sasikanth
Master
Master

QVD Compression Technique


HI All,

Can anybody please explain the technique behind QVD compression,

How it is fast when fetching the data from QVD than database?

Thanks

9 Replies
Clever_Anjos
Employee
Employee

First of all, QlikView will remove any duplicates of each field creating a symbol table.

Then numeric values are stored into binary form.

The answer to your second question is depends how fast your database can be.

I´ve seen 100x qvd x database ratio

sundarakumar
Specialist II
Specialist II

If u create a qvd the original data size will be compressed upto 10 times...

This is done by associative property.

The data is actually saved only once in all othyer places where it should be present is denoted by a pointer...

-Sundar


sasikanth
Master
Master
Author

hi sundar

Can you please explain me how that is done by Associative property?

sundarakumar
Specialist II
Specialist II

PFA,

Please see page 8 and 9, they have explaine hoe data is compressed in qlikview.

sundarakumar
Specialist II
Specialist II

Please go thro the below link for more info

http://www.quickintelligence.co.uk/qlikview-qvd-files/

Qvd file stores data in the format how data is handled in qlikview, so it is the best way to store and to be used.

hope this helps..

-Sundar

Anonymous
Not applicable

Hi Sasikanth,

The way QlikView stores  the data internally that makes difference,fetching data from QVD than Database.

Below example gives accurate picture of how it is done:

If I load a very simple dataset:

Fruit:
LOAD *
INLINE [
Fruit, Color
Banana, Yellow
Blueberry, Blue
Tomato, Red
Tomato, Green
];
And then add both fields as list boxes, I see that there are only 3 fruit listed - even though I have loaded 4 entries. "Tomato" is only stored once by QlikView. Internally, there are binary keys keeping track of the association between "Tomato" along with "Red" and "Green".

In a large data set with a lot of repeating values (especially text values) the fact that each field value is only stored once means that there is a huge saving in the amount of memory needed to store the information. This is why QlikView can store so much data in memory.

Let's add another table of data:Colors:
LOAD * INLINE [
Color, Red, Green, Blue
Yellow, 255, 255, 0
Blue, 0, 0, 255
Red, 255, 0, 0
Green, 0, 255, 0
Amber, 255, 121, 0
];

I now might think that I have 2 tables of data - both of which have a field called "Color". But I don't! There is only one field called "Color" in our dataset and this field will have 5 values. Internally, the association between each of them and their associated values are maintained with binary keys. It just so happens that one of the values - "Amber" - is not associated with any of the values in the "Fruit" field.

Note that there is probably no direct association between "Fruit" and "Red". The association is indirect. However, because of the smaller data set in memory and the binary keys, the association is very fast.

chiru_thota
Specialist
Specialist

Hi,

(1) Attached PPT will clearly explain how data is stored in QVD

(2) It fast because QVD is in Qlikview native format unlike database table

Thanks,

Chiru

Storage1.jpg

Anonymous
Not applicable

Hi Sasikanth,

Please check below link for document compression:

http://qlikviewnotes.blogspot.co.uk/2011/03/qlikview-compression.html