Qlik Community

Ask a Question

New to QlikView

If you’re new to QlikView, start with this Discussion Board and get up-to-speed quickly.

Announcements
Qlik Highlights 2020 Giveaway! Watch, reply and have a chance to win a $200 Amazon Gift Card! Watch Video
cancel
Showing results for 
Search instead for 
Did you mean: 
Not applicable

Size of the Qvd

Hi

suppose i am stroing 30 MB table into Qvd what can be size of Qvd is there any specific formulae for this.

Regards

1 Solution

Accepted Solutions
MVP & Luminary
MVP & Luminary

HI,

Consider a key field consisting of potentially 550,000 distinct values in a fact table containing 100 million rows of data.

To calculate the storage required for this field, we need to consider the list storage as well as the table pointer storage.

Remember that these are minimum amounts, since there is, of course, additional storage that QlikView will require to maintain the data.

For the list data, let's assume that each value is a 10 byte character (text) representation.

If we simply multiple 10 bytes by 550,000, we will get to approximately 5.5 MB. This will be the requirement to store the list, regardless of the number of times this field is referenced in logical tables.

Next, we need to calculate the pointer requirements for the logical table including this field. Since 550,000 distinct values can be represented by 20 bits (2 to the 20th power), we will need a pointer size of 2.5 bytes for each row of non-null data.

As mentioned above, we have 100 million rows of data in our fact table. Since 100 million times 2.5 is 250 million, we will need approximately 250 MB to store the pointers for this field in the logical table.

So, total storage required for this single field in one logical table would be approximately 256 MB. Since this is a key field, by definition, it also exists in one or more additional logical tables. If we assume a dimension table with a single row of data for each key value, then we need to add an additional 1.4 MB for the dimension table pointer storage. This brings our total to about 258 MB for this single field, almost all of which is pointer storage.

Of course, in documents with small fact tables (less than 1 million rows), this is not as much of an issue. But when dealing with large fact tables, you need to be aware of the repercussions of adding and deleting fields


As mentioned above, we have 100 million rows of data in our fact table. Since 100 million times 2.5 is 250 million, we will need approximately 250 MB to store the pointers for this field in the logical table.

So, total storage required for this single field in one logical table would be approximately 256 MB. Since this is a key field, by definition, it also exists in one or more additional logical tables. If we assume a dimension table with a single row of data for each key value, then we need to add an additional 1.4 MB for the dimension table pointer storage. This brings our total to about 258 MB for this single field, almost all of which is pointer storage.

Of course, in documents with small fact tables (less than 1 million rows), this is not as much of an issue. But when dealing with large fact tables, you need to be aware of the repercussions of adding and deleting fields

Regards,Jagan.

View solution in original post

6 Replies
Master II
Master II

Hi,

There is no specific formula for compression or storing data into QVD.

It totally depend on your type of data..

Suppose you have all unique records in your data set then there will not be any compression..

and if you have duplicate values in your data set then there will be some compression..

Because qlikview stores duplicate values as pointers..

HTH

Sushil

MVP & Luminary
MVP & Luminary

Hi,

Size of the QVD completely depends on the number of unique values in the dataset, In QVD only one time the value is stored and pointers are used to refer the values afterwards.

Regards,

Jagan.

MVP & Luminary
MVP & Luminary

Hi,

Check this hope it helps you.

Re: QVD Compression Technique

Regards,

Jagan.

Master II
Master II

Go through this link below:

Data Compression in QlikView

HTH

Sushil

Not applicable

Hi jagan,

is there any specific formula for this counting unique values and size.

in your experience suppose i have 30 Mb file with out any duplicates then how it  Qvd size will be.

MVP & Luminary
MVP & Luminary

HI,

Consider a key field consisting of potentially 550,000 distinct values in a fact table containing 100 million rows of data.

To calculate the storage required for this field, we need to consider the list storage as well as the table pointer storage.

Remember that these are minimum amounts, since there is, of course, additional storage that QlikView will require to maintain the data.

For the list data, let's assume that each value is a 10 byte character (text) representation.

If we simply multiple 10 bytes by 550,000, we will get to approximately 5.5 MB. This will be the requirement to store the list, regardless of the number of times this field is referenced in logical tables.

Next, we need to calculate the pointer requirements for the logical table including this field. Since 550,000 distinct values can be represented by 20 bits (2 to the 20th power), we will need a pointer size of 2.5 bytes for each row of non-null data.

As mentioned above, we have 100 million rows of data in our fact table. Since 100 million times 2.5 is 250 million, we will need approximately 250 MB to store the pointers for this field in the logical table.

So, total storage required for this single field in one logical table would be approximately 256 MB. Since this is a key field, by definition, it also exists in one or more additional logical tables. If we assume a dimension table with a single row of data for each key value, then we need to add an additional 1.4 MB for the dimension table pointer storage. This brings our total to about 258 MB for this single field, almost all of which is pointer storage.

Of course, in documents with small fact tables (less than 1 million rows), this is not as much of an issue. But when dealing with large fact tables, you need to be aware of the repercussions of adding and deleting fields


As mentioned above, we have 100 million rows of data in our fact table. Since 100 million times 2.5 is 250 million, we will need approximately 250 MB to store the pointers for this field in the logical table.

So, total storage required for this single field in one logical table would be approximately 256 MB. Since this is a key field, by definition, it also exists in one or more additional logical tables. If we assume a dimension table with a single row of data for each key value, then we need to add an additional 1.4 MB for the dimension table pointer storage. This brings our total to about 258 MB for this single field, almost all of which is pointer storage.

Of course, in documents with small fact tables (less than 1 million rows), this is not as much of an issue. But when dealing with large fact tables, you need to be aware of the repercussions of adding and deleting fields

Regards,Jagan.

View solution in original post