topic Re: Re: Problem to STORE QVD in QlikView

Problem to STORE QVD

Wed, 20 Nov 2013 22:39:30 GMT

Hi Everybody

I have a question about sentences STORE .

I Concatenate three QVD's (10 , 5 and 4 GB) and created a new QVD. In resulted i hoped a QVD around of 20 GB but i get a QVD of 50 GB. The Scrip is correct and the data too.

This is correct? (50GB)

Thanks!

Re: Problem to STORE QVD

Wed, 20 Nov 2013 23:36:00 GMT

Hi Richard,

Resultant QVD size really depends on nature of your data. Size is dependent on whether your data is loaded as optimized or unoptimized and joins used. If you can explain ur scenario I'll be able to help with some solution.

Re: Problem to STORE QVD

Wed, 20 Nov 2013 23:58:41 GMT

Hi Richard,

Show to us the script please.

Re: Problem to STORE QVD

Thu, 21 Nov 2013 00:03:06 GMT

Are you doing any transformations on the data like adding some new fields or joining the tables.

Re: Problem to STORE QVD

Thu, 21 Nov 2013 00:16:49 GMT

The script is like this...

Thanks a lot

Re: Problem to STORE QVD

Thu, 21 Nov 2013 00:21:48 GMT

Your script looks like clean and I believe the size of qvd will depend on the no of rows.

Re: Re: Problem to STORE QVD

Thu, 21 Nov 2013 00:26:26 GMT

Richard,

There are two suggestions based on what I understood...

Try using Joins instead of concatenate: Concatenate is useful to load chunks of data from multiple tables into one resultant when they have same Column names or almost same columns. Search for posts on differences for concatenate and joins
If your tables are having values in a Matrix format try to convert it into straight table using CROSSTABLE and observe data size

Hope it helps

Re: Problem to STORE QVD

Clever_Anjos — Thu, 21 Nov 2013 00:57:44 GMT

If your concatenated tables have very different fields, this issue can shows up.

A QVD is a very specialized file, containing:

1) A XML header in plain text, that describes all fields and another information (Creation Date, qvw filename that stored the qvd, the 'lineage' of your data, and so on)

2) A 'Symbol Table' which stores all different simbols that are present at your data, converted to binary and optimized

3) A indexed compressed colection of rows (one to each row of your raw data)

This 'smart' format is the key feature that leads to very impressive load time when you do an optimized load from a QVD

If you´re original tables have much more columns, this will impact step 3 what can lead to that issue you have noticed

Re: Problem to STORE QVD

Fri, 22 Nov 2013 19:36:42 GMT

Hi Anjos

I worked for each QVD had the same no of fields and the same kind of data

Now when load the QVD's files i do not have this issue

But the result is the same

Re: Problem to STORE QVD

Clever_Anjos — Fri, 22 Nov 2013 19:40:55 GMT

You have done 'by hand' what QlikView did by itself.

QVD´s are not so 1:1 proportion, it´s not assured that twice the data, twice the size, it all depends of data

Re: Problem to STORE QVD

Fri, 22 Nov 2013 19:59:40 GMT

So, the incremental of size of my QVD is it correct?

Previously of this, i made similar works but never the result was too big

Re: Problem to STORE QVD

Clever_Anjos — Sat, 23 Nov 2013 19:01:51 GMT

It´s hard to check without knowing your data, but probably is correct indeed