Exceeds QVD Size using concatenate

[lawrance]

Hi,

I have four 11 GB QVD files when I try to concatenate four QVD files into Single QVD but my final QVD contains 200 GB.

Can anyone please provide solution why it takes 200 GB.

Thanks,

Lawrance A

[johnca]

What does the data model look like after the concatenation? Did it really make 1 table?

[marcus_sommer]

I assume that you expect that to merge 4 qvd with 11 GB should result in a single qvd with about 44 GB. But the data-handling from qlik isn't just adding the data like with text-files else they will be stored in symbol- and data-tables. This could end in a qvd-size which is significantly smaller as the simple summing of 44 GB but also like in your case with a qvd-size which is much larger - caused probably from larger pointers. How it worked could you see here: Symbol Tables and Bit-Stuffed Pointers - a deeper look behind the scenes.

Beside this: do you really mean GB and not MB? By MB I would be not sure if I would merge the qvd's - by GB I wouldn't think on merging them.

- Marcus

[Oleg_Troyansky]

In addition to everyone's input - you should look for a couple of things:

- a single field that has a high number of distinct values, perhaps a transaction ID or something like that. This field is likely using a lot of memory, both for symbols and for the index. Consider removing the field if you can or soing something to reduce its cardinality. Worst case - you can use Autonumber in your resulting App to save on the symbol space.

- numeric fields or dates that are being stored as Dual values and use more than 8 bytes of memory for each value.

You can easily find these problems using Rob Wunderlich's Document Analyzer - it's a free tool that you can download here: www.QlikViewCookbook.com.

We teach these and many other performance techniques at the Masters Summit for Qlik. Check our agenda and decide if it might be helpful for you.

Cheers,

Oleg Troyansky

Upgrade your Qlik skills with my book QlikView Your Business: An Expert Guide to Business Discovery with QlikView and Qlik Sense