topic Re: Exceeds QVD Size using concatenate in QlikView

Exceeds QVD Size using concatenate

lawrance — Wed, 19 Apr 2017 13:13:14 GMT

Hi,

I have four 11 GB QVD files when I try to concatenate four QVD files into Single QVD but my final QVD contains 200 GB.

Can anyone please provide solution why it takes 200 GB.

Thanks,

Lawrance A

Re: Exceeds QVD Size using concatenate

johnca — Wed, 19 Apr 2017 15:34:02 GMT

What does the data model look like after the concatenation? Did it really make 1 table?

Re: Exceeds QVD Size using concatenate

marcus_sommer — Wed, 19 Apr 2017 17:55:57 GMT

I assume that you expect that to merge 4 qvd with 11 GB should result in a single qvd with about 44 GB. But the data-handling from qlik isn't just adding the data like with text-files else they will be stored in symbol- and data-tables. This could end in a qvd-size which is significantly smaller as the simple summing of 44 GB but also like in your case with a qvd-size which is much larger - caused probably from larger pointers. How it worked could you see here: Symbol Tables and Bit-Stuffed Pointers - a deeper look behind the scenes.

Beside this: do you really mean GB and not MB? By MB I would be not sure if I would merge the qvd's - by GB I wouldn't think on merging them.

- Marcus

Re: Exceeds QVD Size using concatenate

Oleg_Troyansky — Wed, 19 Apr 2017 18:03:47 GMT

In addition to everyone's input - you should look for a couple of things:

- a single field that has a high number of distinct values, perhaps a transaction ID or something like that. This field is likely using a lot of memory, both for symbols and for the index. Consider removing the field if you can or soing something to reduce its cardinality. Worst case - you can use Autonumber in your resulting App to save on the symbol space.

- numeric fields or dates that are being stored as Dual values and use more than 8 bytes of memory for each value.

You can easily find these problems using Rob Wunderlich's Document Analyzer - it's a free tool that you can download here: www.QlikViewCookbook.com.

We teach these and many other performance techniques at the Masters Summit for Qlik. Check our agenda and decide if it might be helpful for you.

Cheers,

Oleg Troyansky

Upgrade your Qlik skills with my book QlikView Your Business: An Expert Guide to Business Discovery with QlikView and Qlik Sense