topic Re: Data Compression between QlikSense and QlikView in QlikView

Data Compression between QlikSense and QlikView

Arthur_Fong — Wed, 25 Nov 2020 16:16:04 GMT

Hi all,

I notice something strange when I tried to load the same QVD files into QV and QS.

The qvw file size after loading the QVD files is 30MB, whereas qvf is only 10MB. (3x lesser than QV)

I looked through the post by Henric, but it says both engine are the same

(which means the file size should be the same is it?):

https://community.qlik.com/t5/Qlik-Design-Blog/Symbol-Tables-and-Bit-Stuffed-Pointers/ba-p/1475369/page/3

Is there other reasons for these file compression to happen?

Thanks and regards,

Arthur Fong

Re: Data Compression between QlikSense and QlikView

Arthur_Fong — Mon, 10 Feb 2020 09:17:32 GMT

This data compression doesn't seems to happen when I autogenerated 10M row of records, stored and loaded the QVD into both QVW and QVF (both file size are quite similar).

Re: Data Compression between QlikSense and QlikView

jonathandienst — Mon, 10 Feb 2020 09:55:40 GMT

Qlikview has an option to select the file compression applied to the QVW files (uncompressed,, medium compression, high compression). I don't recall that as an option in QlikSense, but you can play with the options in Qlikview (on the Document Properties default tab).

The QIX engine may be the same in both, but the file loading and saving is not the same logic. Also bear in mind that both qvw and qvf files contain more than just the data,

Re: Data Compression between QlikSense and QlikView

Arthur_Fong — Mon, 10 Feb 2020 09:58:57 GMT

Thanks Jon, Is there any impact on performance wise if we select high compression option?

I mean why is this an option since data compression should be the higher the better isn't it?

Re: Data Compression between QlikSense and QlikView

jonathandienst — Mon, 10 Feb 2020 12:15:58 GMT

Data compression is good for reducing storage space and network load, but it comes at a performance cost - file saves take longer due to the work of compressing during saving and (to a lesser extent) expanding when opening the file. So YMMV, and you have the option of selecting the method that best suits your disk and network environment.

It has no effect on the performance or RAM requirement of the document once loaded.

Re: Data Compression between QlikSense and QlikView

Arthur_Fong — Mon, 10 Feb 2020 14:15:13 GMT

Seems like QS has much more data compression as I tested this with QV data compression option set as high already.

Re: Data Compression between QlikSense and QlikView

jonathandienst — Tue, 11 Feb 2020 05:32:24 GMT

Or the QVW file needs more space for objects other than data -- sheets, tables, charts, variables, groups, alt states and expressions...

Re: Data Compression between QlikSense and QlikView

Arthur_Fong — Tue, 11 Feb 2020 05:34:05 GMT

I did this comparison using Data Model layer. Front End shouldn't impact much.

Re: Data Compression between QlikSense and QlikView

Arthur_Fong — Tue, 11 Feb 2020 08:49:03 GMT

I came out with a test sample.

Refer attached.

Re: Data Compression between QlikSense and QlikView

Brett_Bleess — Thu, 05 Mar 2020 19:34:44 GMT

Hey guys, I may have an answer, took it awhile to pop into my head, but the delta is most likely that QVW files are still row-based, whereas I believe QVF files are likely column based from what I recall, so that is likely the underlying reason for things, but I am not sure how to confirm it! 🙂 The way things work in QlikView at this point is we open the QVW, row-based, into memory, we convert that data model to column-based in memory and once done we drop the row-based model from memory, that is why QV.exe or QVS.exe spikes when you first open QVW files now. The reverse has to happen when you save things too, just FYI. Best I have, if you do not think this should make any difference, let me know, and I can see if I can get Henric to swing by on this one and take a read and see if he can shed any further light upon things for us, but I suspect I am on the right track here.

Regards,
Brett

Re: Data Compression between QlikSense and QlikView

Arthur_Fong — Fri, 06 Mar 2020 00:55:21 GMT

Thanks Brett,

That explains the spikes when we open large qvw files.

However the topic we are discussing here is about the delta of file size between qvf and qvw, loaded with the same qvd.

I would need further explanation on this so that I did not pass the wrong info to my client.

Thanks and regards,
Arthur Fong

Re: Data Compression between QlikSense and QlikView

Brett_Bleess — Tue, 10 Mar 2020 19:02:45 GMT

Let me see if I can get Henric C. to have a look Arthur! I am pretty sure I am on the right track, but it will be good to know for sure, as I think .qvf is column-based and qvw are row-based still. Might take a day or two or three for Henric to get to things...

Regards,
Brett

Re: Data Compression between QlikSense and QlikView

Arthur_Fong — Wed, 11 Mar 2020 08:16:09 GMT

Sure Brett,

Thanks for looking into this.

Your help are much appreciated!

Re: Data Compression between QlikSense and QlikView

Brett_Bleess — Wed, 11 Mar 2020 11:45:35 GMT

Arthur, I did find another post in Community from a few years ago that might explain a little more:

https://community.qlik.com/t5/New-to-Qlik-Sense/qvf-file-size-is-half-of-qvw/m-p/840691

Regards,
Brett

Re: Data Compression between QlikSense and QlikView

Arthur_Fong — Wed, 11 Mar 2020 11:59:13 GMT

I did saw this post before opening a new post.

But didn't find it convincing to get a full picture of what is happening behind the scene.