Data Compression between QlikSense and QlikView

[Arthur_Fong]

Hi all,

I notice something strange when I tried to load the same QVD files into QV and QS.

The qvw file size after loading the QVD files is 30MB, whereas qvf is only 10MB. (3x lesser than QV)

I looked through the post by Henric, but it says both engine are the same

(which means the file size should be the same is it?):

https://community.qlik.com/t5/Qlik-Design-Blog/Symbol-Tables-and-Bit-Stuffed-Pointers/ba-p/1475369/p...

Is there other reasons for these file compression to happen?

Thanks and regards,

Arthur Fong

[Arthur_Fong]

This data compression doesn't seems to happen when I autogenerated 10M row of records, stored and loaded the QVD into both QVW and QVF (both file size are quite similar).

[jonathandienst]

Qlikview has an option to select the file compression applied to the QVW files (uncompressed,, medium compression, high compression). I don't recall that as an option in QlikSense, but you can play with the options in Qlikview (on the Document Properties default tab).

The QIX engine may be the same in both, but the file loading and saving is not the same logic. Also bear in mind that both qvw and qvf files contain more than just the data,

[Arthur_Fong]

Thanks Jon, Is there any impact on performance wise if we select high compression option?

I mean why is this an option since data compression should be the higher the better isn't it?

[jonathandienst]

Data compression is good for reducing storage space and network load, but it comes at a performance cost - file saves take longer due to the work of compressing during saving and (to a lesser extent) expanding when opening the file. So YMMV, and you have the option of selecting the method that best suits your disk and network environment.

It has no effect on the performance or RAM requirement of the document once loaded.

[Arthur_Fong]

Seems like QS has much more data compression as I tested this with QV data compression option set as high already.

[jonathandienst]

Or the QVW file needs more space for objects other than data -- sheets, tables, charts, variables, groups, alt states and expressions...

[Arthur_Fong]

I did this comparison using Data Model layer. Front End shouldn't impact much.

[Arthur_Fong]

I came out with a test sample.

Refer attached.

[Brett_Bleess]

Hey guys, I may have an answer, took it awhile to pop into my head, but the delta is most likely that QVW files are still row-based, whereas I believe QVF files are likely column based from what I recall, so that is likely the underlying reason for things, but I am not sure how to confirm it! 🙂  The way things work in QlikView at this point is we open the QVW, row-based, into memory, we convert that data model to column-based in memory and once done we drop the row-based model from memory, that is why QV.exe or QVS.exe spikes when you first open QVW files now.  The reverse has to happen when you save things too, just FYI.  Best I have, if you do not think this should make any difference, let me know, and I can see if I can get Henric to swing by on this one and take a read and see if he can shed any further light upon things for us, but I suspect I am on the right track here.  

Regards,
Brett