Qlik Community

Ask a Question

QlikView App Dev

Discussion Board for collaboration related to QlikView App Development.

Announcements
Join this live chat April 6, 10AM EST - QlikView to Qlik Sense REGISTER
cancel
Showing results for 
Search instead for 
Did you mean: 
Arthur_Fong
Partner
Partner

Data Compression between QlikSense and QlikView

Hi all,

I notice something strange when I tried to load the same QVD files into QV and QS.

The qvw file size after loading the QVD files is 30MB, whereas qvf is only 10MB. (3x lesser than QV)

I looked through the post by Henric, but it says both engine are the same

(which means the file size should be the same is it?):

https://community.qlik.com/t5/Qlik-Design-Blog/Symbol-Tables-and-Bit-Stuffed-Pointers/ba-p/1475369/p...

Is there other reasons for these file compression to happen?

 

Thanks and regards,

Arthur Fong

14 Replies
Arthur_Fong
Partner
Partner
Author

This data compression doesn't seems to happen when I autogenerated 10M row of records, stored and loaded the QVD into both QVW and QVF (both file size are quite similar).

jonathandienst

Qlikview has an option to select the file compression applied to the QVW files (uncompressed,, medium compression, high compression). I don't recall that as an option in QlikSense, but you can play with the options in Qlikview (on the Document Properties default tab).

The QIX engine may be the same in both, but the file loading and saving is not the same logic. Also bear in mind that both qvw and qvf files contain more than just the data,

Logic will get you from a to b. Imagination will take you everywhere. - A Einstein
Arthur_Fong
Partner
Partner
Author

Thanks Jon, Is there any impact on performance wise if we select high compression option?

I mean why is this an option since data compression should be the higher the better isn't it?

jonathandienst

Data compression is good for reducing storage space and network load, but it comes at a performance cost - file saves take longer due to the work of compressing during saving and (to a lesser extent) expanding when opening the file. So YMMV, and you have the option of selecting the method that best suits your disk and network environment.

It has no effect on the performance or RAM requirement of the document once loaded.

Logic will get you from a to b. Imagination will take you everywhere. - A Einstein
Arthur_Fong
Partner
Partner
Author

Seems like QS has much more data compression as I tested this with QV data compression option set as high already.

 

jonathandienst

Or the QVW file needs more space for objects other than data -- sheets, tables, charts, variables, groups, alt states and expressions...

Logic will get you from a to b. Imagination will take you everywhere. - A Einstein
Arthur_Fong
Partner
Partner
Author

I did this comparison using Data Model layer. Front End shouldn't impact much.

Arthur_Fong
Partner
Partner
Author

I came out with a test sample.

Refer attached.

Brett_Bleess
Support (Former)
Support (Former)

Hey guys, I may have an answer, took it awhile to pop into my head, but the delta is most likely that QVW files are still row-based, whereas I believe QVF files are likely column based from what I recall, so that is likely the underlying reason for things, but I am not sure how to confirm it! 🙂  The way things work in QlikView at this point is we open the QVW, row-based, into memory, we convert that data model to column-based in memory and once done we drop the row-based model from memory, that is why QV.exe or QVS.exe spikes when you first open QVW files now.  The reverse has to happen when you save things too, just FYI.  Best I have, if you do not think this should make any difference, let me know, and I can see if I can get Henric to swing by on this one and take a read and see if he can shed any further light upon things for us, but I suspect I am on the right track here.  

Regards,
Brett

To help users find verified answers, please do not forget to use the "Accept as Solution" button on any post(s) that helped you resolve your problem or question.
I now work a compressed schedule, Tuesday, Wednesday and Thursday, so those will be the days I will reply to any follow-up posts.