I've experienced troubles with new QVC format. I've load old QVD's from qlikview directly in Sense and I didn't notice any issue (12 qvd's around 400 Mb/15M records each, ) everything works fine. (16Gb RAM laptop, Win8.1)
Then I re-stored the QVD in Sense to convert in then new QVC. When I load one qvc only I observe performance gains in reload time ( (about twice faster) but when I tried to load *.qvc, Qlik Sense seems to struggle, reload time are much more slower from the second QVD loaded and finally get completely stuck after trying to concatenate the third QVD (RAM seems to reach limits , RAM usage curve is really unexpected.
I can provide QVDq and/or QVF if it can help. (Same behaviour in Sense 1.1)
By the way, which gains are expected with this new QVC format?
Hi Michael - I ran this by a few people - and we are trying to understand what you mean by QVC. I am not aware of a .qvc file format. I know at one time, .qvc was referred to as this new structure, but never saw the .qvc extension. Do you mean the new format that Qlik Sense stores the data in, which is still technically a QVD file, or something else?
Can you attach your .qvf file (Qlik Sense App) or script code- so we can take a look?
Indeed I'am talking of the new QVD format (I was assuming QVC with "C" for Columnar QVD ?
Scritp is an easy one :
QVD's available here: https://www.dropbox.com/sh/pfunamkjncent6f/AACqdFyyF43s_eQTNAi7sr8za?dl=0
If your findings are confirmed, this is a big big deal. Many implementations are based on that.
Hope to see how this post will end so joining in the conversation.
Actually I was testing it myself, I came across to that "QVC file" terminology.
Attached is the screen.
Also concatenation slows down a little bit not that dramatic.
It maybe related to the type of data you are trying to concatenate. For example, I am testing with same pattern of data (columns are identical).
Are you concatenating tables with different patterns? Maybe that is slower in Sense but since it is hard to replicate the issue (highly dependent of type data) I did not test that.
I tested Sense QVD concat with QlikView QVD, that did not matter much.
Can you please attach the old QVD's from qlikview so that we can compare load performance between Qlikview QVD and QlikSense QVD columnar?
Thankyou for the feedback.
Please find attached log files. They do not tell a lot about what's happening. See that first QVD load is very fast ~30 sec, then second one is 4 minutes ?? then I stop at the third one when memory is 100%, system freezes.
hope it helps,
Wow !!! I was not expecting this !!!!!!
I loaded just one column from your qvd's with this script using First 100 and forcing the concatenate
And was certainly not expecting to end up with 1,500 rows after loading 4 lots of first 100 rows.
Started loading data
Data << Sense_trip_fare_1_optimized (QVD columnar optimized) Lines fetched: 100 Data << Sense_trip_fare_3_optimized (QVD columnar optimized) Lines fetched: 300 Data << Sense_trip_fare_3_optimized (QVD columnar optimized) Lines fetched: 700 Data << Sense_trip_fare_4_optimized (QVD columnar optimized) Lines fetched: 1,500
0 forced error(s)
0 synthetic key(s)