Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hi All,
I've experienced troubles with new QVC format. I've load old QVD's from qlikview directly in Sense and I didn't notice any issue (12 qvd's around 400 Mb/15M records each, ) everything works fine. (16Gb RAM laptop, Win8.1)
Then I re-stored the QVD in Sense to convert in then new QVC. When I load one qvc only I observe performance gains in reload time ( (about twice faster) but when I tried to load *.qvc, Qlik Sense seems to struggle, reload time are much more slower from the second QVD loaded and finally get completely stuck after trying to concatenate the third QVD (RAM seems to reach limits , RAM usage curve is really unexpected.
I can provide QVDq and/or QVF if it can help. (Same behaviour in Sense 1.1)
By the way, which gains are expected with this new QVC format?
Michael
Hi Michael - I am attempting to find someone internally to look at this for you.
Regards,
Mike Tarallo
Qlik
Hi Michael - I ran this by a few people - and we are trying to understand what you mean by QVC. I am not aware of a .qvc file format. I know at one time, .qvc was referred to as this new structure, but never saw the .qvc extension. Do you mean the new format that Qlik Sense stores the data in, which is still technically a QVD file, or something else?
Can you attach your .qvf file (Qlik Sense App) or script code- so we can take a look?
Regards,
Mike T
Qlik
Hi Michael,
Indeed I'am talking of the new QVD format (I was assuming QVC with "C" for Columnar QVD ?
Scritp is an easy one :
LOAD *
FROM [lib://DATA/Sense_trip_fare_*_optimized.qvd]
(qvd);
QVD's available here: https://www.dropbox.com/sh/pfunamkjncent6f/AACqdFyyF43s_eQTNAi7sr8za?dl=0
Rgds,
Michael
Hello Michael,
If your findings are confirmed, this is a big big deal. Many implementations are based on that.
Hope to see how this post will end so joining in the conversation.
BR
Serhan
Hello,
Actually I was testing it myself, I came across to that "QVC file" terminology.
Attached is the screen.
Also concatenation slows down a little bit not that dramatic.
It maybe related to the type of data you are trying to concatenate. For example, I am testing with same pattern of data (columns are identical).
Are you concatenating tables with different patterns? Maybe that is slower in Sense but since it is hard to replicate the issue (highly dependent of type data) I did not test that.
I tested Sense QVD concat with QlikView QVD, that did not matter much.
BR
Serhan
Hi Michael,
Can you please attach the old QVD's from qlikview so that we can compare load performance between Qlikview QVD and QlikSense QVD columnar?
Thankyou for the feedback.
Regards,
Shankar
Could you share the reload log files ?
Hey Bill,
Please find attached log files. They do not tell a lot about what's happening. See that first QVD load is very fast ~30 sec, then second one is 4 minutes ?? then I stop at the third one when memory is 100%, system freezes.
hope it helps,
Michael
Wow !!! I was not expecting this !!!!!!
I loaded just one column from your qvd's with this script using First 100 and forcing the concatenate
Data :
first 100
LOAD
license_id
FROM [lib://Downloads/Sense_trip_fare_1_optimized.qvd]
(qvd);
Concatenate(Data)
first 100
LOAD
license_id
FROM [lib://Downloads/Sense_trip_fare_3_optimized.qvd]
(qvd);
Concatenate(Data)
first 100
LOAD
license_id
FROM [lib://Downloads/Sense_trip_fare_3_optimized.qvd]
(qvd);
Concatenate(Data)
first 100
LOAD
license_id
FROM [lib://Downloads/Sense_trip_fare_4_optimized.qvd]
(qvd);
And was certainly not expecting to end up with 1,500 rows after loading 4 lots of first 100 rows.
Started loading data
Data << Sense_trip_fare_1_optimized (QVD columnar optimized) Lines fetched: 100 Data << Sense_trip_fare_3_optimized (QVD columnar optimized) Lines fetched: 300 Data << Sense_trip_fare_3_optimized (QVD columnar optimized) Lines fetched: 700 Data << Sense_trip_fare_4_optimized (QVD columnar optimized) Lines fetched: 1,500
App saved
Finished successfully
0 forced error(s)
0 synthetic key(s)