Qlik Community

New to Qlik Sense

Discussion board where members can get started with Qlik Sense.

agilos_mla
Contributor III

Sense Data load : Troubles when concatening QVC

Hi All,

I've experienced troubles with new QVC format. I've load old QVD's from qlikview directly in Sense and I didn't notice any issue (12 qvd's around 400 Mb/15M records each, ) everything works fine. (16Gb RAM laptop, Win8.1)

Then I re-stored the QVD in Sense to convert in then new QVC.  When I load one qvc only I observe performance gains in reload time ( (about twice faster) but when I tried  to load *.qvc, Qlik Sense seems to struggle, reload time are much more slower from the second QVD loaded and finally get completely stuck after trying to concatenate the  third QVD (RAM seems to reach limits , RAM usage curve is really unexpected.

I can provide QVDq and/or QVF if it can help. (Same behaviour in Sense 1.1)

By the way, which gains are expected with this new QVC format?

Michael

9 Replies
Employee
Employee

Re: Sense Data load : Troubles when concatening QVC

Hi Michael - I am attempting to find someone internally to look at this for you.

Regards,

Mike Tarallo

Qlik

Employee
Employee

Re: Sense Data load : Troubles when concatening QVC

Hi Michael - I ran this by a few people - and we are trying to understand what you mean by QVC. I am not aware of a .qvc file format. I know at one time, .qvc was referred to as this new structure, but never saw the .qvc extension. Do you mean the new format that Qlik Sense stores the data in, which is still technically a QVD file, or something else?

Can you attach your .qvf file (Qlik Sense App) or script code- so we can take a look?

Regards,

Mike T

Qlik

agilos_mla
Contributor III

Re: Sense Data load : Troubles when concatening QVC

Hi Michael,

Indeed I'am talking of the new QVD format (I was assuming QVC with "C" for Columnar QVD ?

Scritp is an easy one :

LOAD *

FROM [lib://DATA/Sense_trip_fare_*_optimized.qvd]

(qvd);

QVD's  available here: https://www.dropbox.com/sh/pfunamkjncent6f/AACqdFyyF43s_eQTNAi7sr8za?dl=0

Rgds,

Michael

darkhorse
Valued Contributor

Re: Sense Data load : Troubles when concatening QVC

Hello Michael,

If your findings are confirmed, this is a big big deal. Many implementations are based on that.

Hope to see how this post will end so joining in the conversation.

BR

Serhan

darkhorse
Valued Contributor

Re: Sense Data load : Troubles when concatening QVC

Hello,

Actually I was testing it myself, I came across to that "QVC file" terminology.

Attached is the screen.

Capture.PNG

Also concatenation slows down a little bit not that dramatic.

It maybe related to the type of data you are trying to concatenate. For example, I am testing with same pattern of data (columns are identical).

Are you concatenating tables with different patterns? Maybe that is slower in Sense but since it is hard to replicate the issue (highly dependent of type data) I did not test that.

I tested Sense QVD concat with QlikView QVD, that did not matter much.

BR

Serhan

ssb
New Contributor II

Re: Sense Data load : Troubles when concatening QVC

Hi Michael,

Can you please attach the old QVD's from qlikview so that we can compare load performance between Qlikview QVD and QlikSense QVD columnar?

Thankyou for the feedback.

Regards,

Shankar

Re: Sense Data load : Troubles when concatening QVC

Could you share the reload log files ?

agilos_mla
Contributor III

Re: Sense Data load : Troubles when concatening QVC

Hey Bill,

Please find attached log files. They do not tell a lot about what's happening. See that first QVD load is very fast ~30 sec, then second one is 4 minutes ?? then I stop at the third one when memory is 100%, system freezes.

hope it helps,

Michael

Re: Sense Data load : Troubles when concatening QVC

Wow !!!  I was not expecting this !!!!!!

I loaded just one column from your qvd's with this script using First 100 and forcing the concatenate

Data :

first 100

LOAD

license_id

FROM [lib://Downloads/Sense_trip_fare_1_optimized.qvd]

(qvd);

Concatenate(Data)

first 100

LOAD

license_id

FROM [lib://Downloads/Sense_trip_fare_3_optimized.qvd]

(qvd);

Concatenate(Data)

first 100

LOAD

license_id

FROM [lib://Downloads/Sense_trip_fare_3_optimized.qvd]

(qvd);

Concatenate(Data)

first 100

LOAD

license_id

FROM [lib://Downloads/Sense_trip_fare_4_optimized.qvd]

(qvd);

And was certainly not expecting to end up with 1,500 rows after loading 4 lots of first 100 rows.

Started loading data

Data << Sense_trip_fare_1_optimized (QVD columnar optimized) Lines fetched: 100 Data << Sense_trip_fare_3_optimized (QVD columnar optimized) Lines fetched: 300 Data << Sense_trip_fare_3_optimized (QVD columnar optimized) Lines fetched: 700 Data << Sense_trip_fare_4_optimized (QVD columnar optimized) Lines fetched: 1,500

App saved

Finished successfully

0 forced error(s)

0 synthetic key(s)