Qlik Community

Ask a Question

New to Qlik Sense

If you’re new to Qlik Sense, start with this Discussion Board and get up-to-speed quickly.

Announcements
Welcome to our newly redesigned Qlik Community! Read our blog to learn about all the new updates: READ BLOG and REPORTED ISSUES
cancel
Showing results for 
Search instead for 
Did you mean: 
Partner
Partner

Sense Data load : Troubles when concatening QVC

Hi All,

I've experienced troubles with new QVC format. I've load old QVD's from qlikview directly in Sense and I didn't notice any issue (12 qvd's around 400 Mb/15M records each, ) everything works fine. (16Gb RAM laptop, Win8.1)

Then I re-stored the QVD in Sense to convert in then new QVC.  When I load one qvc only I observe performance gains in reload time ( (about twice faster) but when I tried  to load *.qvc, Qlik Sense seems to struggle, reload time are much more slower from the second QVD loaded and finally get completely stuck after trying to concatenate the  third QVD (RAM seems to reach limits , RAM usage curve is really unexpected.

I can provide QVDq and/or QVF if it can help. (Same behaviour in Sense 1.1)

By the way, which gains are expected with this new QVC format?

Michael

9 Replies
Employee
Employee

Hi Michael - I am attempting to find someone internally to look at this for you.

Regards,

Mike Tarallo

Qlik

Regards,
Mike Tarallo
Qlik
Employee
Employee

Hi Michael - I ran this by a few people - and we are trying to understand what you mean by QVC. I am not aware of a .qvc file format. I know at one time, .qvc was referred to as this new structure, but never saw the .qvc extension. Do you mean the new format that Qlik Sense stores the data in, which is still technically a QVD file, or something else?

Can you attach your .qvf file (Qlik Sense App) or script code- so we can take a look?

Regards,

Mike T

Qlik

Regards,
Mike Tarallo
Qlik
Partner
Partner

Hi Michael,

Indeed I'am talking of the new QVD format (I was assuming QVC with "C" for Columnar QVD ?

Scritp is an easy one :

LOAD *

FROM [lib://DATA/Sense_trip_fare_*_optimized.qvd]

(qvd);

QVD's  available here: https://www.dropbox.com/sh/pfunamkjncent6f/AACqdFyyF43s_eQTNAi7sr8za?dl=0

Rgds,

Michael

Partner
Partner

Hello Michael,

If your findings are confirmed, this is a big big deal. Many implementations are based on that.

Hope to see how this post will end so joining in the conversation.

BR

Serhan

Partner
Partner

Hello,

Actually I was testing it myself, I came across to that "QVC file" terminology.

Attached is the screen.

Capture.PNG

Also concatenation slows down a little bit not that dramatic.

It maybe related to the type of data you are trying to concatenate. For example, I am testing with same pattern of data (columns are identical).

Are you concatenating tables with different patterns? Maybe that is slower in Sense but since it is hard to replicate the issue (highly dependent of type data) I did not test that.

I tested Sense QVD concat with QlikView QVD, that did not matter much.

BR

Serhan

Employee
Employee

Hi Michael,

Can you please attach the old QVD's from qlikview so that we can compare load performance between Qlikview QVD and QlikSense QVD columnar?

Thankyou for the feedback.

Regards,

Shankar

Could you share the reload log files ?

Partner
Partner

Hey Bill,

Please find attached log files. They do not tell a lot about what's happening. See that first QVD load is very fast ~30 sec, then second one is 4 minutes ?? then I stop at the third one when memory is 100%, system freezes.

hope it helps,

Michael

Wow !!!  I was not expecting this !!!!!!

I loaded just one column from your qvd's with this script using First 100 and forcing the concatenate

Data :

first 100

LOAD

license_id

FROM [lib://Downloads/Sense_trip_fare_1_optimized.qvd]

(qvd);

Concatenate(Data)

first 100

LOAD

license_id

FROM [lib://Downloads/Sense_trip_fare_3_optimized.qvd]

(qvd);

Concatenate(Data)

first 100

LOAD

license_id

FROM [lib://Downloads/Sense_trip_fare_3_optimized.qvd]

(qvd);

Concatenate(Data)

first 100

LOAD

license_id

FROM [lib://Downloads/Sense_trip_fare_4_optimized.qvd]

(qvd);

And was certainly not expecting to end up with 1,500 rows after loading 4 lots of first 100 rows.

Started loading data

Data << Sense_trip_fare_1_optimized (QVD columnar optimized) Lines fetched: 100 Data << Sense_trip_fare_3_optimized (QVD columnar optimized) Lines fetched: 300 Data << Sense_trip_fare_3_optimized (QVD columnar optimized) Lines fetched: 700 Data << Sense_trip_fare_4_optimized (QVD columnar optimized) Lines fetched: 1,500

App saved

Finished successfully

0 forced error(s)

0 synthetic key(s)