Qlik Sense takes almost three times longer to reload than QlikView

[gmoraleswit]

Hello,

I have a quite complex script I'm running on QlikSense and I noticed that it takes longer to reload (almost three times more) than in QlikView, I'm reloading both on the same machine.

In QlikView I don't generate a log file since it is a long script It took a long time to save the file. I tried to find a way to disable the option to generate the log file in QlikSense but I couldn't find any, is this even possible?

Thank you in advance

[Anonymous]

Yup, that RESIDENT LOAD now takes 30 secs :

///////////////////////////////////////////////////////////////////////////

let vNow = now() ;

trace $(vNow) ;

///////////////////////////////////////////////////////////////////////////

NoConcatenate Flights :

///////////////////////////////////////////////////////////////////////////

load

  *

resident Temp01Flights

where

  MonthName (  [Scheduled Departure Date LT] ) = MonthName(Today())

  or

  MonthName (  [Scheduled Departure Date LT] ) = MonthName ( AddMonths ( Today() , -1 ) )

;

///////////////////////////////////////////////////////////////////////////

drop table Temp01Flights ;

///////////////////////////////////////////////////////////////////////////

///////////////////////////////////////////////////////////////////////////

let vNow = now() ;

trace $(vNow) ;

///////////////////////////////////////////////////////////////////////////

exit script ;

///////////////////////////////////////////////////////////////////////////

Started loading data

Variables << Variables Lines fetched: 22 Calendar << Generic.Calendar Lines fetched: 62 18 Aug 2014 18:39:59 Temp.EZY.OTP << AUTOGENERATE(1) Lines fetched: 1 18 Aug 2014 18:39:59 Temp01Flights << RSMS.RSMSLIVE.Flights (QVD columnar optimized)

Lines fetched: 4,526,720

18 Aug 2014 18:40:11

18 Aug 2014 18:40:14 Flights << Temp01Flights

Lines fetched: 229,274

18 Aug 2014 18:40:44

App saved

Finished successfully

0 forced error(s)

0 synthetic key(s)

[rwunderlich]

" I don't generate a log file since it is a long script It took a long time to save the file"

I'm curious about that observation. Are you saying that using a document log added a measurable increase to load time? How much?

-Rob

[gmoraleswit]

Hi Rob,

I don't even know how much more time it takes, when I enabled to generate the log file and then reloaded the document I stopped the reload when it reached double the time it takes without generating the log file (30 min without log, more than 1 hr with log) that is for QlikView 11.2 SR5.

[tseebach]

Bill,

My test shows the same results. Loaded 20 mio rows on the same computer with Sense and QlikView.

QlikView: 6,00000014529 seconds

Sense: 14,00000012945 seconds

In both loads, I have a variance of about a 1 plus/minus.

Since Sense autosaves, I've done my timing using this script, which is accurate beyond nanoseconds:

Let _StartTime = Now();

Let _TotalTime = Round((Now()-_StartTime)*60*60*24,0.00000000001);
Trace $(_CalendarName) created in: $(_TotalTime) seconds;

Loading the same data, just generated by Sense took just 6,99999975041 seconds in Sense.

QlikView however cannot read the QVD that Sense has generated!

This is really bad for hybrid (QlikView+Sense) deployments. It will basically force us to load every table twice.

[tseebach]

Gabriela, in QlikView 11.2 SR7 I could not find a measurable time difference when logfiles was enabled or disabled.

[Alexander_Thor]

Torben Seebach wrote:

QlikView however cannot read the QVD that Sense has generated!

This is really bad for hybrid (QlikView+Sense) deployments. It will basically force us to load every table twice.

Yeah for now the recommendation, once the Sense server goes GA that is, for customers running both products side by side is to generate their QVD-layer through QlikView. However down the line it would be preferable to have both products share the same internal format that is columnar instead of row based.

[tseebach]

Alexander and Michael, maybe should tell PM to stop calling the Sense QVDs for QVDs, because if they cannot be read by a program that reads QVDs (QlikView)  then they are not QVDs. That is like calling XLSX as XLS.

It would make sense that Sense could write a new optimized format called QVDX or something, but also write out standard QVDs so they could be loaded by QlikView. Hybrid deployments are going to be a huge mess if this is not sorted - maybe we should start using txt files again.

[Michael_Tarallo]

Hello Torben I have informed PM.

Mike

[gmoraleswit]

I made the same test, I converted the QVD's using the 'store table into..' in Sense, then I ran the script and it was almost 10 times slower than QlikView

Sense (using Sense QVD's): 5hr 53min

Sense (using QlikView QVD's): 1hr 32min

QlikView: 38min

Are we doing something wrong?

[Anonymous]

Gabriella

Do you use WHERE EXISTS on any loads ?

See thread Where Exists on loading from QVD ?that I opened this morning.

Best Regards,     Bill