Qlik Community

New to Qlik Sense

Discussion board where members can get started with Qlik Sense.

Announcements
Please visit COVID-19 group to keep up the discussion. GO TO GROUP
Highlighted

Re: Re: Re: Qlik Sense takes almost three times longer to reload than QlikView

Yup, that RESIDENT LOAD now takes 30 secs :

///////////////////////////////////////////////////////////////////////////

let vNow = now() ;

trace $(vNow) ;

///////////////////////////////////////////////////////////////////////////

NoConcatenate Flights :

///////////////////////////////////////////////////////////////////////////

load

  *

resident Temp01Flights

where

  MonthName (  [Scheduled Departure Date LT] ) = MonthName(Today())

  or

  MonthName (  [Scheduled Departure Date LT] ) = MonthName ( AddMonths ( Today() , -1 ) )

;

///////////////////////////////////////////////////////////////////////////

drop table Temp01Flights ;

///////////////////////////////////////////////////////////////////////////

///////////////////////////////////////////////////////////////////////////

let vNow = now() ;

trace $(vNow) ;

///////////////////////////////////////////////////////////////////////////

exit script ;

///////////////////////////////////////////////////////////////////////////

Started loading data

Variables << Variables Lines fetched: 22 Calendar << Generic.Calendar Lines fetched: 62 18 Aug 2014 18:39:59 Temp.EZY.OTP << AUTOGENERATE(1) Lines fetched: 1 18 Aug 2014 18:39:59 Temp01Flights << RSMS.RSMSLIVE.Flights (QVD columnar optimized)

Lines fetched: 4,526,720

18 Aug 2014 18:40:11

18 Aug 2014 18:40:14 Flights << Temp01Flights

Lines fetched: 229,274

18 Aug 2014 18:40:44

App saved

Finished successfully

0 forced error(s)

0 synthetic key(s)

Highlighted
MVP & Luminary
MVP & Luminary

Re: Qlik Sense takes almost three times longer to reload than QlikView

" I don't generate a log file since it is a long script It took a long time to save the file"

I'm curious about that observation. Are you saying that using a document log added a measurable increase to load time? How much?

-Rob

Highlighted
Partner
Partner

Re: Qlik Sense takes almost three times longer to reload than QlikView

Hi Rob,

I don't even know how much more time it takes, when I enabled to generate the log file and then reloaded the document I stopped the reload when it reached double the time it takes without generating the log file (30 min without log, more than 1 hr with log) that is for QlikView 11.2 SR5.

Highlighted
Luminary
Luminary

Re: Qlik Sense takes almost three times longer to reload than QlikView

Bill,

My test shows the same results. Loaded 20 mio rows on the same computer with Sense and QlikView.

QlikView: 6,00000014529 seconds

Sense: 14,00000012945 seconds

In both loads, I have a variance of about a 1 plus/minus.

Since Sense autosaves, I've done my timing using this script, which is accurate beyond nanoseconds:

Let _StartTime = Now();

Let _TotalTime = Round((Now()-_StartTime)*60*60*24,0.00000000001);
Trace $(_CalendarName) created in: $(_TotalTime) seconds;

Loading the same data, just generated by Sense took just 6,99999975041 seconds in Sense.

QlikView however cannot read the QVD that Sense has generated!

This is really bad for hybrid (QlikView+Sense) deployments. It will basically force us to load every table twice.

Highlighted
Luminary
Luminary

Re: Qlik Sense takes almost three times longer to reload than QlikView

Gabriela, in QlikView 11.2 SR7 I could not find a measurable time difference when logfiles was enabled or disabled.

Highlighted
Employee
Employee

Re: Qlik Sense takes almost three times longer to reload than QlikView

Torben Seebach wrote:

QlikView however cannot read the QVD that Sense has generated!

This is really bad for hybrid (QlikView+Sense) deployments. It will basically force us to load every table twice.

Yeah for now the recommendation, once the Sense server goes GA that is, for customers running both products side by side is to generate their QVD-layer through QlikView. However down the line it would be preferable to have both products share the same internal format that is columnar instead of row based.

Highlighted
Luminary
Luminary

Re: Qlik Sense takes almost three times longer to reload than QlikView

Alexander and Michael, maybe should tell PM to stop calling the Sense QVDs for QVDs, because if they cannot be read by a program that reads QVDs (QlikView)  then they are not QVDs. That is like calling XLSX as XLS.

It would make sense that Sense could write a new optimized format called QVDX or something, but also write out standard QVDs so they could be loaded by QlikView. Hybrid deployments are going to be a huge mess if this is not sorted - maybe we should start using txt files again.

Highlighted
Employee
Employee

Re: Qlik Sense takes almost three times longer to reload than QlikView

Hello Torben I have informed PM.

Mike

Regards,
Mike Tarallo
Qlik
Highlighted
Partner
Partner

Re: Qlik Sense takes almost three times longer to reload than QlikView

I made the same test, I converted the QVD's using the 'store table into..' in Sense, then I ran the script and it was almost 10 times slower than QlikView

Sense (using Sense QVD's): 5hr 53min

Sense (using QlikView QVD's): 1hr 32min

QlikView: 38min

Are we doing something wrong?

Highlighted

Re: Re: Qlik Sense takes almost three times longer to reload than QlikView

Gabriella

Do you use WHERE EXISTS on any loads ?

See thread Where Exists on loading from QVD ?that I opened this morning.

Best Regards,     Bill