Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hello,
I have a quite complex script I'm running on QlikSense and I noticed that it takes longer to reload (almost three times more) than in QlikView, I'm reloading both on the same machine.
In QlikView I don't generate a log file since it is a long script It took a long time to save the file. I tried to find a way to disable the option to generate the log file in QlikSense but I couldn't find any, is this even possible?
Thank you in advance
Yup, that RESIDENT LOAD now takes 30 secs :
///////////////////////////////////////////////////////////////////////////
let vNow = now() ;
trace $(vNow) ;
///////////////////////////////////////////////////////////////////////////
NoConcatenate Flights :
///////////////////////////////////////////////////////////////////////////
load
*
resident Temp01Flights
where
MonthName ( [Scheduled Departure Date LT] ) = MonthName(Today())
or
MonthName ( [Scheduled Departure Date LT] ) = MonthName ( AddMonths ( Today() , -1 ) )
;
///////////////////////////////////////////////////////////////////////////
drop table Temp01Flights ;
///////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////
let vNow = now() ;
trace $(vNow) ;
///////////////////////////////////////////////////////////////////////////
exit script ;
///////////////////////////////////////////////////////////////////////////
Started loading data
Variables << Variables Lines fetched: 22 Calendar << Generic.Calendar Lines fetched: 62 18 Aug 2014 18:39:59 Temp.EZY.OTP << AUTOGENERATE(1) Lines fetched: 1 18 Aug 2014 18:39:59 Temp01Flights << RSMS.RSMSLIVE.Flights (QVD columnar optimized)
Lines fetched: 4,526,720
18 Aug 2014 18:40:11
18 Aug 2014 18:40:14 Flights << Temp01Flights
Lines fetched: 229,274
18 Aug 2014 18:40:44
App saved
Finished successfully
0 forced error(s)
0 synthetic key(s)
" I don't generate a log file since it is a long script It took a long time to save the file"
I'm curious about that observation. Are you saying that using a document log added a measurable increase to load time? How much?
-Rob
Hi Rob,
I don't even know how much more time it takes, when I enabled to generate the log file and then reloaded the document I stopped the reload when it reached double the time it takes without generating the log file (30 min without log, more than 1 hr with log) that is for QlikView 11.2 SR5.
Bill,
My test shows the same results. Loaded 20 mio rows on the same computer with Sense and QlikView.
QlikView: 6,00000014529 seconds
Sense: 14,00000012945 seconds
In both loads, I have a variance of about a 1 plus/minus.
Since Sense autosaves, I've done my timing using this script, which is accurate beyond nanoseconds:
Let _StartTime = Now();
Let _TotalTime = Round((Now()-_StartTime)*60*60*24,0.00000000001);
Trace $(_CalendarName) created in: $(_TotalTime) seconds;
Loading the same data, just generated by Sense took just 6,99999975041 seconds in Sense.
QlikView however cannot read the QVD that Sense has generated!
This is really bad for hybrid (QlikView+Sense) deployments. It will basically force us to load every table twice.
Gabriela, in QlikView 11.2 SR7 I could not find a measurable time difference when logfiles was enabled or disabled.
Torben Seebach wrote:
QlikView however cannot read the QVD that Sense has generated!
This is really bad for hybrid (QlikView+Sense) deployments. It will basically force us to load every table twice.
Yeah for now the recommendation, once the Sense server goes GA that is, for customers running both products side by side is to generate their QVD-layer through QlikView. However down the line it would be preferable to have both products share the same internal format that is columnar instead of row based.
Alexander and Michael, maybe should tell PM to stop calling the Sense QVDs for QVDs, because if they cannot be read by a program that reads QVDs (QlikView) then they are not QVDs. That is like calling XLSX as XLS.
It would make sense that Sense could write a new optimized format called QVDX or something, but also write out standard QVDs so they could be loaded by QlikView. Hybrid deployments are going to be a huge mess if this is not sorted - maybe we should start using txt files again.
Hello Torben I have informed PM.
Mike
I made the same test, I converted the QVD's using the 'store table into..' in Sense, then I ran the script and it was almost 10 times slower than QlikView
Sense (using Sense QVD's): 5hr 53min
Sense (using QlikView QVD's): 1hr 32min
QlikView: 38min
Are we doing something wrong?
Gabriella
Do you use WHERE EXISTS on any loads ?
See thread Where Exists on loading from QVD ?that I opened this morning.
Best Regards, Bill