Does a QVD file really save time?

[brandonblaine]

This is from https://support.qlik.com/articles/000005537

"QlikView Data (QVD) file is a optimized file format for storing QlikView data tables. The file has a binary data format that can be loaded directly to computer memory without any processing required, which makes it the fastest way to load data into a QlikView application."

If it takes 20 minutes to load the QVD from a sql statement, actual load time - not sql execution time, and the QVD then loads into the presentation app immediately, it still takes 20 minutes to load data. Besides the data manager working better with the QVD format than a table loaded straight from sql, what is the real benefit of a QVD when the net load time is the same?

Thanks, a bit grumpy tonight. 🙂

[Lisa_P · Accepted solution]

You don't load QVD files from a SQL statement. You may load original data using a SQL statement, but you then export it to QVD format for other apps to use. If it takes 20 minutes to load from SQL it may only take a matter of seconds from a QVD.
Also you can use the incremental load concept to only get changes from SQL and use QVD as main data source.

[brandonblaine]

Hi Lisa,

Thanks for replying. Let me go through an example that I am currently working with.

I have an app that shows the current status of our Accounts Receivable, AR. It has a couple million rows and about 50 columns, not extremely large.

Since our AR changes everyday in regards to payments, adjustments, etc. an incremental load will not work efficiently, the logic to determine which rows to keep and which ones to load would take me longer than I have time for. 

The data that I need to load is on a table that already had all of the SQL logic applied to it, it is just a straight data load, no where clause or joins are involved. Each night this table is loaded into Qlik so the billing and collection team can see trends, who the biggest offending payers are, AR balances by aging, etc.

It takes ~20 minutes to load this data into a qvd using a statement like: 

     TempAR:
     LOAD *;
     SQL SELECT *
     FROM ais.DailyAR;
     STORE TempAR INTO [lib://FileSystem/ARDaily.qvd] (qvd);
     DROP TABLE TempAR; 

I've found that if I don't use the QVD format, the data editor will never load, it will just keep spinning and spinning. I use the data editor to pick up the auto calendar function and other this. I know this can all be done on the load script but I am not a good Qlik scriptor yet.

I then have the presentation app that loads from the QVD which of course is almost instant.

Since the total overhead time for me is the time it takes to prepare the QVD (~20 minutes) + the time it takes to load it (0 mins) I am wondering if I am doing something that I shouldn't.

I completely understand that in most environments, several apps can use the same QVD so there is a large saving there. Hopefully I can start to take advantage of incremental loads, etc.