Improving Qlikview Load Performance

[obriened83]

Hi,

I am working on improving our Qlikview Load performance and I was wondering if creating multibple QVDs (such as a QVD for each day) is more efficient than creating one giant single QVD? Thanks again.

Cheers Eamonn

[vijay_iitkgp]

You can use the Increamental load to improve performance.

[jagan]

Hi,

The Load Performance would be better if you load from a single file, instead of multiple files.  Because you need to concatenate the data for every QVD. 

We need to check whether all the columns are using in the report, if not remove those columns from script.

Check whether you are renaming any column names in LOAD statement, if you are doing that do it in file which stores the data in QVD file.  If we load the data from QVD file without any it loads very faster.

Regards,

Jagan.

[#4 · Accepted solution]

Hi Eamonn,

If you have each of your daily QVD's in exactly the same format, you are able to auto concatenate these.

When doing a single full load QVD vs this then the single QVD will perform slightly better yes (when loading the entire QVD). You ideally want to have this as an optimised load (i.e not alias or transforming being done during the load).

Having said that I have found having daily QVD's files the better option on numerous occasions as they give you much more flexibility.

For instance, when you have a smaller 'lite' version of an application, containing only a small number of days,

loading just a few days from your single QVD would be slower than looping through, say 5 separate daily QVD files.

The same would be true when thinking about rolling old data out of the end application but still retaining the history in your QVDs, your single QVD would be every growing but you would increasing load less from it as time goes by.

hope that helps

Joe

[#5]

Hello Eamonn,

Pleace check once Historical data is required or not?

Create 2 QVW files.

Good to use Sum() column function instead of Count().

Instead of IF condition use SET analysis.

[jonathandienst]

Hi

I would consider the number of records per day, per month, per year to decide on the QVD distribution.

I generally try to keep the number of rows in the QVD no more than a few million. The chances of corruption and failure is higher while writing the QVD and the risk increases with the size of the file.

There is a small performance hit when reading multiple files, but I have an application that reads 2 years worth of daily files (~40 million total rows) which takes a couple of minutes longer reading than a single file.

HTH

Jonathan

[#7]

Agreed, and even with larger volumes(500m+) I've only seen a couple of minute difference