topic How to improve efficiency in incremental load ? in QlikView

How to improve efficiency in incremental load ?

Mon, 14 Sep 2009 21:25:07 GMT

Hi there,

Suppose I have one QVW with 2009 Aug data loaded in by 200908.qvd, now I will use incremental reload to append 2009 Sep data to the same QVW.

But it seems like the old qvd will be reloaded again, is there any way to keep the old data resident in QVW without being reloaded?

i.e. We only pull in Sept data without touching Aug data?

Or is there any way for better efficiency?

Thank you!!

How to improve efficiency in incremental load ?

Oleg_Troyansky — Tue, 15 Sep 2009 01:06:52 GMT

You can use "Partial Reload" with a prefix "ADD" on all those loads that you want performed. It's not very much recommended by the Developers.

If your QVD load is running in an optimized mode, it can't be THAT long, even if it's large. I suspect you are adding some calculations or conditions in your load, and that forces it into a non-optimized mode, which is slow.

Your best improvement is to try and avoid any additional calculations or conditions in your QVD load - let it run in optimized load, and it should be fast enough.

cheers!

How to improve efficiency in incremental load ?

Tue, 15 Sep 2009 02:41:51 GMT

Thanks Oleg, but

1. Why isn't "ADD" recommended? What problem could it cause?

2. I am running the final application in optimized mode, and my client is trying to find out the most efficient way.

So I just need one answer for the client :

is it true that existing data can NOT be resident in QlikView, everytime we load new data in, the old data has to reloded again?

Thank you!!

How to improve efficiency in incremental load ?

peter_turner — Tue, 15 Sep 2009 16:08:06 GMT

Hello ptung17,

I have a similar situation to you and would like to know the same answer.

My optimised reload can take around 1min which isn't too bad, but i want to reload the data every 10mins, so it can become a problem.
Also loading all my QVD data from the HDD will effect other server programs trying to access the disk at the same time -like SQL (i know you should really have a separate server for that and QVS).

It doesn't seem right to drop all data and reload again every time, so maybe 'add' is the best way for me also.
Peter.

How to improve efficiency in incremental load ?

Wed, 16 Sep 2009 00:08:49 GMT

Hi, attached is a presentation on incremental loading.