topic Incremental load in QlikView

Incremental load

Mon, 05 Sep 2011 07:33:35 GMT

I read that i can use the exist () phrase for a qvd load and still the qlikview will load the qvd as "optimized Load".

So if i'll replace the "where DateField > $(VDate)" part in an incremental load statment with "where exist (AlreadyLoadDateField)" i will get better performance ?

did someone use it or try it out ?

thanks for your help.

Incremental load

Tue, 06 Sep 2011 07:56:21 GMT

Can someone help me with this ?

Re: Incremental load

gandalfgray — Tue, 06 Sep 2011 08:05:33 GMT

Yes I think you would.

But You should rather use Not.

...

Where Not Exists(DateField);

/gg

Incremental load

Tue, 06 Sep 2011 09:26:29 GMT

i dont think you will get better performance! because if you load from a source (except a qvd-file) with the "not exists"-statement, the script has to load all the data from the source and then filter the already existing datasets out of it. if you load with an "where DateField > $(VDate)" statement, then the source maybe can filter the datasets for itself and only submit the correct Data to your script. Therfore not so much data has to be streamed.

MFG

Incremental load

gandalfgray — Tue, 06 Sep 2011 09:35:39 GMT

... well I thought "a qvd load" meant that Matan was loading from a qvd...

If it's a load from a relational database, the performance will depend on the indexes in the database etc.

Re: Incremental load

Tue, 06 Sep 2011 13:21:12 GMT

Thanks for your comments.

I probably didnt make my self clear.

I know that the part in the script that load the new data from relational database will not be affected by using exist () insted of "where DateField > $(VDate)" .

in the second part - the loading of the saved data (stored in qvd) i want to load the old data without the last saved day.

for that part i considered of using the exist () phrase.

so...will it work better ?

Re: Incremental load

gandalfgray — Tue, 06 Sep 2011 13:31:46 GMT

If you have first loaded data (by sql or something) into a field DateField

you could then load from a qvd using

Where Not Exists(DateField)

to avoid loading records with DateField-values that is already loaded into your qvw.

I believe it should be faster than

Where DateField>$(VDate)

hth/gg

Re: Incremental load

Tue, 06 Sep 2011 13:51:45 GMT

Thank a lot