Incremental loading

[shree909]

Hi ,

i have a question regarding incremental loading.

i have like 6 yrs of data as a qvd and i want to add the data to the latest year.

so what are the ways that can be done. i dont want to reload the application as its huge.

are there any alternatives...

[magavi_framsteg]

Hi shree909.

What do you mean with "want to add the data to the latest year"?

Could you try and explain what you want to do with the six years of data.

[shree909]

i want to do add some new datato the existing data for the latest year.

and i dont want to reload the complete qvd, is there a way that i can add the data????

[#4]

No.

You can make incremental load working in the following way

- load for the first time and store in qvd

- when there is new data, only load the new stuff (make sure you have a way to determine "what is new" or use peek function to get the max ID or max(date) of your data in the qvd)

- load your existing data from qvd and concattenate your new stuff

- store this in your qvd, so now your qvd contains the old and new stuff

I think on this forum there are many examples that can help you.

If there is need to add to the existing year... perhaps you decide to have a qvd for each year. I think then the principle stays the same...

[#5]

Dear,

You will find the scenario of incremental load in the Qlikview  reference manual in details PageNo 509.

and also in qlikview Help menu.(search in Help --> index  By --> Append only)

Thanks,

Mukram.

[shree909]

Thanks for the reply..

ru sure that ADD function can be used as a incremental loading purpose..

are there any other ways?

[magavi_framsteg]

Hi shree909.

Statements prefixed with ADD are only executed when doing Partial Reloads.

So the ADD part of you script can load only the newest data.

For example, you can do (pseudo code):

ADD CONCATENATE (old_table) LOAD ................ SQL SELECT .............. where first date/id is newer than the last date/id in old_table;

Or something similar to accomplish your needs.

Kind regards

Magnus Åvitsland

BI Architect Consultant

Framsteg Business Intelligence Corp.

[stevedark]

Hi there, I have written a blog posting on incremental loads, giving examples on how you can use incremental load in three different scenariois:

http://bit.ly/VvY4MN

Personally, I tend not to use partial reloads for incrementals, rather I have a QVD layer so that the extraction of the data can be done separate to the presentation.  As long as the laod from QVD is optimised you can typically load all of the data from the QVD layer in a very short amount of time.

- Steve

http://www.quickintelligence.co.uk/

[shantanu73]

Hello Magnus Åvitsland

If I am not wrong in understanding the requirment of Shree909 is that He want to add new data in QVD file without uploading the old data from QVD file memory.

My requirment is also the same.

Here by adding the word "ADD" (which is Partial Load Functionalty) it will not add the data in QVD File from the source data. If I am wrong please correct.

Here, 1st we have to upload the old data from QVD and then give the Partical Load functionlty from the Source data. This is very time consuming job. Think about if there are about more the Millions/Billion/Trillion of records. Then the performance get effect.

Is there r some other way to handle with only one QVD File only?.

Regards

Shantanu