Incremental load

[#1]

hi,

I have a scenario for incremental load. Please suggest me about this.

I have 5 months data. Everyday I needs to load application. Also there is chance of changing or updating records from the history i.e in 5 months of data currently.

So daily (before curent day's new data+last 5 months updated and history data also) needs to load these data.

Any idea or sample of incremental script. Please help.

[#2]

see in HELP

den serch for incrementall load u ll get nice explanation

[#3]

Hi,

You can maintain two saperate load files one for full load(historical data) and other for incremental(daily) load.

you run full load script file in long intervals like daily,weekly and save data in qvds.In this file you need to pull data from a specific date to current date( 5 months data). And maintain thet each table should have one key filed.

Now  run incremental load file in small interval like per 5 minutes, per etc. and save data in qvd's. In this pull data only current date or last 2 days data based on your requirment.

Now let the view to load data from the qvd's created by incremental load file.

Then load data from  the qvd's created by full load file by checking the condition keyfield should not exists.

Here I attached sample files, go through them.

hope it will help you.

Niranjan.

[#4]

Thanks for you quick response. 

In the view.qvw which you attched , you  used the Primary Key (KEY_CUST_PROD)two times. i.e where not exists(....,....) Like below. Is there any specific meaning or typo mistake. Kindly respond.

Data:

LOAD * FROM Data_Incr.qvd (qvd);

LOAD * FROM Data.qvd (qvd) Where not Exists(KEY_CUST_PROD,KEY_CUST_PROD);

[#5 · Accepted solution]

Hi,

That is not mistake. The purpose of using that filed is to avoid duplicate records.

first I loaded incremental records then full load records. so in full load, incremental load records may exists but we don't need them as we are already loaded.

To load records only which are not loaded by incremental load file we use that filed.

And it is neccessary to have such filed for every table.

Niranjan.

[#6]

Thanks for your reply.

In this whole process, We are loading new and updated records. We also need to have the history data also i.e history(before updation all tables)

[jonathandienst]

Niranjan

You could also use the single parameter form of Exists in this case:

     Exists(KEY_CUST_PROD)

This is equivalent to

     Exists(KEY_CUST_PROD,KEY_CUST_PROD)

Regards

Jonathan