topic Re: Incremental load in QlikView

Incremental load

Mon, 03 Dec 2012 06:11:36 GMT

hi,

I have a scenario for incremental load. Please suggest me about this.

I have 5 months data. Everyday I needs to load application. Also there is chance of changing or updating records from the history i.e in 5 months of data currently.

So daily (before curent day's new data+last 5 months updated and history data also) needs to load these data.

Any idea or sample of incremental script. Please help.

Re: Incremental load

Mon, 03 Dec 2012 06:48:15 GMT

see in HELP

den serch for incrementall load u ll get nice explanation

Re: Incremental load

Mon, 03 Dec 2012 06:52:34 GMT

Hi,

You can maintain two saperate load files one for full load(historical data) and other for incremental(daily) load.

you run full load script file in long intervals like daily,weekly and save data in qvds.In this file you need to pull data from a specific date to current date( 5 months data). And maintain thet each table should have one key filed.

Now run incremental load file in small interval like per 5 minutes, per etc. and save data in qvd's. In this pull data only current date or last 2 days data based on your requirment.

Now let the view to load data from the qvd's created by incremental load file.

Then load data from the qvd's created by full load file by checking the condition keyfield should not exists.

Here I attached sample files, go through them.

hope it will help you.

Niranjan.

Re: Incremental load

Mon, 03 Dec 2012 07:10:40 GMT

Thanks for you quick response.

In the view.qvw which you attched , you used the Primary Key (KEY_CUST_PROD)two times. i.e where not exists(....,....) Like below. Is there any specific meaning or typo mistake. Kindly respond.

Data:

LOAD * FROM Data_Incr.qvd (qvd);

LOAD * FROM Data.qvd (qvd) Where not Exists(KEY_CUST_PROD,KEY_CUST_PROD);

Re: Incremental load

Mon, 03 Dec 2012 07:18:34 GMT

Hi,

That is not mistake. The purpose of using that filed is to avoid duplicate records.

first I loaded incremental records then full load records. so in full load, incremental load records may exists but we don't need them as we are already loaded.

To load records only which are not loaded by incremental load file we use that filed.

And it is neccessary to have such filed for every table.

Niranjan.

Re: Incremental load

Mon, 03 Dec 2012 07:29:23 GMT

Thanks for your reply.

In this whole process, We are loading new and updated records. We also need to have the history data also i.e history(before updation all tables)

Re: Incremental load

jonathandienst — Mon, 03 Dec 2012 08:06:22 GMT

Niranjan

You could also use the single parameter form of Exists in this case:

Exists(KEY_CUST_PROD)

This is equivalent to

Exists(KEY_CUST_PROD,KEY_CUST_PROD)

Regards

Jonathan