topic Re: Incremental load in QlikView

Incremental load

Fri, 19 Apr 2013 11:54:07 GMT

Hello,

I want to know about incremental load.

For an example:

1. I have made an "RA.qvw" with all my scripting in (getting data from my DB) - And Stored all my tables into QVDs

2. I have loaded QVD into an RB.qvw - with nothing done (QVD optimized load)

3. I have binary loaded RB.qvw into RC.qvw (loads really really fast) 1 sec or so for 25mil rows

My question is, how do i do incremental reload? and where do i do it?

I would say full load from RA.qvw and Incremental load from RB.qvw??

Please help!

Thanks in advanced

Re: Incremental load

kaushiknsolanki — Fri, 19 Apr 2013 11:58:16 GMT

Hi,

Do the increment load at RA level. - this will not take much time.

Do a full load on RB - this might take time for full load.

And full load on RC - This will not take much time

Regards,

Kaushik Solanki

Re: Incremental load

jagannalla — Fri, 19 Apr 2013 12:01:31 GMT

Please find the attachment for Incremental load procedure.

It may help you.

Re: Incremental load

Fri, 19 Apr 2013 12:03:59 GMT

Can you explain why?

Why full load on RB?

So basically i wanna do incremental load from DB to RA - full load from RA to RB (why not incremental)?

Re: Incremental load

Fri, 19 Apr 2013 12:04:34 GMT

Thanks i have seen this - But im not sure i totally understand it. Where would i insert these scripts? In RA??

Re: Incremental load

kaushiknsolanki — Fri, 19 Apr 2013 12:07:33 GMT

Hi,

See the reason for doing incremental at RA level is, the load on your database will be less cause it will send (or QlikView will fetch ) only the incremental data.

Full load on RB is needed cause i believe that you are creating the datamodel at RB level and thus needs to have full data.

Full load on RC is you load full data from RB. and thus will be fast, so doesnt need incremetal here.

egards,

Kaushik Solanki

Re: Incremental load

Fri, 19 Apr 2013 12:22:32 GMT

Thank you! But how do i do incremantal load (insert,update) ? Ive seen the script being posted, but i cant quite figure out how it will see updated rows.

For an example:

If i have a DimProduct which contains

ProductKey, ProductID, ProductName

1, 1, ok

2, 1, ok

3, 2, hej

4, 3, heeeeee

If i do incremental load on insert i just want to check if ProductKey exists in my QVW

but what i have an update row eg like

ProductKey, ProductID, ProductName

1, 1, UPLYLSA

2, 1, ok

3, 2, hej

4, 3, heeeeee

Re: Incremental load

kaushiknsolanki — Fri, 19 Apr 2013 12:30:58 GMT

Hi,

Its always better that you have a field called LastUpdated, which will be a timestamp, which will tell us when the row is updated.

Incremental will become very hard without LastUpdated field.

Once you have that field the methods shown in PPT will work.

Regards,

Kaushik Solanki

Re: Incremental load

Fri, 19 Apr 2013 12:33:03 GMT

Ahh - so i need a lastupdatedfield. But thats just not very often you have that in Dimensions. But thanks for the answer!

Last questions.

Where do you wanna put the script shown in PPT?

And should it be for each table?

Re: Incremental load

kaushiknsolanki — Fri, 19 Apr 2013 12:38:06 GMT

Hi,

The lastupdated doesnt need to be a dimension, it will only help for incremental.

The script shown in the PPT will be put in RA qvw as a script.

Give a little efforts in searching the community about incremental load on qlikview, this will give you so many different scenarios.

Regards,

Kaushik Solanki

Re: Incremental load

Fri, 19 Apr 2013 12:53:29 GMT

I know - but you still need some date in your dimension to check when the row is changed, otherwise you wont get it in your incremental load. Right?

Re: Incremental load

kaushiknsolanki — Fri, 19 Apr 2013 13:12:32 GMT

Hi,

Yes you will need to store it as a field in your QVD.

Regards,

Kaushik Solanki