topic Re: Incremental data load in Talend Studio

Incremental data load

Anonymous — Sat, 16 Nov 2024 11:06:40 GMT

Hi all,
I am new using Talend DI and I have to made an incremental load from one Oracle Database to another Oracle Database. I have to compare the time of the last execution with a column call Last Modified Date to update the rows modified or insert the new ones.
Can anyone help me with this? I just start with talend and i feel a bit lost with this.
Thanks very much!!

Re: Incremental data load

Anonymous — Wed, 05 Aug 2015 08:33:09 GMT

There is no build in method in Talend.
The way you have to work goes like this:
Create a context variable to keep an Date object (call it start_date) and set it to a very early date (which does not exists in your data)
Tweak the query of the database input to restrict the data above this date.
Keep track of the highest last modified date in your data within the flow
Memories the youngest last modified date and write it e.g. in a control table or file
Next job run read the last last modified date and set the value in the mentioned context variable (mentioned above).
You can use a component suite which supports this method very well:
https://exchange.talend.com/#marketplaceproductoverview:gallery=marketplace%252F1&pi=marketplace%252F1%252Fproducts%252F556%252Fitems%252F1732

Re: Incremental data load

Anonymous — Wed, 05 Aug 2015 08:52:22 GMT

Thanks a lot for your answer, I understand the most part and I will try to follow your steps.

Re: Incremental data load

Anand5 — Wed, 17 Aug 2016 08:17:57 GMT

Hi,
Can you please provide the screen shots to implement incremental load.

Regards
Anand V

Re: Incremental data load

Anonymous — Wed, 17 Aug 2016 09:14:12 GMT

Take a look at the component documentation I mentioned in my post.

Re: Incremental data load

seelam_sreeram — Wed, 26 Oct 2016 04:36:13 GMT

Hi,
Looking for incremental load approach with some screen shots. Can any one help me please ?
Thanks,
Rama.