How to do incremental loading in talend

[Anonymous]

Hi,
I am getting source data on daily basis, how can we do incremental loading in talend. Please provide me the sample example.
Thanks in Advance.

[karandama2006]

Hi @nthampi ,

I want to know your opinion on Delta load for tracking insert and updates using the last_modified_date.

Is it beneficial to use Insert or Update option in the output component (after filtering the source with records greater than last_updated_timestamp) , or to add a left outer join using tmap , Update the matched record and insert the rejects .

will there be a difference in performance ? , what is the recommended approach ? 

Thanks ,

Karan

[Anonymous]

Hi Karan,

     Lookup in tMap might be a costly option if your lokup tables have million of rows. In that case, its better to select a key value in your output DB component and use Insert or Update option. It will then use the primary key of the underlying table which will be faster.

     Again it is not a hard and fast rule for all the scenarios and DBs. Some DBs/DWs will give maximum throughput for insert only transactions where as some others will give decent performance for both insert and update in same flow.

     So always do the performance tests before taking the call to select the right option.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

[Anonymous]

Hi,

Could you please share the steps.

Thanks,

[Soumya_M]

Hi Nikhil,

I have a question - what if I've the last run date as 1-1-2023 and i want to load a file that's dated 20-12-2022 ?