topic Re: Incremental loading in Talend in Talend Studio

Incremental loading in Talend

Soumya_M — Fri, 15 Nov 2024 22:03:51 GMT

Hello, I've created a talend job for incremental loading similar to the one given in this link - which only loads the latest records after comparing the timestamps.

https://community.talend.com/s/feed/0D53p00007vCkMrCAK?language=en_US

But how do i achieve updation of older records ? as my job does incremental loading. Two things i need to incorporate in the job

To load any older date files, older than the last successful run timestamp
How do i update any changed records ?

Re: Incremental loading in Talend

Anonymous — Wed, 01 Mar 2023 07:50:32 GMT

I have a question about your record, does there exist timestamp field in your record, and this field is updated whenever the records is updated or inserted?

Regards

Shong

Re: Incremental loading in Talend

Soumya_M — Wed, 01 Mar 2023 08:14:13 GMT

the timestamp remains the same, only the records get updated. It's like I've already loaded the data for a particular date file, later I get a file with modified records for the existing date file

Re: Incremental loading in Talend

Anonymous — Wed, 01 Mar 2023 08:40:24 GMT

what is the target application? Database? if there is no a flag indicating that the record has been updated, I think you need to load all data from the file you get.

Re: Incremental loading in Talend

Soumya_M — Wed, 01 Mar 2023 09:59:37 GMT

target is postgresql database, how do i need to get the flag indicating which record has been changed ?

Re: Incremental loading in Talend

Soumya_M — Thu, 02 Mar 2023 06:16:05 GMT

Instead of updating, first I want to delete the matching records and insert the latest records

Re: Incremental loading in Talend

Anonymous — Thu, 02 Mar 2023 07:13:15 GMT

@Soumya M , you need to investigate if it is possible to know which records have been deleted or updated from your source file, if not, you have to do full comparison between source records and all of target data to get the deleted records and updated records, then, delete the corresponding record and update the corresponding record in the target database. I don't know your data source and data structure, so there is no way to give a specific job design.

Re: Incremental loading in Talend

Soumya_M — Thu, 02 Mar 2023 07:24:13 GMT

Okay, because i don't know which records are updated I've to do full comparison between source and target records. How do i delete existing records and insert updated records to the single target table? source and target tables belong to postgres database.