Incremental loading in Talend

[Soumya_M]

Hello, I've created a talend job for incremental loading similar to the one given in this link - which only loads the latest records after comparing the timestamps.

https://community.talend.com/s/feed/0D53p00007vCkMrCAK?language=en_US

But how do i achieve updation of older records ? as my job does incremental loading. Two things i need to incorporate in the job

1. To load any older date files, older than the last successful run timestamp
2. How do i update any changed records ?

[Anonymous]

Hi

I have a question about your record, does there exist timestamp field in your record, and this field is updated whenever the records is updated or inserted?

Regards

Shong

[Soumya_M]

the timestamp remains the same, only the records get updated. It's like I've already loaded the data for a particular date file, later I get a file with modified records for the existing date file

[Anonymous]

what is the target application? Database? if there is no a flag indicating that the record has been updated, I think you need to load all data from the file you get.

[Soumya_M]

target is postgresql database, how do i need to get the flag indicating which record has been changed ?

[Soumya_M]

Instead of updating, first I want to delete the matching records and insert the latest records

[Anonymous]

@Soumya M​ , you need to investigate if it is possible to know which records have been deleted or updated from your source file, if not, you have to do full comparison between source records and all of target data to get the deleted records and updated records, then, delete the corresponding record and update the corresponding record in the target database. I don't know your data source and data structure, so there is no way to give a specific job design.

[Soumya_M]

Okay, because i don't know which records are updated I've to do full comparison between source and target records. How do i delete existing records and insert updated records to the single target table? source and target tables belong to postgres database.