topic Re: Duplicate data in Talend Studio

Duplicate data

TharunJoshi — Sat, 29 Jun 2019 20:00:35 GMT

Hi
I have a question
I have one Excel I loaded it last week
Now again I got the same Excel file they have just copied the old file and sent me that's it now if you load it in to database it will create duplicate
I don't want duplicate data to be filled I don't have any update in my project everything I do is insert but not duplicate
Eg: If I have a file with 100 records where 50 are previous records and 50 are new so this previous data is duplicate I don't want to push this into database what is the way I can follow
Please explain clearly
If you made a job of this kind share at 12e41a0222@gmail.com

Re: Duplicate data

TRF — Sun, 30 Jun 2019 11:44:25 GMT

You need to read the record from the database, then using a tMap with the proper inner join, exclude records from the Excel file (main row) existing into the database (lookup row).

If the previous is still there, you can also replace the database by this previous file.

This is a very common pattern.

Re: Duplicate data

TharunJoshi — Mon, 01 Jul 2019 12:52:20 GMT

Screenshots would have helped me as a beginner