Remove duplicate row based on all column match before inserting to DB

kaps1602 — Sat, 16 Nov 2024 02:22:31 GMT

I need to remove all duplicate records from a file before it get inserted to DB. In my case a record will be duplicate only in case all of the columns has same value. Example : The first two record will be considered as duplicate but third one is not. Here it is 5 columns , but could be more 10, 20.

ip host_name os_name os_version

1.1.1.1 abc.com Windows 8

1.1.1.1 abc.com Linux 5.6

I need to insert records to Redshift DB after De-Dupe is done.

Re: Remove duplicate row based on all column match before inserting to DB

TRF — Wed, 20 May 2020 22:12:17 GMT

Use tUniqRow and select all fields as keys

topic Remove duplicate row based on all column match before inserting to DB in Talend Studio

Remove duplicate row based on all column match before inserting to DB

Re: Remove duplicate row based on all column match before inserting to DB