Remove duplicate row based on all column match before inserting to DB

[kaps1602]

I need to remove all duplicate records from a file before it get inserted to DB. In my case a record will be duplicate only in case all of the columns has same value. Example : The first two record will be considered as duplicate but third one is not. Here it is 5 columns , but could be more 10, 20.

ip             host_name       os_name           os_version

1.1.1.1        abc.com        Windows               8

1.1.1.1        abc.com        Windows               8

1.1.1.1        abc.com         Linux                   5.6

I need to insert records to Redshift DB after De-Dupe is done.

[TRF · Accepted solution]

Use tUniqRow and select all fields as keys