CSV to DataBase ETL issue when there is composite primary key in table

[Anonymous]

I have created CSV to database table job where multiple columns in table makes the composite primary key.
There are multiple CSV which are to be inserted in same table. when my job runs when a duplicate entry of record comes in a same file, that record is not inserted and remaing all records are inserted. But when duplicate records comes in 2 different csv files then Job stops at the duplicate record I want Job to transfer all the records In all the csv files ignoring if record is duplicate(should not insert duplicate record and continue with remaining) as per primary key in table.

[akumar2301]

Did you try "Action on Data" as Insert or Update

In this case , if duplicate comes , it will update the record with latest information.

Let us know if you are expecting another behavior

[Anonymous]

Yes I know that option but in case of duplicate I dont want to update any data. but I want that the duplicate record is ignored and job continues until all the CSV files get processed with all the records in them ignoring if any duplicate

[manodwhb]

@roshan_wani ,i suggest you to do the do the lookup with target table and csv file and do inner join and take the inner rejects and load to the table.

[manodwhb]

@roshan_wani ,if you are using the enterprise one you can log  from tOracleOutput to error flow to a file.

[Anonymous]

Hi manodwhb,

Thank you for your prompt answer, but it would be great if you show by some example how to get inner rejections

as I want to store csv records that are not present in lookup table(as per primary key).

[Anonymous]

Hi,

   If you do below method, you can take only new records (which are not present in DB).

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

[manodwhb]

@roshan_wani , @nthampi has provide ,that is what i explained.

[akumar2301]

As in your case 1st occurrence of Key is valid for update or insert ,  I would prefer to merge all CSV file and using tAggregateRow/tuniqrow to get the First entry(for the key) , then use tDBOutput or Bulk components.