topic Re: CSV to DataBase ETL issue when there is composite primary key in table in Talend Studio

CSV to DataBase ETL issue when there is composite primary key in table

Anonymous — Sat, 16 Nov 2024 06:33:27 GMT

I have created CSV to database table job where multiple columns in table makes the composite primary key.
There are multiple CSV which are to be inserted in same table. when my job runs when a duplicate entry of record comes in a same file, that record is not inserted and remaing all records are inserted. But when duplicate records comes in 2 different csv files then Job stops at the duplicate record I want Job to transfer all the records In all the csv files ignoring if record is duplicate(should not insert duplicate record and continue with remaining) as per primary key in table.

Re: CSV to DataBase ETL issue when there is composite primary key in table

akumar2301 — Tue, 19 Feb 2019 11:08:57 GMT

Did you try "Action on Data" as Insert or Update

In this case , if duplicate comes , it will update the record with latest information.

Let us know if you are expecting another behavior

Re: CSV to DataBase ETL issue when there is composite primary key in table

Anonymous — Tue, 19 Feb 2019 11:25:16 GMT

Yes I know that option but in case of duplicate I dont want to update any data. but I want that the duplicate record is ignored and job continues until all the CSV files get processed with all the records in them ignoring if any duplicate

Re: CSV to DataBase ETL issue when there is composite primary key in table

manodwhb — Tue, 19 Feb 2019 11:46:48 GMT

@roshan_wani ,i suggest you to do the do the lookup with target table and csv file and do inner join and take the inner rejects and load to the table.

Re: CSV to DataBase ETL issue when there is composite primary key in table

manodwhb — Tue, 19 Feb 2019 12:19:16 GMT

@roshan_wani ,if you are using the enterprise one you can log from tOracleOutput to error flow to a file.

Re: CSV to DataBase ETL issue when there is composite primary key in table

Anonymous — Wed, 20 Feb 2019 10:33:08 GMT

Hi manodwhb,

Thank you for your prompt answer, but it would be great if you show by some example how to get inner rejections

as I want to store csv records that are not present in lookup table(as per primary key).

Re: CSV to DataBase ETL issue when there is composite primary key in table

Anonymous — Wed, 20 Feb 2019 10:55:10 GMT

Hi,

If you do below method, you can take only new records (which are not present in DB).

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

Re: CSV to DataBase ETL issue when there is composite primary key in table

manodwhb — Wed, 20 Feb 2019 11:07:09 GMT

@roshan_wani , @nthampi has provide ,that is what i explained.

Re: CSV to DataBase ETL issue when there is composite primary key in table

akumar2301 — Wed, 20 Feb 2019 11:07:29 GMT

As in your case 1st occurrence of Key is valid for update or insert , I would prefer to merge all CSV file and using tAggregateRow/tuniqrow to get the First entry(for the key) , then use tDBOutput or Bulk components.