topic Re: Talend job is failing to load data in Talend Studio

Talend job is failing to load data

Anonymous — Sat, 16 Nov 2024 04:14:53 GMT

Hi All

I have created a job to load mysql table data into postgresql table.I have a unique column constraint for email in the table. If unique constraint is failed in one insert line the entire batch of data following that line will failed to load.Please provide me a solution. I'm loading 1.4 million of data. I'm using talend open studio for data integration

Re: Talend job is failing to load data

fdenis — Tue, 29 Oct 2019 08:31:31 GMT

add filter on tMap to valid constraint before inserting.
there are other way more complex inserting by 1000 then if error insert by 100 and so on to skip error.

Re: Talend job is failing to load data

Anonymous — Wed, 30 Oct 2019 09:06:21 GMT

Although formally, in my opinion, this is not a bug. AUTO_INCREMENT = xxx refers to table metadata, not data. I think he needed to get not a bug report, but a feature request to add the -no-autoincrement-value key, and the problem would have been solved long ago.

Re: Talend job is failing to load data

Anonymous — Mon, 11 Nov 2019 07:33:46 GMT

Hi fdenis,

Thank you for the answer.

Can you please elaborate more about the solution as what component to use? I'm new to talend.

Re: Talend job is failing to load data

fdenis — Wed, 13 Nov 2019 13:54:12 GMT

no, I do not have enought time for that…
star by splitting your data by 10000 then try to insert with commit at the end.
if fail try to insert by 1000 and so on.

Re: Talend job is failing to load data

Anonymous — Thu, 14 Nov 2019 05:29:48 GMT

Hi,

Why don't you load the entire data first to a stage table in PostgreSQL? Then you can do a inner join with target table to identify the duplicate records using tPostgreSQLRow or Input component. Once you identify the record which are duplicate, you can either delete them in stage or target based on your use case. Then you should be able to pump data easily to target table.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved