topic talend import data into a database without duplicating data in Talend Data Catalog

talend import data into a database without duplicating data

oguedda — Sat, 16 Nov 2024 00:24:43 GMT

Hello everybody,

i have a job which imports data from an API REST into a database SQL Server and this job is scheduled to be executed every month, the problem is that even if the data already exists on the database it injects it as same.

I would like to inject that data which does not exist in the database.

Any help please,

thank you in advance.

Re: talend import data into a database without duplicating data

Anonymous — Fri, 19 Mar 2021 06:49:10 GMT

Hello,

Would you mind posting your job design screenshots here? Are you trying to capturing changed data via tMap component in your work flow?

If you want to capture the changed data and only load these changed data into target table to achieve table sync, you can compare tables by using tMap.

The work flow should be:

Target DB-->tMap(inner join for input and set the "Catch lookup inner join" as true)-->output

Source DB-->

The output will be the changed data.

Best regards

Sabrina

Re: talend import data into a database without duplicating data

oguedda — Fri, 19 Mar 2021 08:38:00 GMT

Hello @Xiaodi Shi ,

Thank you for your answer,

I found the solution by adding the primary key in my table and on the database parameters you have to choose "insert if not exist" instead of "insert" as indicated in the screenshoot

Hoping that help someone

Re: talend import data into a database without duplicating data

Anonymous — Mon, 22 Mar 2021 04:41:33 GMT

Hello,

Great it is resolved by yourself. Thanks for letting us know your solution.

Best regards

Sabrina