topic Re: How to remove duplicates from final output in Talend Studio

How to remove duplicates from final output

AV2022 — Fri, 15 Nov 2024 21:49:44 GMT

Hello Team,

I have connected two Input components to extract data from 2 different tables from my source DB which is the Access DB. Exporting the data from both the tables are working fine and i am able to import into my Oracle tables successfully. Attached screenshot shows how i have connected the components.

My issue is i have removed the duplicate entries using tUniqrow for both the input tables before it is connected to tMap, when the data is moved from tMap to respective output oracle tables i am still seeing duplicate entries, basically the data from 2 different input tables are pushed to same output oracle tables

For Example: Input Table Values InputTable2 Values

1 1

2 3

Output Oracle table Values

Required result : My output table should not have two values of 1

I am not sure if its the problem with the way i have connected the tUniqrow component or not. Kindly help me with this issue.

Re: How to remove duplicates from final output

Anonymous — Wed, 10 May 2023 06:31:07 GMT

Cache the two outputs into memory using tHashOutput component. In Next subjob, read the data back from memory using tHashInput, and link it to tUniqrow to remove the duplicated records, write the result to Oracle table.

Regards

Shong

Re: How to remove duplicates from final output

AV2022 — Wed, 10 May 2023 06:51:40 GMT

Hello Shong,

Thanks for the response, but unfortunately i am not able to find the tHash component in the palette. Please see the screenshot, kindly suggest us what needs to be done.

Thanks

Anand V

Re: How to remove duplicates from final output

AV2022 — Wed, 10 May 2023 06:56:07 GMT

Hello Shong,

Please ignore the above comment. tHash components where hidden in the panel. I have added it to the palette. Let me try the solution which you have suggested and update you back

Thanks

ANand V

Re: How to remove duplicates from final output

AV2022 — Wed, 10 May 2023 12:28:42 GMT

Hello Shong,

I am bit confused here, can u share a screenshot which shows how i need to connect the tHash components, in my case,do i need to connect the tHashOutput component after tMap? please clarify

Thanks

Anand V

Re: How to remove duplicates from final output

jlolling — Wed, 10 May 2023 19:30:34 GMT

Could you show us the component config of the tUniqueRow. Actually your design should work well.

Re: How to remove duplicates from final output

AV2022 — Thu, 11 May 2023 04:35:03 GMT

Hello,

Please find the component config details as requested. When i extract the data from single input table by using tUniqueRow iam getting the desired output, problem occurs when data is extracted from multiple tables and the input from both tables are pushed to same oracle output tables. The values of N_PROCESSO should be duplicate in the output tables

Note: Both the Input tables have the same column N_PROCESSO