topic Re: duplicate rejection does not work (as usual) in Talend Studio

duplicate rejection does not work (as usual)

Anonymous — Sat, 16 Nov 2024 04:29:08 GMT

Hello,

the content of a csv need to be imported in a table, avoiding duplicates on a field

- I try to reject duplicates using tMap, but it does not filter anything

- now I use not tUniqRow, but I really dont understand why the tMap rejections was not working ???

(even if I set the "reload at each row" and even with a tDBCommit)

please help me, this should definetely work

thanks

my setup :

rejection setup :

here is the run and imported file :

Re: duplicate rejection does not work (as usual)

TRF — Thu, 03 Oct 2019 10:54:20 GMT

tMap rejections was not working just because you don't ask it to remove duplicates.

What you do is just an inner join berween your main flow (row2) and a lookup (affiliate_lookup) saying "get records from main flow as soon as there no record in the lokkup with same value for the AffiliateCommercialName).

If you want to remove duplicates using tMap you can do this:

The difference is the tMap local variable named as the current AffiliateCommercialName value and populated with the number of occurence for this value. Then on the output flow you have a filter expression to remove every records but the 1rst one.

Re: duplicate rejection does not work (as usual)

TRF — Fri, 04 Oct 2019 06:53:01 GMT

Did this help?
If so thanks to mark your case as solved.

Re: duplicate rejection does not work (as usual)

Anonymous — Tue, 08 Oct 2019 08:08:39 GMT

strange, all the examples I found (which were working so far)

did not need an inner variable nor an expression filter

https://community.talend.com/t5/Design-and-Development/resolved-Remove-all-duplicate-rows-from-flow-including-original/td-p/113243

Re: duplicate rejection does not work (as usual)

TRF — Tue, 08 Oct 2019 08:26:09 GMT

This example use a tFilter after the tMap which a similar solution of mine.
Once again, your tMap just tell "get records from main flow as soon as there is no record in the lookup with same value for the AffiliateCommercialName)".

There is nothing about duplicates. You have to decide and manage this case by yourself. The solution I proposed is lighter than the one you give as an example (less components used).

As it works and answer your question, thanks to mark your case as solved (Kudos also appreciated).

Re: duplicate rejection does not work (as usual)

Anonymous — Tue, 08 Oct 2019 11:51:08 GMT

I'll give it a try

Re: duplicate rejection does not work (as usual)

Anonymous — Tue, 08 Oct 2019 11:55:47 GMT

it works !!

which means I have to modify all my previous talend jobs, lol

I really dont understand that inner join reject, I means it's so confusing, since it looks like it's rejecting lookup matches

well thanks dude