duplicate rejection does not work (as usual)

[Anonymous]

Hello,

the content of a csv need to be imported in a table, avoiding duplicates on a field

- I try to reject duplicates using tMap, but it does not filter anything

- now I use not tUniqRow, but I really dont understand why the tMap rejections was not working ???

(even if I set the "reload at each row" and even with a tDBCommit)

please help me, this should definetely work

thanks

my setup :

rejection setup :

here is the run and imported file :

[TRF · Accepted solution]

tMap rejections was not working just because you don't ask it to remove duplicates.

What you do is just an inner join berween your main flow (row2) and a lookup (affiliate_lookup) saying "get records from main flow as soon as there no record in the lokkup with same value for the AffiliateCommercialName).

If you want to remove duplicates using tMap you can do this:

The difference is the tMap local variable named as the current AffiliateCommercialName value and populated with the number of occurence for this value. Then on the output flow you have a filter expression to remove every records but the 1rst one.

[TRF]

Did this help?
If so thanks to mark your case as solved.

[Anonymous]

strange, all the examples I found (which were working so far)

did not need an inner variable nor an expression filter

https://community.talend.com/t5/Design-and-Development/resolved-Remove-all-duplicate-rows-from-flow-...

[TRF]

This example use a tFilter after the tMap which a similar solution of mine.
Once again, your tMap just tell "get records from main flow as soon as there is no record in the lookup with same value for the AffiliateCommercialName)".

There is nothing about duplicates. You have to decide and manage this case by yourself. The solution I proposed is lighter than the one you give as an example (less components used).

As it works and answer your question, thanks to mark your case as solved (Kudos also appreciated).

[Anonymous]

I'll give it a try

[Anonymous]

it works !!

which means I have to modify all my previous talend jobs, lol

I really dont understand that inner join reject, I means it's so confusing, since it looks like it's rejecting lookup matches

well thanks dude