topic Re: Data Prep & Duplicates in Data Quality

Data Prep & Duplicates

Anonymous — Sat, 16 Nov 2024 07:43:53 GMT

Hi,

Using The Data Prep tool I am trying to find duplicates within a column, is there a way to find those that are an exact match as well as those that are a close match (similar to how excel duplicates and fuzzy match works)?

Re: Data Prep & Duplicates

aashish_21nov — Mon, 03 Sep 2018 05:04:02 GMT

tuniqrow will provide list of all duplicate records.

Re: Data Prep & Duplicates

Anonymous — Mon, 03 Sep 2018 05:08:29 GMT

Hi Aashish,

Is tuniqrow used in the Talend Data Prep Tool, I don't see can I select this and apply it to a column?

Regards

Re: Data Prep & Duplicates

aashish_21nov — Mon, 03 Sep 2018 05:33:23 GMT

yes you can apply it to a column but keep in mind your external link should be duplicate.

Re: Data Prep & Duplicates

Anonymous — Mon, 03 Sep 2018 07:52:39 GMT

Hi,

tUniqueRow is not available in Talend Data Preparation, it solely exists in Talend Studio.

So far in Data Prep, there is no equivalent to tUniqueRow, but addressing deduplication is part of the roadmap. Few things can already help spotting and solving duplicates:

The charts on the bottom right will show the most common values. Therefore, exact matches (if any) will show up at the top of the chart. See https://help.talend.com/reader/n~7tYndHPhSgSAN0bpenIA/rTPaBMcsHltniefJI2Caaw for more details.
The function "find and group similar text" performs text clustering with fuzzy matching. See https://help.talend.com/reader/087PIlN0I5BrDxt9016xRA/RVQQzV9CAYXQVh1m6DYWkA

Regards,

Gwendal

Re: Data Prep & Duplicates

Anonymous — Mon, 03 Sep 2018 21:10:07 GMT

Thanks Gwendal,
Is there an idea of when this would be in place - I've seen mention of this in some posts in 2016/2017?
Regards,
Darren

Re: Data Prep & Duplicates

Anonymous — Wed, 05 Sep 2018 11:06:40 GMT

Hi Darren,

That is something which will be addressed in the course of next year.

Regards,

Gwendal