topic Re: data cleaning checks in Data Quality

data cleaning checks

Anonymous — Tue, 02 Oct 2018 03:21:32 GMT

Hi Experts,

How to apply given below data validation rules in talend. which components we can use with example. thanks

Max_length_if_not_empty
Null_blank_check
Invalid_pattern_check_if_not_empty
Is_empty
Date_Comparision
Datetime_format_check_if_not_empty
Valid_values_if_not_empty (in from of reference data/lookup)
Valid_values_if

Re: data cleaning checks

fdenis — Tue, 02 Oct 2018 12:30:41 GMT

The anser is you can do it with talend.

But: How why where?
as a consultant questions are:
how many times do you want to check that!
is it for migration or for run
are rules fix or they are going to change?
do you want to add rules?
what is the language of rules?
where are the data in db, file, cloud?
why do you want talend to do that!
and some others….

Re: data cleaning checks

Morpheus — Tue, 02 Oct 2018 14:35:48 GMT

Hi @abhityagi,

Are you trying to do something like a regular data health check to increase the quality of your daily/weekly feeds or you are trying to just check the overall data health as a pre-check before a migration?

Re: data cleaning checks

Anonymous — Fri, 05 Oct 2018 06:05:13 GMT

Thanks for your reply. I am looking for overall data health as a pre-check before a migration.

Thanks,

Abhishek

Re: data cleaning checks

fdenis — Fri, 05 Oct 2018 08:02:04 GMT

you are looking for Rule-based system like drools-