Best component or practice to apply data quality rules

vidya821 — Tue, 02 Jul 2019 05:51:28 GMT

Hi All,

I need to apply certain data quality rules on columns in a row.

However i would like to apply these rules at once and not one after another.

also i need to collect all failures for all the columns.

Can someone pls guide for best practice to do so or best component to use

e.g Table 1

Columns ---->A | B | C

DQ Rules ---->Not null | Should be > 0 | Value in list [1,2,3]

now to check not null - we can use tschemaComplianceCheck

should be > 0 - we can use tFilter

Value in list[1,2,3] - we can use tMap

and moreover, these rules has to dynamically applied, for e.g these rules can be stored in oracle table so that we can change the rule whenever required and need not change anything in the job.

Pls help.

Thanks

Re: Best component or practice to apply data quality rules

Anonymous — Tue, 02 Jul 2019 08:42:00 GMT

Hi vidya,

I maybe don't get all your requirements but i think you can do everything in a tMap.

In the expression filter you can put something like :

row.A!=null&&row.B>0&&(row.C=1||row.C=2||row.C=3)

For sure there is a better way to test the column C but this solution he's working i think.

Warm regards

topic Best component or practice to apply data quality rules in Talend Studio

Best component or practice to apply data quality rules

Re: Best component or practice to apply data quality rules