Best component or practice to apply data quality rules

[vidya821]

Hi All,

I need to apply certain data quality rules on columns in a row.

However i would like to apply these rules at once and not one after another.

also i need to collect all failures for all the columns.

Can someone pls guide for best practice to do so or best component to use

e.g Table 1 

Columns  ---->A | B | C

DQ Rules ---->Not null | Should be > 0 | Value in list [1,2,3]

now to check not null - we can use tschemaComplianceCheck

should be > 0 - we can use tFilter

Value in list[1,2,3] - we can use tMap

and moreover, these rules has to dynamically applied, for e.g these rules can be stored in oracle table so that we can change the rule whenever required and need not change anything in the job.

Pls help.

Thanks

[Anonymous]

Hi vidya,

I maybe don't get all your requirements but i think you can do everything in a tMap.

In the expression filter you can put something like :

row.A!=null&&row.B>0&&(row.C=1||row.C=2||row.C=3)

For sure there is a better way to test the column C but this solution he's working i think.

Warm regards