Talend Data Validation Business Rules

[Anonymous]

Hi,

I want to define data validation business rules in talend.

Right now I define all comparsion rules in the tmap, but now i have to put all rules in more then 100 columns which is not preferable in tmap.

Because it will difficult if any one of the rule will update in future so maintain those things will difficult in tmap.

Is any another solution or component there in talend like MDS where we can define business or conversion rules in a separate enties.

[Jesperrekuh]

Could you please elaborate on the way your business rules work or you want them to work?
* column names en datatypes
* Regular expressions
* Constraints, -25 > x <= 81
* Exist in list/array [A:B:Z ]
* RFC specifictions

Is it like you add a field EMAIL and check whether it's compliant to RFCx ?
I once developed a class with custom routines and loaded all regex definitions from a json or xml file to make it flexible and manageable.

function validateField(fieldDefinitionType, value, cleanseValue)
Within tMap you could call this function like Routines.validateField('EMAIL', row1.emaddress, true);

Good luck.

[Anonymous]

Thank you for reply.

Actually I have to compare my source data to target data with some business rules.

I have to validate that all source data are properly converted and transformed according to the business rules and uploaded successfully in target location.

It's just like data reconciliation with some business rules.

[Jesperrekuh]

Properly converted -> datatype and formatting

Business rules ->  formula, (non) equal, bigger, smaller, between and is matched with a record containing reference values.

Show us an example please?

- How many lookup records? How do you match is it 1:1 or 1:n ? 

- Is it multicolumn validation like column combinations  colA & colB -> match busrulCol X  -> res(colA*colB) <= 113

Elaborate please?!?

[Anonymous]

Business Rules means compare(table a and table b) column_a.1 to column_b.1 and when column_a.1 ="A" then column_b.3="true/yes".

This kind of business rules i have to mention for 100+ columns.

Some field are directly validate means string to string like source values are properly loaded into the target location or not?

---

@Dijke wrote:

Properly converted -> datatype and formatting

Business rules ->  formula, (non) equal, bigger, smaller, between and is matched with a record containing reference values.

Show us an example please?

- How many lookup records? How do you match is it 1:1 or 1:n ? 

- Is it multicolumn validation like column combinations  colA & colB -> match busrulCol X  -> res(colA*colB) <= 113

Elaborate please?!?

---

[Anonymous]

Is any way in Talend like all business rules i will define in some user interface URL link or dashboard.

MDS have this functionality that we can create some business specific business rules and show into the user interface URL.

[Jesperrekuh]

Ok, get it, I think not, but maybe Talends MDM / Datagovernance products will suit your needs.
Some stuff you mention is something you could check with tFilterRow but is on a single column. It has an advanced option you could write custom java code... but not what you;re looking for.

I suggest use something outside of talend, more focussed on Business Rule and DataQuality...