topic Re: Talend Data Validation Business Rules in Talend Studio

Talend Data Validation Business Rules

Anonymous — Tue, 22 May 2018 06:19:46 GMT

Hi,

I want to define data validation business rules in talend.

Right now I define all comparsion rules in the tmap, but now i have to put all rules in more then 100 columns which is not preferable in tmap.

Because it will difficult if any one of the rule will update in future so maintain those things will difficult in tmap.

Is any another solution or component there in talend like MDS where we can define business or conversion rules in a separate enties.

Re: Talend Data Validation Business Rules

Jesperrekuh — Tue, 22 May 2018 20:08:16 GMT

Could you please elaborate on the way your business rules work or you want them to work?
* column names en datatypes
* Regular expressions
* Constraints, -25 > x <= 81
* Exist in list/array [A:B:Z ]
* RFC specifictions

Is it like you add a field EMAIL and check whether it's compliant to RFCx ?
I once developed a class with custom routines and loaded all regex definitions from a json or xml file to make it flexible and manageable.

function validateField(fieldDefinitionType, value, cleanseValue)
Within tMap you could call this function like Routines.validateField('EMAIL', row1.emaddress, true);

Good luck.

Re: Talend Data Validation Business Rules

Anonymous — Wed, 23 May 2018 06:20:07 GMT

Thank you for reply.

Actually I have to compare my source data to target data with some business rules.

I have to validate that all source data are properly converted and transformed according to the business rules and uploaded successfully in target location.

It's just like data reconciliation with some business rules.

Re: Talend Data Validation Business Rules

Jesperrekuh — Wed, 23 May 2018 11:24:14 GMT

Properly converted -> datatype and formatting

Business rules -> formula, (non) equal, bigger, smaller, between and is matched with a record containing reference values.

Show us an example please?

- How many lookup records? How do you match is it 1:1 or 1:n ?

- Is it multicolumn validation like column combinations colA & colB -> match busrulCol X -> res(colA*colB) <= 113

Elaborate please?!?

Re: Talend Data Validation Business Rules

Anonymous — Tue, 29 May 2018 08:12:52 GMT

Business Rules means compare(table a and table b) column_a.1 to column_b.1 and when column_a.1 ="A" then column_b.3="true/yes".

This kind of business rules i have to mention for 100+ columns.

Some field are directly validate means string to string like source values are properly loaded into the target location or not?

@Dijke wrote:

Properly converted -> datatype and formatting

Business rules -> formula, (non) equal, bigger, smaller, between and is matched with a record containing reference values.

Show us an example please?

- How many lookup records? How do you match is it 1:1 or 1:n ?

- Is it multicolumn validation like column combinations colA & colB -> match busrulCol X -> res(colA*colB) <= 113

Elaborate please?!?

Re: Talend Data Validation Business Rules

Anonymous — Tue, 29 May 2018 08:15:10 GMT

Is any way in Talend like all business rules i will define in some user interface URL link or dashboard.

MDS have this functionality that we can create some business specific business rules and show into the user interface URL.

Re: Talend Data Validation Business Rules

Jesperrekuh — Tue, 29 May 2018 21:39:36 GMT

Ok, get it, I think not, but maybe Talends MDM / Datagovernance products will suit your needs.
Some stuff you mention is something you could check with tFilterRow but is on a single column. It has an advanced option you could write custom java code... but not what you;re looking for.

I suggest use something outside of talend, more focussed on Business Rule and DataQuality...