topic Re: Validating a data file against a schema file in Hadoop in Talend Studio

Validating a data file against a schema file in Hadoop

_AnonymousUser — Sat, 16 Nov 2024 11:14:49 GMT

Hi,
I have a requirement to validate a data file in Hadoop against a given schema file. Please note that the schema file is always dynamic. The data and schema file is always a combination.
For example the schema file would look something like this:
MKey|int|30|||
PKey|varchar|25|||
CNumber|varchar|25|||
CStatus|varchar|1|||
DOS|date|10||MM/DD/YYYY|
Above schema file has 5 fields.
The data file looks something like below:
123111111|456||D|12/04/2014
Can someone help me how I can design this in Talend? Any help is greatly appreciated.
Thanks
Ramakrishna

Re: Validating a data file against a schema file in Hadoop

Anonymous — Mon, 04 May 2015 04:26:12 GMT

Hi,
Have you already checked component TalendHelpCenter:tSchemaComplianceCheck which validates all input rows against a reference schema or check types, nullability, length of rows against reference values.
Best regards
Sabrina

Re: Validating a data file against a schema file in Hadoop

_AnonymousUser — Tue, 05 May 2015 02:36:17 GMT

Sabrina,
tSchemaComplianceCheck needs the schema to be static, but my schema file is dynamic. I have the requirement to validate the schema of a data file against the provided schema file in the format described in my original post.
Thanks
Ramakrishna