topic Re: Performing Schema/Data Validation in Talend Studio

Performing Schema/Data Validation

Anonymous — Mon, 18 Jun 2012 17:00:27 GMT

Hi,
We are developing several integration jobs with TIS 4.2.4 and we are trying to design an approach to validate the data in the input files in the jobs. I used tSchemaComplianceCheck and it works to some extent.
1. If a numeric field in the schema is defined as "int" and the file has a string "abc" in it, talend is skipping such records with a message "For Input String : abc", it does not give the record number or any other clue to identify such records. Is there a way to handle such schema-mistmatched data?
2. If a date field in the schema is defined as "yyyy-MM-dd" and the date in the file is received as "MM-dd-yyyy", the tSchemaComplianceCheck component is catching it and shows in the rejects. Is there a way to use tSchemaComplianceCheck component to catch data issues as mentioned in #1?
Thanks for any info in catching such issues.
Thanks in advance,
Balaji.

Re: Performing Schema/Data Validation

Anonymous — Tue, 19 Jun 2012 03:46:01 GMT

Re: Performing Schema/Data Validation

Anonymous — Tue, 19 Jun 2012 11:59:58 GMT

Thanks Pedro. We use almost all kinds of input files since we are developing several integrations. Some of them are below.
tFileInputPositional
tFileInputDelimited
tFileInputMSPositional
tFileInputMSDelimited
Is this setting available all other file input components? Majority of our input files are tFileInputPositional components.
Thanks,
Balaji.

Re: Performing Schema/Data Validation

Anonymous — Wed, 20 Jun 2012 02:55:03 GMT

Hi Balaji
I'm sorry. Only tFileInputDelimited supports schema validation.
Regards,
Pedro

Re: Performing Schema/Data Validation

Anonymous — Wed, 20 Jun 2012 12:53:57 GMT

So, there is no option to find the exact record # of skipped records for other components? Can I post a feature request for this? Our organization is using TIS 4.2.4 as an enterprise level ETL tool and we get the priority support from talend when there is an issue. I can ask our company's Talend Admin group to write a feature request.
Let me know your thoughts.
Thanks,
Balaji.

Re: Performing Schema/Data Validation

Anonymous — Thu, 21 Jun 2012 02:47:47 GMT

Hi Balaji
Yes. For any new feature, please report it on BugTracker.
Add the "Check each row structure against schema" option for positional components.
Regards,
Pedro

Re: Performing Schema/Data Validation

karthikj18 — Thu, 21 Jun 2012 06:43:13 GMT

Hi Balaji,
You can try this to get the rejected records for the fixed length files (File Positional).
tFileInputFullrow (Get the full record)
|
V
tExtractFilePositional (Give the positions of the fields) ----> Main Row -----> output component.
|
V
Reject rows (It has "Check each row structure against schema" option)
|
V
output component

Re: Performing Schema/Data Validation

_AnonymousUser — Thu, 05 Jan 2017 06:57:44 GMT

Hi,
I have to fetch few csv files with different schema and store them into a database with a common schema.
how can i do it using a single job?