Validate thousand columns of a data row

Anonymous — Sat, 16 Nov 2024 09:26:30 GMT

Hi everyone,
I'm new to Talend and I need some supports.
The requirement here:
Get data from Redshift, export valid/error data to CSV file.
Check all columns of each data rows if it match with schema of an existing table.
Export error data to a CSV file, infos need to export are the value of primary key, column name and error messages.
Export the valid data row to other one.
Conditions are:
- Check data type
- Check data lenght, if it's a float check the lenght after the comma
- Check date pattern
- Check if the primary key is duplicated with any one in the getted data list.

* If a column has more error, only export to one row in CSV, error mesages are separated by comma.
* If one data row has more error columns, export to multiple rows.

** Here a big note: a data row has more than 1000 column.

Can you give some solution for this requirement ?
Sorry for my English
Thank you so much!

Re: Validate thousand columns of a data row

Anonymous — Mon, 07 Aug 2017 02:26:51 GMT

Use tSchemaComplianceCheck component. And some additional checks.

Re: Validate thousand columns of a data row

Anonymous — Mon, 07 Aug 2017 07:24:06 GMT

Thank you. But my account don't want to use a component to do it, they need us do it by routine, any more sugest ?

topic Re: Validate thousand columns of a data row in Talend Studio

Validate thousand columns of a data row

Re: Validate thousand columns of a data row

Re: Validate thousand columns of a data row