Do not input private or sensitive data. View Qlik Privacy & Cookie Policy.
Skip to main content

Announcements
Qlik Open Lakehouse is Now Generally Available! Discover the key highlights and partner resources here.
cancel
Showing results for 
Search instead for 
Did you mean: 
Anonymous
Not applicable

Validate thousand columns of a data row

Hi everyone,
I'm new to Talend and I need some supports.
The requirement here:
Get data from Redshift, export valid/error data to CSV file.
Check all columns of each data rows if it match with schema of an existing table.
Export error data to a CSV file, infos need to export are the value of primary key, column name and error messages.
Export the valid data row to other one.
Conditions are:
- Check data type
- Check data lenght, if it's a float check the lenght after the comma
- Check date pattern
- Check if the primary key is duplicated with any one in the getted data list.

* If a column has more error, only export to one row in CSV, error mesages are separated by comma.
* If one data row has more error columns, export to multiple rows.

** Here a big note: a data row has more than 1000 column.

Can you give some solution for this requirement ?
Sorry for my English 0683p000009MACJ.png 0683p000009MACJ.png
Thank you so much!

Labels (3)
1 Solution

Accepted Solutions
Anonymous
Not applicable
Author

Use tSchemaComplianceCheck component.  And some additional checks.

View solution in original post

2 Replies
Anonymous
Not applicable
Author

Use tSchemaComplianceCheck component.  And some additional checks.

Anonymous
Not applicable
Author

Thank you. But my account don't want to use a component to do it, they need us do it by routine, any more sugest ?