topic Re: Validate Schema of Entire File - Not Just Row by Row in Talend Studio

Validate Schema of Entire File - Not Just Row by Row

ml1662663516 — Fri, 15 Nov 2024 22:12:14 GMT

We are reading in a file using tFileInputDelimited and are validating schema column formats using tSchemaComplianceCheck.

We see we can use main and reject to check each row of the file.

However, we do not want to processes only the good rows.

We would like to set the job up to only process the entire file if there are no bad rows. Kind of an all-or-nothing type scenario.

Would anybody have any ideas on the components and flow we could use to perform such a scenario?

Thank you!

Re: Validate Schema of Entire File - Not Just Row by Row

Anonymous — Thu, 12 Jan 2023 16:43:50 GMT

There is no component (that I am aware of....new components arrive all the time) to check the whole file. But what you could do here is carry on checking line by line and store the output in memory. If you have any rejected rows, you will not proceed and work on the file. If you do not get any rejected rows, then you would process the data you have kept in memory.

To do this, you could use the tHSQLDB components (https://help.talend.com/r/en-US/7.3/hsqldb/hsqldb) or the Hash components (tHashInput - https://help.talend.com/r/en-US/8.0/technical/thashinput and tHashOutput - https://help.talend.com/r/en-US/8.0/technical/thashoutput).

Re: Validate Schema of Entire File - Not Just Row by Row

ml1662663516 — Fri, 13 Jan 2023 15:03:51 GMT

Thank you for your response.

We did look into using the components you mentioned.

However, our ultimate solution was using the tFileInputRegex component and then splitting off using the main connector and the reject connector and incrementing a global variable counter when a bad match is found.

We branch off from that subjob with an onSubjobOk and the check the global variable and proceed with an if connector.

I don't know if that is the most efficient workflow, but it does seem to be working correctly and reliably.

Thanks!

Re: Validate Schema of Entire File - Not Just Row by Row

Anonymous — Fri, 13 Jan 2023 16:47:07 GMT

So long as it suits your requirements 🙂