File Data Type: Dynamic - Execute awk command on tSystem to store the output in a file

[jaink]

Hi,

I need to check that the number of fields in a file is as expected. As i am using dynamic data type, i am not able to check the same using getCountColumn() function, as it always takes the max number of fields present.

Therefore, as a workaround thought of using awk and storing the rows which matches the expected number of fields to a file.
Also, it is needed to store the non matching rows to a file.

Example - 
source file - a.txt (data type defined as dynamic)
A,B,C
1,2,3
4,5,6,7
8,9

tSystem1 --> (main) tSystem2
Executed command - "" on tSystem
"/bin/bash -c awk -F, '{if(NF == 2) print $0 }' a.txt > b.txt"

But this command is not executed only i guess, because when executed, the command is passed as - '/bin/bash -c awk -F, '{if(NF == 2) print $0 }' a.txt

Please let me know if there is any other solution for the same, or what changes are needed in the existing code?

[manodwhb]

@kritikajain24,can you try to use tSchemaComplianceCheck?

please go-though the below link to more about tSchemaComplianceCheck

https://help.talend.com/reader/g8zdjVE7fWNUh3u4ztO6Dw/dmSJyJcYe2p5cHIsy22nnw

[jaink]

I cannot define the schema explicitly. Therefore might not be able to use compliance check component

[Anonymous · Accepted solution]

Hi

Take a look at this article and try the solutions.

https://community.talend.com/t5/Design-and-Development/Executing-a-Linux-command-with-a-pipe-using-a...

Regards

Shong