topic Re: File Data Type: Dynamic - Execute awk command on tSystem to store the output in a file in Talend Studio

File Data Type: Dynamic - Execute awk command on tSystem to store the output in a file

jaink — Sat, 16 Nov 2024 07:57:23 GMT

Hi,

I need to check that the number of fields in a file is as expected. As i am using dynamic data type, i am not able to check the same using getCountColumn() function, as it always takes the max number of fields present.

Therefore, as a workaround thought of using awk and storing the rows which matches the expected number of fields to a file.
Also, it is needed to store the non matching rows to a file.

Example -
source file - a.txt (data type defined as dynamic)
A,B,C
1,2,3
4,5,6,7
8,9

tSystem1 --> (main) tSystem2
Executed command - "" on tSystem
"/bin/bash -c awk -F, '{if(NF == 2) print $0 }' a.txt > b.txt"

But this command is not executed only i guess, because when executed, the command is passed as - '/bin/bash -c awk -F, '{if(NF == 2) print $0 }' a.txt

Please let me know if there is any other solution for the same, or what changes are needed in the existing code?

Re: File Data Type: Dynamic - Execute awk command on tSystem to store the output in a file

manodwhb — Tue, 17 Jul 2018 06:23:47 GMT

@kritikajain24,can you try to use tSchemaComplianceCheck?

please go-though the below link to more about tSchemaComplianceCheck

https://help.talend.com/reader/g8zdjVE7fWNUh3u4ztO6Dw/dmSJyJcYe2p5cHIsy22nnw

Re: File Data Type: Dynamic - Execute awk command on tSystem to store the output in a file

jaink — Tue, 17 Jul 2018 06:30:29 GMT

I cannot define the schema explicitly. Therefore might not be able to use compliance check component

Re: File Data Type: Dynamic - Execute awk command on tSystem to store the output in a file

Anonymous — Tue, 17 Jul 2018 06:36:26 GMT

Take a look at this article and try the solutions.

https://community.talend.com/t5/Design-and-Development/Executing-a-Linux-command-with-a-pipe-using-a-tSystem-component/ta-p/21671

Regards

Shong