topic CSV files (double quotes) and parquet files in Talend Studio

CSV files (double quotes) and parquet files

JackStrong — Tue, 07 Dec 2021 11:32:01 GMT

Hi All.

I have 3 questions. If someone can help I will grateful 🙂 .

1.[tFileOutputDelimited] Is there any way to force Talend to add double quotes as the text enclosure but only for such values where there is a field separator inside the string?

Desired output:

e.g. name;age;comment

Agent Smith;30;convalescent

Neo;29;"convalescent;12/04/21"

(only one field (semicolon inside) is double quoted)

Whenever I tried to achieve above I received:

e.g. name;age;comment

"Agent Smith";"30";"convalescent"

"Neo";"29";"convalescent;12/04/21"

2.Is it possible to write Parquete file by using only the standard jobs? And how to do that?

3.Is it possible to call the Big Data Batch child job from standard job? And if it is possible, is it safe/stable and recommended?

Best regards,

Jack Strong

Re: CSV files (double quotes) and parquet files

Anonymous — Wed, 08 Dec 2021 01:41:54 GMT

There isn't anything "out of the box" that will do this BUT this is one of the big advantages of Talend Studio. You can build the functionality to do this in a routine.

Here is a very quick example I have just knocked up.....

package routines;

public class ExtraCSVOptions {

public static String wrapSeparatorStrings(String data, String separator, String wrapCharacter) {

String returnVal = null;

if(data!=null && separator!=null && wrapCharacter!=null && data.indexOf(separator)>-1) {

returnVal = wrapCharacter+data+wrapCharacter;

}

return returnVal;

}

You would use this in a tMap or similar. It would be used for every column. So if you have 3 String columns like this.....

myColumn1

myColumn2

myColumn3

....you would use the above routine like this (assuming the separator is ";" and the wrap character is ")...

routines.ExtraCSVOptions.wrapSeparatorStrings(row1.myColumn1, ";", "\"")

routines.ExtraCSVOptions.wrapSeparatorStrings(row1.myColumn2, ";", "\"")

routines.ExtraCSVOptions.wrapSeparatorStrings(row1.myColumn3, ";", "\"")

For Parquet files we have the tFileInputParquet and tFileOutputParquet components.

You can start jobs via the TMC or TAC using the APIs.

Re: CSV files (double quotes) and parquet files

JackStrong — Mon, 17 Jan 2022 13:08:15 GMT

Sorry for being late.

@rhall, thank you very much. We added "else if" and it works fine!

Below whole routine:

package routines;

public class textWrapper {

public static String wrapSeparatorStrings(String data, String separator, String wrapCharacter) {

String returnVal = null;

if(data!=null && separator!=null && wrapCharacter!=null && data.indexOf(separator)>-1) {

returnVal = wrapCharacter+data+wrapCharacter;

}

else if(data!=null && separator!=null && wrapCharacter!=null && !(data.indexOf(separator)>-1)) {

returnVal = data;

}

return returnVal;

}

Regarding parquet file I'm not sure if I understand. Are you saying that I can trigger the standard job in TMC and then in some way I can call Big Data job?

Re: CSV files (double quotes) and parquet files

Anonymous — Mon, 17 Jan 2022 14:40:24 GMT

Oops, I missed the "else" condition there. I think I was focussed on solving the issue for records affected and overlooked unaffected records. Well spotted.

Regarding reading and writing Parquet files, the components I mentioned are mentioned here....

https://help.talend.com/r/en-US/7.3/parquet/parquet

You can make use of the TMC API for running jobs. The swagger API documentation is here...

https://api.us.cloud.talend.com/tmc/swagger/swagger-ui.html