tFileOutputDelimited

Anonymous — Sat, 16 Nov 2024 05:20:52 GMT

Dear community,

I have the following scenario: I want to output a lot of rows (couple of millions) to multiple CSV files (chunks) using tFileOutputDelimited as I do not want to have just one big file. Those files have to be compressed with gzip and uploaded to the cloud. As disk space is limited I cannot simply export everything and then do the compression and upload but it must be done for each chunk before exporting the next chunk.

There are two options to get that done (Option 2 is currently working but can not be used in this cases unfortunately):

option 1) Using the split option in the tFileOutputDelimited and define how much rows can be wrote before creating a new chunk.

option 2) Make a loop and limit the number of rows for each iteration for the export and send it to tFileOutputDelimited () instead of getting all rows at a time like

To see the whole post, download it here

Re: tFileOutputDelimited

Anonymous — Mon, 05 Aug 2019 07:47:31 GMT

Hello,

If you want to get the filename of each chunk per iteration, here is a tflowtoiterate component which reads data line by line from the input flow and stores the data entries in iterative global variables.

For more information, please have a look at this component reference about:TalendHelpCenter:tFlowToIterate.

Best regards

Sabrina

topic tFileOutputDelimited in Talend Studio

tFileOutputDelimited

Re: tFileOutputDelimited