I have the following scenario: I want to output a lot of rows (couple of millions) to multiple CSV files (chunks) using tFileOutputDelimited as I do not want to have just one big file. Those files have to be compressed with gzip and uploaded to the cloud. As disk space is limited I cannot simply export everything and then do the compression and upload but it must be done for each chunk before exporting the next chunk.
There are two options to get that done (Option 2 is currently working but can not be used in this cases unfortunately):
option 1) Using the split option in the tFileOutputDelimited and define how much rows can be wrote before creating a new chunk.
option 2) Make a loop and limit the number of rows for each iteration for the export and send it to tFileOutputDelimited () instead of getting all rows at a time like
If you want to get the filename of each chunk per iteration, here is a tflowtoiterate component which reads data line by line from the input flow and stores the data entries in iterative global variables.
