Design Question: Create separate state files from 1 input file
Hi,
I have a design question (Talend Open Studio for Data Integration) : I have 1 input csv file that contains over 1 million rows with 70 columns per row. One of the columns is the state code i.e. FL (Florida), I would like to create separate output files (CSV and excel) for each state (50 states).
Years ago, I created a separate job for each state containing the following:
(1 Input file (tFileInputDelimited) and tMap and 2 output files (tFileOutputDelimited and tFileOutputExcel).
So, I'm reading/passing the same input file 50 times in 50 jobs. This takes forever.
Is there a better way to create these output files by state in one or less jobs?
@richard profusek , first read states and itereate those using tflowtoiterate and then read the hole file and filter values based on the what you get from tflowtoiterate and use in tfileoutput component to generate the file name.
