How to split the ouput files into multiple files depending upon the Date

[Aish123]

Hi @rhall , @shong ,

I have a input CSV file with around 5-6million records.

I want to generate an XML file as the output but i want to split the XML files into multiple depending upon the year.

There is invoice date column in the CSV file.

So, the first output file should be 2007.xml which will have only the data of 2007.

Then the 2nd  file will be 2008.xml which will have only the data for 2008 and so on.

Can you please let me know how this can be achieved dynamically as the no of rows for each year are not fixed. 

Also, i want to store the unique count of each year somewhere..ie. 2007 year has 6rows , 2008 has 5 rows and so on.

Thank you.

[Anonymous]

A way you could approach this might be to try something along the lines of this solution I posted....
https://community.talend.com/t5/Design-and-Development/Write-records-into-multiple-files-based-on-a-...

Now you will need to add to this solution, since the tJavaFlex simply writes to a file using Java. In your case, you need to chunk up your records into batches (per year) and then read them out in batches by year. To achieve this, you might be able to crib from this solution I posted....
https://community.talend.com/t5/Design-and-Development/Batch-processing-in-talend-job/m-p/52000/high...

Then, once you are able to group by year and read batches by year, you should be able to use a tXMLMap to produce your XML by year.

[Aish123]

Hi @rhall ,

Thanks for your response.

I tried with the first link which you have provided and i was able to generate different files depending on the year.

But i am not able to understand how to split them into different files based on years. I went through the second solution as well but not able to understand how to implement that. Can you please help me with this.

[Anonymous]

Hi @not specified not specified,

You are very close to the solution. My idea would be to add another column for the year in a tMap. There you need a little Java to extract the year from the date. Talend already offers a Java method for this.

Create a new column called "Year" in the output, which you then use to split the files. You calculate this with

TalendDate.formatDate("yyyy", row1.InvoiceDate)

Cheers,

Steffen