topic Re: Creating file names based on row content in Talend Studio

Creating file names based on row content

kleinmat — Wed, 22 Jul 2015 17:39:31 GMT

Hi,
I Need to write a Job that will read a file that carries a date in each row. Example:
2015-07-20 bla
2015-07-20 blu
2015-07-21 lala
I will then Need to sort each row into a file that is named after the date:
File 2015-07-20_file will receive 2 rows, file 2015-07-21 will receive 1 row
How can that be done?

Re: Creating file names based on row content

Anonymous — Wed, 22 Jul 2015 18:13:24 GMT

This can get complicated.
One you can put all your data rows into an object to read through them. Use the tHashInput and tHashOutput
Then feed that tHashInput to a remove duplicates based on the date column. if you have it, use the tUniqueRow.
Then for each unique date, use a tJavaRow to create a file with the date on it. This is one SubJob
Then another subjob starts and feed it the tHashInput with all the dates in it.
Then use the tJavaRow, to read the date and feed that row to its corresponding Date_File.
this is one quick way to do it. hope this helps.

Re: Creating file names based on row content

kleinmat — Wed, 22 Jul 2015 18:22:54 GMT

Hi, thanks for the quick Response. The Problem with that approach might be that the files I am talking about are really large, so I can't really put them into an object.
I was hoping to simply extract the date from each row e.g. in a tMap and store that Information in some variable - which I could use in a tFileOutput component to create a dynamic Name.
But I don't know how that could work..

Re: Creating file names based on row content

Anonymous — Wed, 22 Jul 2015 18:48:09 GMT

I was going to suggest pretty much the same; but was beaten to it.
If your data is too large for HashMap, then try dropping it to a file.
It doesn't look like you've got a delimiter, so I'm guessing you have a fixed length record.
One general approach might be to...
Read the file using tFileInputPositional. Strip out the dates using tMap (othr components can do this too). Make the rows unique tUniqRow and then write then to tHashOutput.
Add a tHashInput and then connect this to tFlowToIterate. Now that you're iterating, you can refer to an entry put on tGlobalMap, so you've got the date.
You can then use this value to fire a single row from tFixedFlowInput, that you can then join back to a second tHashInput.
You should now be extracting the correct data, within each iteration.
You can now write this to a new file, again referencing the date that was put on globalMap, for your file name generation.