topic Iterate over multiple files and create output by file name in Talend Studio

Iterate over multiple files and create output by file name

MDVV — Tue, 01 Jul 2025 18:58:27 GMT

Hello Team,

I hope you can help me with the following problem in talend studio:

I have several files in a directory, and I need to join them by file name. This will generate an output containing the iterated file names.

Example:

1-1000739.out
2-1000739.out
3-1000739.out
4-1000739.out
I need a single file to be created with the union of the 4 files and with the name: Output_1000739.json
and so on with the rest of the files found in the directory:

1-1000687.out
2-1000687.out
3-1000687.out
4-1000687.out

Output_1000687.json

I have a Subjob:

t_Filelits -->Iterate-->tFileInputRow-->tFileOutputDelimited(check Append option)

But it joins all the files into one.
Any ideas to solve this?

Thanks in advance!

Re: Iterate over multiple files and create output by file name

quentin-vigne — Wed, 02 Jul 2025 07:38:43 GMT

Hello @MDVV

You have the logic but what is missing is first a way of iterating only for a given "identifier" like "1000739" or "1000687"

What I would do is first : setup a tFileList to list all files that are like "*.out"

Then you can use a tJavaRow do apply this transformation and keep all the "identifier" in memory :

String fileName = input_row.currentFileName; // e.g., 1-1000739.out String[] parts = fileName.split("-"); String idPart = parts[1].replace(".out", ""); // 1000739 output_row.filePath = input_row.currentFilePath; output_row.identifier = idPart;

After this, setup a tUniqRow to keep only 1 occurence of each identifier and then add a tFlowToIterate (this way you can iterate over the uniques rows) then you can do your current flow : tFileList (add a filemask to use the current identifier from the tFlowToIterate) -->Iterate-->tFileInputRow-->tFileOutputDelimited (check Append option)

I hope this is clear enough

- Quentin

Re: Iterate over multiple files and create output by file name

MDVV — Wed, 02 Jul 2025 20:48:05 GMT

Hello, thanks for your response @quentin-vigne

I tried two options following your recommendation.

Option 1. With tJavaRow, which cannot be connected to a tFilelist, so I connected it to tFileInputRow (my files are not formatted). The tJavaRow code is as follows:

String fileName = ((String)globalMap.get("tFileList_6_CURRENT_FILE"));
String[] parts = fileName.split("-");
String idPart = parts[1].replace(".out", ""); // 1000739
output_row.content = ((String)globalMap.get("tFileList_6_CURRENT_FILEPATH"));
output_row.content = idPart;
 
and the subjob is as follows: 
The file returns the append of all the source files (.out).
Option 2. With tjava, which can be connected or not to a previous transformation. 
The tjava code is as follows:
String fileName = ((String)globalMap.get("tFileList_9_CURRENT_FILE")); // e.g., 1-1000739.out
String[] parts = fileName.split("-");
String idPart = parts[1].replace(".out", ""); // 1000739
String filePath = ((String)globalMap.get("tFileList_9_CURRENT_FILEPATH"));
String identifier = idPart;
 
and the subjob is as follows:
 
With this option, it also returns a single output file with the append of all the source files. 
I've been trying to resolve this issue for weeks. I'm new to Talend, and it's been very complicated.
 
 

Re: Iterate over multiple files and create output by file name

quentin-vigne — Thu, 03 Jul 2025 08:23:20 GMT

Hi @MDVV

I managed to do it with a change : In my first post I forgot to use the tIterateToFlow.

First of all :

tFileList_1 : add a file mask with "*.out". This way this component will list only desired files in your directory
tIterateToFlow_1 : This way you will transform the list of files in your directory to a simple row. Don't forget to add a column "CURRENT_FILE" for example with a value of

((String)globalMap.get("tFileList_1_CURRENT_FILE"))

Then you cann the tJavaRow I was talking about with this code :

String fileName = input_row.CURRENT_FILE; // e.g., 1-1000739.out String[] parts = fileName.split("-"); String idPart = parts[1].replace(".out", ""); // 1000739 output_row.identifier = idPart;

Edit the schema and add to the output a column named "identifier". You can also change it later if you want but you will have to change the outpur_row.identifier in the code.

Then add the tUniqRow and check the "Key attribute" for identifier column.
Add a tFlowToIterate to start an iteration over the uniques values of your files
Add a tFileList with a file mask having this value :

"*" + ((String)globalMap.get("row3.identifier")) + ".out"

with this we list every file that are like "*1000739.out" for example

Add your tFileInputRaw and fill in filename with the global variable :

((String)globalMap.get("tFileList_2_CURRENT_FILEPATH"))

Now you can finally add the tFileOutputDelimited with the output filename being

"C:/TalendWorkspace/test/Output_" + ((String)globalMap.get("row3.identifier")) + ".json"

I used some test file and it works on my end.

Have a good day

- Quentin

Re: Iterate over multiple files and create output by file name

MDVV — Fri, 04 Jul 2025 00:16:28 GMT

Your recommendation worked!

Thank you so much for sharing your knowledge! Thanks in advance!