topic Merge Multiple Text Files into one with a row at the beginning of each file in Talend Studio

Merge Multiple Text Files into one with a row at the beginning of each file

MRup — Sat, 16 Nov 2024 00:00:48 GMT

Hi,

I am new to Talend and need help from your guys.

I have to Merge multiple text files from a directory which I was able to do with following

tFileList -> tFileInputFullRow -> tFileOutputDelimited

Here is the challenging part for me ....

At the Beginning of each text file merge I have to add a row to the output and the row should contain (as though it's an identification that a new file data has begun)

Note : Above line should only occur only once when a new text file is picked to be merged

Files are sitting in following directory

U:\Talend_GOAnywhere_Jobs\Inventory_Jobs\s0325366\up\IM25366.txt

U:\Talend_GOAnywhere_Jobs\Inventory_Jobs\s1628062\up\IM28062.txt

U:\Talend_GOAnywhere_Jobs\Inventory_Jobs\s2951288\up\IM51288.txt

<foldername> holds value 0325366 for 1st file; 1628062 for 2nd file; 2951288 for 3rd file and so on .......

yyyymmddhhmmss -> Current Talend Date

11 character space

Sequenceno of text file merging

So for 2nd text file following is what will be added at the beginning before the file gets merged

162806220210625031406 0000002

I tried adding getCurrentDate variable to tMap but it appending to every row in text file at the beginning.

Each text file can have as many as 400-500 lines and it's appending each of those line.

Please help

Thanks

Re: Merge Multiple Text Files into one with a row at the beginning of each file

Anonymous — Thu, 01 Jul 2021 10:38:06 GMT

Need to check if it is the 1st file or the 2nd file using runIf trigger, for example, the job design looks like:

tFileList--iterate--tJava--runIf1--tFileInputFullRow -> tFileOutputDelimited

--runIf2--tFixedFlowInput--main-tFileOutputDelimited--onComponentOK--tFileInputFullRow -> tFileOutputDelimited

on tJava, get the current folder name, current date string, generate the Sequenceno and store these value to global variables for used later on tFixedFlowInput.

on tFixedFlowInput, generate the new row and append it to the target file before reading the 2nd file.

set the condition of runIf1:

((Integer)globalMap.get("tFileList_1_NB_FILE"))==1

set the condition of runIf2:

((Integer)globalMap.get("tFileList_1_NB_FILE"))!=1

Hope it helps!

Regards

Shong

Re: Merge Multiple Text Files into one with a row at the beginning of each file

MRup — Thu, 08 Jul 2021 18:46:41 GMT

Thankyou so much shong for your help.

It works. Truly Appreciated