topic merging multiple files with the same schema in Talend Studio

merging multiple files with the same schema

Anonymous — Sat, 16 Nov 2024 13:48:37 GMT

I have multiple files with the same schema. They are already sorted. I want to merge them together into a single file that is also sorted by the same field.
I can use tUnite --> tSortRow but this is highly inefficient, plus I get an out-of-memory error
Is there a "version" of tUnite that reads multiple files and outputs the records in sorted order?
Suggestions?
Thanks,
--eric

Re: merging multiple files with the same schema

Anonymous — Fri, 21 Aug 2009 03:12:35 GMT

Hello
tUnite is the most suitable component for merging records.
You can try the following way:
1)Go to Windows/Preferences/Talend/RunDebug, and modify the vm argument in "Job Run VM arguments" table.
2)Split your job into two subjob, one is merge all the records into a temporary file, another is extract records from temporary file and sort them, output them to target file.
Best regards

shong