topic Re: Split file rows into multiple files in Talend Studio

Split file rows into multiple files

Anonymous — Sat, 16 Nov 2024 14:12:48 GMT

Hi all
I have a 80 Million row file, which brings my oracle instance to explode...
I can handle 10 Mio rows in one iteration.
Has anybody a suggestion how to read the file and iterate over each 10 Mio segment ?
Thanks in advance,
Benjamin

Re: Split file rows into multiple files

Anonymous — Tue, 30 Sep 2008 17:02:37 GMT

hi,
i get teh same probleme.
I want to split a file every change of a key
AA;e
AA;r
BB;e
the result would be 1 file with all the AA and one file with BB etc
thx

Re: Split file rows into multiple files

Anonymous — Tue, 30 Sep 2008 18:23:05 GMT

Dobprofi - on the tFileOutputDelimited component - see this ;
http://talendforge.org/forum/postgallery.php?pid=10763&filename=schema.png
Chris - have a look at this....
http://talendforge.org/forum/viewtopic.php?id=2533

Re: Split file rows into multiple files

Anonymous — Wed, 01 Oct 2008 08:40:37 GMT

Hi SMaz
great contribution, Thank you!
I was actually not looking at this component, because I am using the tOracleOutputBulk, which is in fact about twice as fast as tFileOutput. But after your suggestion I reconsider using tFileOutput.
Cheers
Benjamin

Re: Split file rows into multiple files

Anonymous — Fri, 17 Oct 2014 08:58:48 GMT

Hi,
I've written a new tutorial on "how to split a file into many files regarding a key on each record" which explains how to solve this kind of task. It is actually only available in french. The tutorial shows 3 different technics to achieve this task.
Hope it can be useful.