topic how to split text file in talend DI in Talend Studio

how to split text file in talend DI

Anonymous — Sun, 29 Nov 2015 08:55:32 GMT

I have 100000 lines of text file, how to split file in target folder as each text file of 100 lines each

Re: how to split text file in talend DI

Anonymous — Sun, 29 Nov 2015 13:43:21 GMT

on Linux/Unix use the split commd:

split <textfile> --lines=100

Re: how to split text file in talend DI

Anonymous — Sun, 29 Nov 2015 15:31:03 GMT

Thanks for reply Robert,
Do we have any talend component to split file because I need to develop a job that should work for any platform.

Re: how to split text file in talend DI

Anonymous — Mon, 30 Nov 2015 06:42:38 GMT

Hi,
There is an option 'Split output in several files' in the advanced setting tab of tFileOutputDelimid. You can generate N rows(100) in each file.

Feel free to let us know if it is what you are looking for.
Best regards
Sabrina

Re: how to split text file in talend DI

Anonymous — Wed, 02 Dec 2015 14:33:43 GMT

Thanks for your reply,
But while writing from tFileList to tFileoutputdelimited, it writes only last file from directory.
Please give the solution to writes all files data from dir to target.
Please ref image attahced

Re: how to split text file in talend DI

Anonymous — Thu, 03 Dec 2015 04:22:29 GMT

Hi praneeth31,

Have you tried to check out "Append" option in tFileoutputdelimited component to see if works?

Best regards
Sabrina

Re: how to split text file in talend DI

Anonymous — Thu, 03 Dec 2015 07:53:55 GMT

Yes, I used Append selection at tFIleOutputDelimited!
refer diagram to check any mistake i did.

Re: how to split text file in talend DI

Anonymous — Thu, 03 Dec 2015 09:10:04 GMT

Hi,
Could you please give us more descriptions about your job design?
We see the work flow:
tfileinputregex_1-->tfileouputdelimited_1(split your 100000 lines into 100 lines each? )-->oncomponentOk-->tfilelist-->iterate-->tfileinputregex_2-->tmap-->tfileoutputdelimited_2.
What's the purpose for subjob2? How did you set filemask on tfilelist component?

Best regards
Sabrina

Re: how to split text file in talend DI

Anonymous — Thu, 03 Dec 2015 10:48:23 GMT

I didn't use any file masking, I just want to split file to dir, add all files using tFileList to target by adding new columns(I didn't used it now) using tMap.

Re: how to split text file in talend DI

preddy79 — Sat, 11 Mar 2017 11:52:04 GMT

Hi,
I've similar job to pull 100000 from db into an outputfile to split each file with 1000 records. When the job generates a file it writes 100 files in 2 mins but it keeps writing again and again starts with file1 till file100 and continues for 1 hour. I dont understand why it is doing so...
When i write single file it completes in 5 mins... but when i split because of this over writing it is running for 1 hour. Can yu please help me why it is happening and how to fix it.
thanks

Re: how to split text file in talend DI

vapukov — Sat, 11 Mar 2017 11:59:24 GMT

preddy79 wrote:
Hi,
I've similar job to pull 100000 from db into an outputfile to split each file with 1000 records. When the job generates a file it writes 100 files in 2 mins but it keeps writing again and again starts with file1 till file100 and continues for 1 hour. I dont understand why it is doing so...
When i write single file it completes in 5 mins... but when i split because of this over writing it is running for 1 hour. Can yu please help me why it is happening and how to fix it.
thanks

You need attache screenshots of Your Job, without it only guessing