topic Re: tfilefetch - java.io.FileNotFoundException (Too many open files) in Talend Studio

tfilefetch - java.io.FileNotFoundException (Too many open files)

Anonymous — Sat, 16 Nov 2024 13:23:24 GMT

Hi,
I have a job, (TIS 4.0.1 - Java), which is meant to retrieve approx 85,000 images from urls, im using the tFileFetch component to do this, however of these 85,000 it did 34,764 then i recieved the below error saying that there are too many open files, the job server is set to allow 1024 open files. the timeout on the tfilefetch component is 5000.
I was hoping not to simply increase the number of allowed open files to get round the problem? Is the component not closing the open files properly? Will reducing the timeout of the component help at all?
I dont know why it states defaultfilename.txt as the filename as it wasn't set to get any file called defaultfilename.txt?
Exception in component tFileFetch_1
java.io.FileNotFoundException: /mnt/datafeeds/step02/6108/defaultfilename.txt (Too many open files)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.(FileOutputStream.java:179)
at java.io.FileOutputStream.(FileOutputStream.java:131)
at spitfire_dev.sf007_dealeredit_0_1.SF007_dealeredit$1tFileFetch_1Thread.run(SF007_dealeredit.java:47951)
at routines.system.ThreadPoolWorker.runIt(TalendThreadPool.java:159)
at routines.system.ThreadPoolWorker.runWork(TalendThreadPool.java:150)
at routines.system.ThreadPoolWorker.access$0(TalendThreadPool.java:145)
at routines.system.ThreadPoolWorker$1.run(TalendThreadPool.java:122)
at java.lang.Thread.run(Thread.java:619)
Thanks in advance for your help.

Re: tfilefetch - java.io.FileNotFoundException (Too many open files)

Anonymous — Thu, 17 Jun 2010 04:40:30 GMT

Hello

I dont know why it states defaultfilename.txt as the filename as it wasn't set to get any file called defaultfilename.txt?

How do you set the URI? Can you upload a screenshot of your job?
Best regards
Shong

Re: tfilefetch - java.io.FileNotFoundException (Too many open files)

Anonymous — Thu, 17 Jun 2010 07:51:06 GMT

To set the maximum number of open file descriptors, try followed command on terminal:
ulimit -n size

Re: tfilefetch - java.io.FileNotFoundException (Too many open files)

Anonymous — Thu, 17 Jun 2010 09:05:57 GMT

Hi attached is the part of the job to get the images,
I tried creating it as a separate job and running it as an independant process, this didnt work.
Thanks for your help.

Re: tfilefetch - java.io.FileNotFoundException (Too many open files)

Anonymous — Thu, 17 Jun 2010 10:44:31 GMT

Hi Chris
From the screenshots, I see you are using 8 iterate at one time, so there are so many files open at one time.
1)Have a try to decrease the iterate number, x2 or x4.
2)Increase the number of allowed open files.
Best regards
Shong

Re: tfilefetch - java.io.FileNotFoundException (Too many open files)

Anonymous — Thu, 17 Jun 2010 10:48:01 GMT

thanks for the response,
this job has to retrieve over 85,000 images though so reducing the speed isnt really an option, with a single thread it was going to take over 15 hours. and still failed at 31k.
Why would 8 executions cause so many files to be open? surely once it has retrieved that image the file is closed?

Re: tfilefetch - java.io.FileNotFoundException (Too many open files)

Anonymous — Thu, 17 Jun 2010 11:12:59 GMT

Why would 8 executions cause so many files to be open? surely once it has retrieved that image the file is closed?

Before the file are closed, maybe there are many files are open by different threads at one time, because you are using 8 executions. however, you said you still get the same problem when using a single thread, what's the result if you increase the number of allowed open files? see
http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/
Best regards
Shong

Re: tfilefetch - java.io.FileNotFoundException (Too many open files)

Anonymous — Thu, 17 Jun 2010 11:14:24 GMT

we're just trying with 4096 allowed files. will let you know how we get on with this...

Re: tfilefetch - java.io.FileNotFoundException (Too many open files)

Anonymous — Tue, 22 Jun 2010 12:42:41 GMT

We tried this, it still failed.

Re: tfilefetch - java.io.FileNotFoundException (Too many open files)

Anonymous — Fri, 16 Jul 2010 15:33:24 GMT

Hello,
I have same problem.
Me i have litte more than 2000 files and works only for about 1600.
I have try linux command about maximum opened files and it gives me more 1,5 million so i don't think it's an OS limit only.
And i don't understand if i try to up more parallel execution it's better than alone ?

Re: tfilefetch - java.io.FileNotFoundException (Too many open files)

Anonymous — Fri, 16 Jul 2010 15:37:57 GMT

Hi dtournant
I can't reproduce the problem at the moment, can you simple your job and help us to reproduce it?
Best regards
Shong

Re: tfilefetch - java.io.FileNotFoundException (Too many open files)

Anonymous — Fri, 16 Jul 2010 18:07:46 GMT

Ok Shong, thanks.
I have try to increase iterate value, and it looks better and same work for tfilecopy but not for TfileFectch.
If some URL are bad , it doesn't go away to the problem ?

Re: tfilefetch - java.io.FileNotFoundException (Too many open files)

Anonymous — Fri, 16 Jul 2010 18:53:39 GMT

I have done a simple job and launch it.
I have at left file with 2500 URL. On the 1002 rows and i get again error "too many open files"