tfilefetch - java.io.FileNotFoundException (Too many open files)

[Anonymous]

Hi,
I have a job, (TIS 4.0.1 - Java), which is meant to retrieve approx 85,000 images from urls, im using the tFileFetch component to do this, however of these 85,000 it did 34,764 then i recieved the below error saying that there are too many open files, the job server is set to allow 1024 open files. the timeout on the tfilefetch component is 5000.
I was hoping not to simply increase the number of allowed open files to get round the problem? Is the component not closing the open files properly? Will reducing the timeout of the component help at all?
I dont know why it states defaultfilename.txt as the filename as it wasn't set to get any file called defaultfilename.txt?
Exception in component tFileFetch_1
java.io.FileNotFoundException: /mnt/datafeeds/step02/6108/defaultfilename.txt (Too many open files)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.(FileOutputStream.java:179)
at java.io.FileOutputStream.(FileOutputStream.java:131)
at spitfire_dev.sf007_dealeredit_0_1.SF007_dealeredit$1tFileFetch_1Thread.run(SF007_dealeredit.java:47951)
at routines.system.ThreadPoolWorker.runIt(TalendThreadPool.java:159)
at routines.system.ThreadPoolWorker.runWork(TalendThreadPool.java:150)
at routines.system.ThreadPoolWorker.access$0(TalendThreadPool.java:145)
at routines.system.ThreadPoolWorker$1.run(TalendThreadPool.java:122)
at java.lang.Thread.run(Thread.java:619)
Thanks in advance for your help.

[Anonymous]

Hello

I dont know why it states defaultfilename.txt as the filename as it wasn't set to get any file called defaultfilename.txt?

How do you set the URI? Can you upload a screenshot of your job?
Best regards
Shong

[Anonymous]

To set the maximum number of open file descriptors, try followed command on terminal:
ulimit -n size

[Anonymous]

Hi attached is the part of the job to get the images,
I tried creating it as a separate job and running it as an independant process, this didnt work.
Thanks for your help.

[Anonymous]

Hi Chris
From the screenshots, I see you are using 8 iterate at one time, so there are so many files open at one time.
1)Have a try to decrease the iterate number, x2 or x4.
2)Increase the number of allowed open files.
Best regards
Shong

[Anonymous]

thanks for the response,
this job has to retrieve over 85,000 images though so reducing the speed isnt really an option, with a single thread it was going to take over 15 hours. and still failed at 31k.
Why would 8 executions cause so many files to be open? surely once it has retrieved that image the file is closed?

[Anonymous]

Hi

Why would 8 executions cause so many files to be open? surely once it has retrieved that image the file is closed?

Before the file are closed, maybe there are many files are open by different threads at one time, because you are using 8 executions. however, you said you still get the same problem when using a single thread, what's the result if you increase the number of allowed open files? see
http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/
Best regards
Shong

[Anonymous]

we're just trying with 4096 allowed files. will let you know how we get on with this...

[Anonymous]

We tried this, it still failed.

[Anonymous]

Hello,
I have same problem.
Me i have litte more than 2000 files and works only for about 1600.
I have try linux command about maximum opened files and it gives me more 1,5 million so i don't think it's an OS limit only.
And i don't understand if i try to up more parallel execution it's better than alone ?