topic Too many opened files in Talend Studio

Too many opened files

Anonymous — Wed, 08 Feb 2017 16:19:44 GMT

This is a schema of my job:
tMySqlConnection_1 ----onComponentOk----> tMysqlInput_1 --row1--> tFlowIterator_1 --X20--> tFileFetch_1 ----onComponentOk---->
(continue) -> tFileInputXml_1 --row2--> tJavaRow_1 --row3--> tMap_1 --main (insert)--> tMysqlOutput_1
To put in words:
Connect to the db (tMySqlConnection_1), query from the db (tMysqlInput_1).
For each row (tFlowIterator_1), fetch an XML file (tFileFetch_1) and save it.
Open each file (tFileInputXml_1), do some logic (tJavaRow_1, tMap_1) and insert it to a table (tMysqlOutput_1).
When the iterator's parallel value was 300, I would get "too many open files exception" after <3000 files. When I lowered it to 20, I get the exception after 155,000 files. I have a total of ~200,000 files I need to process.
I changed the Xmx to 2048, didn't help.
So my questions are:
1. Is there a way to enforce closing the files or have a max opened files enforced?
2. Another solution is to maybe have a master job that'll run this job several times for a different batch each time
Please advise, I'm stuck on this for more than a week...
Thanks!

Re: Too many opened files

Anonymous — Thu, 09 Feb 2017 03:11:02 GMT

Hi Leeransetton,
If you are on linux and have control of the machine, you could try to change ulimit to " ulimit -n "
You can find more information there as well http://stackoverflow.com/questions/4289447/java-too-many-open-files
Tell us if it has solved your issue.
Thanks,

Re: Too many opened files

Anonymous — Thu, 09 Feb 2017 16:41:18 GMT

Hey Adrien,
Thanks for the reply!
Raised the number to 4096, but the problem persists..

Re: Too many opened files

Anonymous — Sun, 12 Feb 2017 18:04:48 GMT

UPDATE:
Ended up writing the job in Java becasue nothing helped..