This is a schema of my job:
tMySqlConnection_1 ----onComponentOk----> tMysqlInput_1 --row1--> tFlowIterator_1 --X20--> tFileFetch_1 ----onComponentOk---->
(continue) -> tFileInputXml_1 --row2--> tJavaRow_1 --row3--> tMap_1 --main (insert)--> tMysqlOutput_1
To put in words:
Connect to the db (tMySqlConnection_1), query from the db (tMysqlInput_1).
For each row (tFlowIterator_1), fetch an XML file (tFileFetch_1) and save it.
Open each file (tFileInputXml_1), do some logic (tJavaRow_1, tMap_1) and insert it to a table (tMysqlOutput_1).
When the iterator's parallel value was 300, I would get "too many open files exception" after <3000 files. When I lowered it to 20, I get the exception after 155,000 files. I have a total of ~200,000 files I need to process.
I changed the Xmx to 2048, didn't help.
So my questions are:
1. Is there a way to enforce closing the files or have a max opened files enforced?
2. Another solution is to maybe have a master job that'll run this job several times for a different batch each time
Please advise, I'm stuck on this for more than a week...
Thanks!
Hi Leeransetton,
If you are on linux and have control of the machine, you could try to change ulimit to "
ulimit -n "
You can find more information there as well
http://stackoverflow.com/questions/4289447/java-too-many-open-files Tell us if it has solved your issue.
Thanks,
