One Star

Too many opened files

This is a schema of my job:
tMySqlConnection_1 ----onComponentOk----> tMysqlInput_1 --row1--> tFlowIterator_1 --X20--> tFileFetch_1 ----onComponentOk---->
(continue) -> tFileInputXml_1 --row2--> tJavaRow_1 --row3--> tMap_1 --main (insert)--> tMysqlOutput_1
To put in words:
Connect to the db (tMySqlConnection_1), query from the db (tMysqlInput_1).
For each row (tFlowIterator_1), fetch an XML file (tFileFetch_1) and save it.
Open each file (tFileInputXml_1), do some logic (tJavaRow_1, tMap_1) and insert it to a table (tMysqlOutput_1).
When the iterator's parallel value was 300, I would get "too many open files exception" after <3000 files. When I lowered it to 20, I get the exception after 155,000 files. I have a total of ~200,000 files I need to process.
I changed the Xmx to 2048, didn't help.
So my questions are:
1. Is there a way to enforce closing the files or have a max opened files enforced?
2. Another solution is to maybe have a master job that'll run this job several times for a different batch each time
Please advise, I'm stuck on this for more than a week...
Six Stars

Re: Too many opened files

Hi Leeransetton,
If you are on linux and have control of the machine, you could try to change ulimit to " ulimit -n "
You can find more information there as well
Tell us if it has solved your issue.

Books are the quietest and most constant of friends; they are the most accessible and wisest of counsellors, and the most patient of teachers.
--Charles W. Eliot (1834 - 1926), The Happy Life, 1896
One Star

Re: Too many opened files

Hey Adrien,
Thanks for the reply!
Raised the number to 4096, but the problem persists..
One Star

Re: Too many opened files

Ended up writing the job in Java becasue nothing helped..