tfilefetch - java.io.FileNotFoundException (Too many open files)

One Star

tfilefetch - java.io.FileNotFoundException (Too many open files)

Hi,
I have a job, (TIS 4.0.1 - Java), which is meant to retrieve approx 85,000 images from urls, im using the tFileFetch component to do this, however of these 85,000 it did 34,764 then i recieved the below error saying that there are too many open files, the job server is set to allow 1024 open files. the timeout on the tfilefetch component is 5000.
I was hoping not to simply increase the number of allowed open files to get round the problem? Is the component not closing the open files properly? Will reducing the timeout of the component help at all?
I dont know why it states defaultfilename.txt as the filename as it wasn't set to get any file called defaultfilename.txt?
Exception in component tFileFetch_1
java.io.FileNotFoundException: /mnt/datafeeds/step02/6108/defaultfilename.txt (Too many open files)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.(FileOutputStream.java:179)
at java.io.FileOutputStream.(FileOutputStream.java:131)
at spitfire_dev.sf007_dealeredit_0_1.SF007_dealeredit$1tFileFetch_1Thread.run(SF007_dealeredit.java:47951)
at routines.system.ThreadPoolWorker.runIt(TalendThreadPool.java:159)
at routines.system.ThreadPoolWorker.runWork(TalendThreadPool.java:150)
at routines.system.ThreadPoolWorker.access$0(TalendThreadPool.java:145)
at routines.system.ThreadPoolWorker$1.run(TalendThreadPool.java:122)
at java.lang.Thread.run(Thread.java:619)
Thanks in advance for your help.
Community Manager

Re: tfilefetch - java.io.FileNotFoundException (Too many open files)

Hello
I dont know why it states defaultfilename.txt as the filename as it wasn't set to get any file called defaultfilename.txt?

How do you set the URI? Can you upload a screenshot of your job?
Best regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
Employee

Re: tfilefetch - java.io.FileNotFoundException (Too many open files)

To set the maximum number of open file descriptors, try followed command on terminal:
ulimit -n size
One Star

Re: tfilefetch - java.io.FileNotFoundException (Too many open files)

Hi attached is the part of the job to get the images,
I tried creating it as a separate job and running it as an independant process, this didnt work.
Thanks for your help.
Community Manager

Re: tfilefetch - java.io.FileNotFoundException (Too many open files)

Hi Chris
From the screenshots, I see you are using 8 iterate at one time, so there are so many files open at one time.
1)Have a try to decrease the iterate number, x2 or x4.
2)Increase the number of allowed open files.
Best regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: tfilefetch - java.io.FileNotFoundException (Too many open files)

thanks for the response,
this job has to retrieve over 85,000 images though so reducing the speed isnt really an option, with a single thread it was going to take over 15 hours. and still failed at 31k.
Why would 8 executions cause so many files to be open? surely once it has retrieved that image the file is closed?
Community Manager

Re: tfilefetch - java.io.FileNotFoundException (Too many open files)

Hi
Why would 8 executions cause so many files to be open? surely once it has retrieved that image the file is closed?

Before the file are closed, maybe there are many files are open by different threads at one time, because you are using 8 executions. however, you said you still get the same problem when using a single thread, what's the result if you increase the number of allowed open files? see
http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/
Best regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: tfilefetch - java.io.FileNotFoundException (Too many open files)

we're just trying with 4096 allowed files. will let you know how we get on with this...
One Star

Re: tfilefetch - java.io.FileNotFoundException (Too many open files)

We tried this, it still failed.
One Star

Re: tfilefetch - java.io.FileNotFoundException (Too many open files)

Hello,
I have same problem.
Me i have litte more than 2000 files and works only for about 1600.
I have try linux command about maximum opened files and it gives me more 1,5 million so i don't think it's an OS limit only.
And i don't understand if i try to up more parallel execution it's better than alone ?
Community Manager

Re: tfilefetch - java.io.FileNotFoundException (Too many open files)

Hi dtournant
I can't reproduce the problem at the moment, can you simple your job and help us to reproduce it?
Best regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: tfilefetch - java.io.FileNotFoundException (Too many open files)

Ok Shong, thanks.
I have try to increase iterate value, and it looks better and same work for tfilecopy but not for TfileFectch.
If some URL are bad , it doesn't go away to the problem ?
One Star

Re: tfilefetch - java.io.FileNotFoundException (Too many open files)

I have done a simple job and launch it.
I have at left file with 2500 URL. On the 1002 rows and i get again error "too many open files"