I have a job that reads a list of image url's from a DB and downloads each in an iterate loop. The job hangs at a random url in the tFileFetch. The debugger is no help:
I have the timeout set for 10 seconds (10000 ms) in the tFileFetch
I set the timeout to 10ms and it did finish but missed 80% of the files.
I set it to 100ms and it hung...
Any advice at all on how to proceed would be greatly appreciated. I am running v6.2.1
Could you please also post your current job design screenshot into forum? How does your loop work?
Here is the main job. It looks at two url sources. Then I iterate to a subjob to fetch the two images. The url's are passed as params:
The subjob checks to see if the url field is null or blank. If not it tries the fetch
There are about 4000 images to download, and It get anywhere between 10 and 500 before it hangs. Files come from many different locations. When it hangs I see the file it's trying to download in the folder in my work directory, but it shows a size of 0. When it hangs it will sit forever - until you manually cancel the job.
I'm going to try building a simplified test job to see if I can get more insight. One without a subjob and multiple iterate operations. And another test job that just tries to download the same file in a iterate loop.
another update. I created a simple job and connected a tloop for 1 to 3000 ----> iterate -----> tfilefetch
the file is the same file every time:
after a random number of downloads, the Talend job hangs. I quickly try the url in a browser, and it spins. So the server is slow...
But the question is, why doesn't the tfilefetch give up and move on to the next iteration after the specified number of milliseconds?