the purpose of my job is transferring files connecting to multiple ftp whose connection parameters are retrieved from a database.
The directory containing the files to be delivered is scanned and the files are sent to the expected ftp.
There are two main loops in my job as can be seen on the screenshot.
I have some trouble on the FTP part of my job as described below :
While the two FTP components have their "Die on error" parameter unchecked, my job isn't reliable if an ftp server is not available or is returning an error code.
The error is not caught as expected.
Indeed my job fails at the first error, for example a bad password, and my job will quit while I would like it to continue and process the waiting files for the remaining ftp.
=> How can I make the ftp components truly "fail-proof" (ignore the rotten apple) ?
Thank you for your answer.
You can use tContextLoad or implicit tContextLoad to retrieve your FTP connection parameters from a database.
The first part can be triggered the next subjob by using "OnSubJobOk" connection type on the condition that the main subjob completed without error. This connection is to be used only from the start component of the Job.
The workflow looks like:
retrieving the ftp parameters and creating a successful connection was OK, I managed to do the first loop, thank you.
My problem is actually on the second loop, handling errors in ftp transfers :
what I observe is that my job fails if there's any connection error on a ftp, for example a timeout, a bad password, a wrong host, etc, whereas I want my job to continue and process the next files and their related ftps remaining in the queue.
Please note that while the "Die on error" parameter of the ftp components is uncheked my job still "dies".
A design with a job for the first loop and a subjob for the ftp part (second loop) would make it "fail-proof" ?
Introduction to Talend Open Studio for Data Integration.
Practical steps to developing your data integration strategy.
Create systems and workflow to manage clean data ingestion and data transformation.