the purpose of my job is transferring files connecting to multiple ftp whose connection parameters are retrieved from a database.
The directory containing the files to be delivered is scanned and the files are sent to the expected ftp.
There are two main loops in my job as can be seen on the screenshot.
I have some trouble on the FTP part of my job as described below :
While the two FTP components have their "Die on error" parameter unchecked, my job isn't reliable if an ftp server is not available or is returning an error code.
The error is not caught as expected.
Indeed my job fails at the first error, for example a bad password, and my job will quit while I would like it to continue and process the waiting files for the remaining ftp.
=> How can I make the ftp components truly "fail-proof" (ignore the rotten apple) ?
Thank you for your answer.
You can use tContextLoad or implicit tContextLoad to retrieve your FTP connection parameters from a database.
The first part can be triggered the next subjob by using "OnSubJobOk" connection type on the condition that the main subjob completed without error. This connection is to be used only from the start component of the Job.
The workflow looks like:
retrieving the ftp parameters and creating a successful connection was OK, I managed to do the first loop, thank you.
My problem is actually on the second loop, handling errors in ftp transfers :
what I observe is that my job fails if there's any connection error on a ftp, for example a timeout, a bad password, a wrong host, etc, whereas I want my job to continue and process the next files and their related ftps remaining in the queue.
Please note that while the "Die on error" parameter of the ftp components is uncheked my job still "dies".
A design with a job for the first loop and a subjob for the ftp part (second loop) would make it "fail-proof" ?
I've got the same issue... and i'm really interested in the solution.
as well if the connections settings are embedded in a tFTPPut or if they are inherited from an existing tFTPConnection
Is there a solution arround "run if" and the "__DIE_ON_ERROR__" variable ?
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Pick up some tips and tricks with Context Variables
Learn how media organizations have achieved success with Data Integration
Create systems and workflow to manage clean data ingestion and data transformation.