I have a job that fetches n rows from a database at the start and then processes each row, fetching data from REST APIs for the row and then storing the row in a database table. The database output is committed for each row one at a time. If there is an error from the REST API or database store then an error message or SQL exception gets logged to the console and the flow continues onto the next row.
At the end of the job I want to minimally know that at least one row had an error so I can send email to administrators. In the best case I would have a list of every error message.
What is the best practice to capture errors when a failed row does not kill the job?
I have done this before by attaching the error row from every REST and database component to a tJavaRow that stores the error data in the global map. I inspect that map at the end of the job and send email if the map is not empty. This is not ideal because it adds a lot of extra, custom, redundant coding. I have to add the tJavaRow to every component that could possibly error.
What is the solution with minimal coding that leverages Talend's builtin error logging capabilities?
Thanks in advance for any help.
Would you mind posting your work flow screenshots on community? Which will be helpful for us to get more information and details from your workflow.
Please mask your sensitive data.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Part 2 of a series on Context Variables
Learn how to do cool things with Context Variables
Find out how to migrate from one database to another using the Dynamic schema