Error logging best practice with single row commit.

Nine Stars

Error logging best practice with single row commit.

I have a job that fetches n rows from a database at the start and then processes each row, fetching data from REST APIs for the row and then storing the row in a database table.  The database output is committed for each row one at a time.  If there is an error from the REST API or database store then an error message or SQL exception gets logged to the console and the flow continues onto the next row.

 

At the end of the job I want to minimally know that at least one row had an error so I can send email to administrators.  In the best case I would have a list of every error message.

 

What is the best practice to capture errors when a failed row does not kill the job?

I have done this before by attaching the error row from every REST and database component to a tJavaRow that stores the error data in the global map.  I inspect that map at the end of the job and send email if the map is not empty.  This is not ideal because it adds a lot of extra, custom, redundant coding.  I have to add the tJavaRow to every component that could possibly error.

 

What is the solution with minimal coding that leverages Talend's builtin error logging capabilities?

 

Thanks in advance for any help.

--
Please give Kudos and mark topics as solved where appropriate.
Moderator

Re: Error logging best practice with single row commit.

Hello,

Would you mind posting your work flow screenshots on community? Which will be helpful for us to get more information and details from your workflow.

Please mask your sensitive data.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog