It appears that the tStatCatcher and tLogCatcher are actually run only at the end of a job -- which sort of makes sense given what they collect. Can anyone confirm? The problem this appears to create is, when using tMSSqlClose component to end the connection at post-job, it is getting fired before one or the other of the tStat or tLog are completed.
I would have expected that everything was completed before the tPostJob is fired.
If I remove the tMSSqlClose, it runs without errors -- but then the connection is not closed, right?
We've done something very similar, but rather than writing to a database, we write to a Kafka topic. We then have a consumer that reads from the Kafka topic and pushes the log messages to the ELK stack that comes with the TAC, but you could easily write a consumer that writes them to the database of your choice. Better still, once you have the log messages on the Kafka topic, you could write them to both an ELK setup and database at the same time.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Learn how to do cool things with Context Variables
Find out how to migrate from one database to another using the Dynamic schema
Pick up some tips and tricks with Context Variables