It appears that the tStatCatcher and tLogCatcher are actually run only at the end of a job -- which sort of makes sense given what they collect. Can anyone confirm? The problem this appears to create is, when using tMSSqlClose component to end the connection at post-job, it is getting fired before one or the other of the tStat or tLog are completed.
I would have expected that everything was completed before the tPostJob is fired.
If I remove the tMSSqlClose, it runs without errors -- but then the connection is not closed, right?
We've done something very similar, but rather than writing to a database, we write to a Kafka topic. We then have a consumer that reads from the Kafka topic and pushes the log messages to the ELK stack that comes with the TAC, but you could easily write a consumer that writes them to the database of your choice. Better still, once you have the log messages on the Kafka topic, you could write them to both an ELK setup and database at the same time.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Pick up some tips and tricks with Context Variables
Learn how media organizations have achieved success with Data Integration
Create systems and workflow to manage clean data ingestion and data transformation.