cLog, logstash, kibana: how to have custom logs from routes to logstash

Five Stars

cLog, logstash, kibana: how to have custom logs from routes to logstash

Hello everybody,
 

I have many complex routes with error handling. I would like to log the errors in logstash in order to monitor the runtime execution in kibana. Now I am logging to files (or sending email for critical errors) but I am going crazy to keep everything under control (see attached picture).

I searched both the community posts and the official Talend Documentation but I didn't get a detailed procedure on how to reach this goal. 

 

How can I send custom logs to logstash from my route? Should I use cLog or a cMessagingEndpoint to eventlogging: or anything else?
Can I use topics and categories to separate logs originated from different error handling? Should I have to set special headers in order to do that?

What are the best practices in term of logging and routes?

Thank you in advance for any help you'll provide.

Manuel

Ten Stars

Re: cLog, logstash, kibana: how to have custom logs from routes to logstash

I more or less exactly did this... 

Send it to ElasticSearch directly.

I've added my json to build the index from within Kibana, adjust and add.

 

Jobs

flowMeter -> (a childjob)toJson -> (a childjob Elastic) tRestClient

logCatcher -> (a childjob)toJson -> (a childjob Elastic) tRestClient.

logCatcher -> (a childjob)toJson -> (childjob Elastic) tRestClient.