How Kibana is working for log display in Talend

Five Stars

How Kibana is working for log display in Talend

Hi,

I want to use Kibana in TAC for logging purpose.

 

Understanding: It says Kibana is just an displayer of data and it gets data from eleastic search. Is it correct ?
I've ESB jobs are running and want to log ESB events in Kibana.

 

Doubt:
1) From where ESB logs should be picked up by Elasticsearch and display in Kibana ? Is it somewhere in Elasticsearch data folder ?


Any help ?

Six Stars

Re: How Kibana is working for log display in Talend

DISCLAIMER: This is all stuff I figured out myself. I don't know what the Talend way of doing this is. I suspect it probably involves the AMC. The stuff below is all for version 6.4.1. This version of the TAC uses version 2.4.1 of the ELK stack. I've tried to link only to 2.4-specific documentation below.

 

The TAC ships with an integrated Elasticsearck-Logstash-Kibana (ELK) stack. Lots of resources online if you Google "ELK stack".

 

Elasticsearch stores the log messages and indexes them for easy and fast retrieval. It listens on localhost at port 9200 on the TAC host. It's started at system bootup by systemd(1). The service is called "talend-elastic-6.4.1". Do a "systemctl status talend-elastic-6.4.1" to see if it's running.

 

Logstash writes the messages into Elasticsearch. It provides a variety of interfaces. They call them "plugins". The TAC uses the Log4j plugin. The configuration for this plugin is at installdir/Talend-6.4.1/logserv/logstash-talend.conf. This service is also managed by systemd(1). The service is called "talend-elastic-6.4.1". Do a "systemctl status talend-elastic-6.4.1" to see if it's running.

 

Kibana is a Web interface for querying and graphing data from Elasticsearch. It also allows you to build dashboards from your queries and graphs. The TAC installation comes with a canned default dashboard that shows you logging from the TAC itself. The TAC documentation for using and external ELK stack has a pretty good overview of the files used to build the default dashboards. Kibana startup is supposed to also be handled by systemd(1), but it's broken on our setup, for some reason. I start it manually using installdir/Talend-6.4.1/logserv/start_kibana_daemon.sh.

 

You have two choices for writing your own log messages to the TAC ELK stack. You can write documents directly to Elasticsearch. You should be comfortable being out in the deep woods without a compass or a map if you choose to go this way.

 

You can modify installdir/Talend-6.4.1/logserv/logstash-talend.conf and add your own Log4j plugin, or some other Logstash plugin you want to use.