Configuring JobServer and ESB Runtime with Filebeat

Overview

This article introduces the best practices that Talend suggests you follow when working with Filebeat in JobServer and ESB Runtime.

 

This is a continuation of the Talend Community Knowledge Base article, Talend logging using Filebeat.

 

JobServer

The JobServer installation doesn’t include the Filebeat component, so you'll need to copy a Filebeat installation folder from another installation folder (for example the one installed with Log Server) and configure a dedicated service.

 

Configuring TAC portal

TAC uses two folders dedicated to the Job generation and execution. Configure these folders on TAC by following these steps.

  1. Log in to TAC.
  2. Change the location of the Job conductor logs on the TAC Configuration page, as follows:

    • Generated jobs folder: %JOBSERVER_installation%/Administrator/generatedJobs
    • Task logs folder: %JOBSERVER_installation%/Administrator/executionLogs

    06.png

    These folders should be created manually on the machine, and the TAC service user should have permission on it.

 

Configuring Studio Log4j

Before enabling Filebeat and redirecting the logs to Log Server, you need to review the Log4j configuration for Job execution. The Job execution log doesn’t include a timestamp for each line, and this may cause Filebeat not to recognize and map all log details.

 

For this reason, you need to change the Log4j configuration to generate a timestamp into the log.

  1. Open the project in Talend Studio, then select File > Edit Project properties > Log4j.

    07.png

     

  2. Scroll to <!-- CONSOLE appender --> and change the <param name="ConversionPattern" value="[%-5p]: %c - %m%n" /> value to:

    <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss} %-5p: %c - %m%n" />
  3. Click Apply, then click OK.

  4. Redeploy the Job.

 

Configuring Filebeat

Configure Filebeat, on the installation location, to permit Log Server to redirect logs to a specific logstash instance, by performing the following steps:

  1. Open the %filebeat_installation%/filebeat.yml file and change output.logstash: > hosts: with the location of your Log Server, as shown below:

    output.logstash:
     # The Logstash hosts
     hosts: ["logserver_ip:5044"]
     #hosts: ["localhost:5044"]
  2. Change the filebeat.prospectors: section as shown below:

    filebeat.prospectors:
    - type: log
      enabled: true
      paths:
        - %JOBSERVER_installation_path%/Administrator/executionLogs/*/*.log
      fields:
        app_id: JOBSERVER
      fields_under_root: true
      multiline.pattern: '([12]\d{3}-(0[1-9]|1[0-2])-(0[1-9]|[12]\d|3[01]))'
      multiline.negate: true
      multiline.match: after
      exclude_lines: ['###']

    If you want to send the entire log file to one entry, use the following command:

    filebeat.prospectors:
    - type: log
      enabled: true
      paths:
        - %JOBSERVER_installation_path%/Administrator/executionLogs/*/*.log
      fields:
        app_id: JOBSERVER
      fields_under_root: true
      multiline.pattern: '([### Job STARTED'
      multiline.negate: true
      multiline.match: after
  3. Restart the Filebeat service.

 

Kibana portal results

The following Kibana portal screenshots show logs related to Job execution:

(filter - application.keyword: “JOB_SERVER”)

08.png

09.png

 

ESB Runtime

On Talend 7.1, the event logging solution has been deprecated and replaced by Event Monitoring.

 

Event Monitoring is an implementation of Apache Karaf Decanter, monitoring solution running in Apache Karaf.

 

The ESB Runtime installation doesn’t include the Filebeat component, so you'll need to copy a Filebeat installation folder from another installation folder (for example, the one installed with Log Server or TAC) and configure a dedicated service.

 

Configuring containers

ESB Containers need to be configured to redirect logs to logstash. To configure and redirect all logs to the centralized Log Server, perform the following steps.

  1. Open the %ESB_runtime_installation%/esb/%container_name%/etc/filebeat_em.yml file.

    • Change paths with the right %container_name_folder%.
    • Change output.logstash: > hosts with the location of your Log Server.
    filebeat.prospectors:
    
    - type: log
      enabled: true
      paths:
       - ${LOG_PATH:/opt/Talend-
    7.1.1/esb/%container_name_folder%/data/eventmonitoring/em*.log*}
     fields:
       app_id: ${APP_NAME:ESB-app}
     fields_under_root: true
     json.keys_under_root: true
     json.message_key: log
    
    logging.metrics.enabled: false
    #----------------------------- Logstash output --------------------------------
    output.logstash:
     # The Logstash hosts
     hosts: ["${LOGSTASH_HOSTS:logserver_ip}:${LOGSTASH_PORT:5044}"]
     #hosts: ["${LOGSTASH_HOSTS:localhost}:${LOGSTASH_PORT:5044}"]
  2. ESB container uses different log collectors, which permits sending different log categories to Log Server. Usually, not all collectors are used at the same time.

    1. Open the following %ESB_runtime_installation%/esb/%container_name%/etc/ org.apache.karaf.decanter.collector.* files:
      • org.apache.karaf.decanter.collector.log.cfg
      • org.apache.karaf.decanter.collector.jms.cfg
      • org.apache.karaf.decanter.collector.eventadmin-sam.cfg
      • org.apache.karaf.decanter.collector.eventadmin-karaf.cfg
      • org.apache.karaf.decanter.collector.eventadmin-framework.cfg
    2. In each file:
      • Change “fields.add.agentId” = agent_%container_name%_%log_type% (all collectors)
      • Change “fields.rename.renderedMessage logMessage” to “fields.rename.renderedMessage message” (only collector-log)
      • Change “fields.rename.level severity” to “fields.rename.level priority” (only for collector-log)
      • Change “fields.remove.message = true” to “#fields.remove.message = true” (all collectors)
      • Change “fields.rename.event.topics = logMessage” to “fields.rename.event.topics = message” (except collector-log)
      • Add “fields.rename.loggerName logger_name” (only collector-log)
  3. Connect in ssh to the Karaf container, by using the following command:
    #ssh karaf@runtime_ip -p port
  4. Run the following command:
    #tesb:start-em-default

    This command starts to collect all log category to event monitoring log file (under data/eventmonitoring/em.log). This file is the one referred by Filebeat.

  5. Start Filebeat by using the following command (or create a service):
    filebeat -e -c {KARAF_HOME}/etc/filebeat_em.yml

     

Kibana portal results

The following Kibana portal screenshots show logs related to ESB Runtime:

 

Container logs:

filter - agentid: “agent_app_log”

10.png

11.png

 

Karaf login logs:

 filter - agentid: “agent_app_karaf”

12.png

13.png

 

ESB service deploy logs:

filter - agentid: “agent_app_framework”

14.png

15.png

 

For more information on configuring Talend components with Filebeat, see the following Talend Community Knowledge Base (KB) articles:

 

Version history
Revision #:
14 of 14
Last update:
‎07-05-2019 04:02 AM
Updated by: