Talend ESB Services and Resources Monitoring using Nagios

Scenario

This article provides details about monitoring your system and developed user services using Nagios. Nagios installation can be skipped if a different corporate-wide monitoring tool is available. Nagios in this example is not limited to Talend use only, and can be used as a centralized monitoring tool.

 

Prerequisites

  1. Talend ESB installed and sample service—HelloWorldService—is deployed successfully
  2. Linux distribution—this example uses Ubuntu 16.0.4
  3. Active Internet Connection—to download dependencies and installation packages
  4. Network Communication among Talend ESB, the monitoring server (Nagios), the firewall, ports, and so on

 

Talend Runtime

Talend has integrated the Jolokia Agent into its standard ESB product. Jolokia is an agent-based approach for remote JMX access, where the communication between the client and the agent goes over HTTP (either GET or POST) and the request and response payload is represented in JSON. For more information about Jolokia, see Jolokia Overview.

 

  1. To verify that the package is installed and is in the Active state, run the following command:

    karaf@trun<>> list | grep -i jolokia

    image1.png

     

  2. Design and develop a simple HelloWorldService and make sure the Service is working by invoking its method. Service-managed beans will only be available after its first successful call.

 

Installing Nagios on Linux

In this example, Nagios will be installed on Ubuntu, a Linux distribution.

  1. To update the installation packages, run the following command:

    sudo apt-get update

    update.png

     

  2. Before installing Nagios, install third party libraries and tools.

    sudo apt-get install wget build-essential apache2 php apache2-mod-php7.0 php-gd libgd-dev sendmail unzip

    libraries.png

     

  3. Add the Nagios user and group, and provide access to Nagios data on the apache public folder for the Nagios web interface, by running the following commands:

    sudo useradd nagios
    sudo groupadd nagcmd
    sudo usermod -a -G nagcmd nagios
    sudo usermod -a -G nagios,nagcmd www-data

    adduser.png

     

  4. Install Nagios using the Ubuntu apt-get installation repository. Nagios3 is the latest version available on APT Package Repository. For other versions, paths and configuration may look different.

    sudo apt-get install nagios3

    installnagios.png

     

  5. During the installation process, you will be asked for a password. Note down the password, as it will be required to access the web interface.

    password.png

     

  6. Retype the password to confirm it. A password manager is recommended to keep track of passwords.

    confirm.png

     

 

Installing the Nagios Remote Plugin Executor

The Nagios NRPE (Nagios Remote Plugin Executor) Plugin is required for remote server communication. For more information about NRPE, which allows you to remotely execute Nagios plugins on other Linux/Unix machines, see the NRPE page on github.

 

  1. To install the NRPE server, run the following command:

    sudo apt-get install Nagios-nrpe-server

    installnagiosserver.png

     

  2. To install the NRPE plugin using the APT package manager, run the following command:

    sudo apt-get install nagios-nrpe-plugin

    installnagiosplugin.png

     

 

Accessing the Nagios web interface

After Nagios finishes installing, the web interface is available at a standard apache port. To access it, provide your login information:

Default Username:  nagiosadmin
Password:   ******  (the one you created during the installation process)

login.png

 

After a successful login, the Nagios admin page appears.

loggedin.png

 

Under Host, it will show some statistics of your local machine.

 image12.png

 

Jmx4Perl

Jmx4perl was the former name for Jolokia. For more information about it, see the Jmx4Perl page.

 

  1. Before Jmx4Perl installation, you must install the helping library libxml12-dev:

    sudo apt-get install libxml12-dev

    libxml12.png

     

  2. In this example, CPAN is used to install Jmx4Perl:

    perl -MCPAN -e shell
     cpan[1]> install JMX::Jmx4Perl

    jmx4perl.png

     

    It will ask a script whether it should be installed in addition to the core Perl modules during installation. Each of these scripts (jmx4perl, j4psh, jolokia, check_jmx4perl) will introduce a set of new dependencies on Perl modules, which in turn might depend on other Perl modules or system libraries. Perl modules are resolved and installed automatically.

  3. Once installation is finished and it displays success, confirm it by using the jmx4perl --help command:

    $ jmx4perl --help

    The usage and options for jmx4perl will appear.

 

Configuring the Service

The Talend ESB Package contains sample configuration files for Hyperic and Nagios. They are available in the add-on/adapters folder of ESB. The Nagios folder contains configuration template files and sample files for monitoring CXF, Camel, and Activemq using Nagios.

 

template/jmx_commands.cfg Command template file used to define the commands to monitor CXF, Camel, and Activemq

template/cxf.cfg

template/camel.cfg

template/activemq.cfg

Template files used to define checks for CXF, Camel, and Activemq metrics. The server URL should reflect the Talend Runtime Server.

sample/cxf_host.cfg

sample/camel_host.cfg

sample/activemq_host.cfg

Sample configuration files used to define host and service for monitoring CXF, Camel, and Activemq using Nagios. You can define your own xxx_host.cfg for monitoring specific metrics and specific resources such as CXF services and Camel routes.

 

Configuring the installation

Copy the configuration template files below into the folder /etc/nagios3/etc/objects/. The path may vary based on Linux and the version of Nagios.

  • template/jmx_commands.cfg
  • template/cxf.cfg
  • template/camel.cfg
  • template/activemq.cfg

image.png

 

  1. Modify cxf.cfg and camel.cfg to change the server address to the Talend Runtime URL:

    # This is template of jmx4perl plugin configuration for CXF enpoint monitoring
    # Define server connection parameters
    <Server tesb_cxf>
       Url = http://talend-esb-test:8040/jolokia
    </Server>

     

  2. Modify the cxf_host.cfg file to monitor your sample service HelloWorldService:

    # CXF service definition template - This is not a real host, just a template.
    
    define host{
    	name				cxf-service	           ; The name of this host template
    	use				generic-host	           ; This template inherits other values from the generic-host template
    	check_period			24x7		           ; By default, CXF service are checked round the clock
    	check_interval		        5		           ; Actively check the host every 5 minutes
    	retry_interval		        1		           ; Schedule host check retries at 1 minute intervals
    	max_check_attempts		10		           ; Check each Linux host 10 times (max)
            check_command           	check_jmx4perl_cxf!EndpointState!HelloWorldServicePort!{http://www.talend.org/service/}HelloWorldService  ; Default command to check CXF service
    	notification_period		workhours	           ; Admins hate to be woken up, so we only notify during the day							   
    	notification_interval		120	                   ; Resend notifications every 2 hours
    	notification_options		d,u,r		           ; Only send notifications for specific host states
    	contact_groups		        admins		           ; Notifications get sent to the admins by default
    	register			0		           ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL HOST, JUST A TEMPLATE!
    	}
    
    # Define a host for the jolokia machine
    
    define host{
            use                           cxf-service                  ; Name of host template to use					
    	host_name                     tesb_cxf
            alias                         tesb_cxf
            }
    
    
    
    ###############################################################################
    ###############################################################################
    #
    # SERVICE DEFINITIONS
    #
    ###############################################################################
    ###############################################################################
    define service {
        use                   generic-service
        service_description   HelloWorldService_NumInvocations
        display_name          HelloWorldService_NumInvocations
        check_interval        1
        host_name             tesb_cxf
        check_command         check_jmx4perl_cxf!NumInvocations!HelloWorldServicePort!{http://www.talend.org/service/}HelloWorldService
        }
    
    define service {
        use                   generic-service
        service_description   HelloWorldService_TotalHandlingTime
        display_name          HelloWorldService_TotalHandlingTime
        check_interval        1
        host_name             tesb_cxf
        check_command         check_jmx4perl_cxf!TotalHandlingTime!HelloWorldServicePort!{http://www.talend.org/service/}HelloWorldService
        }
    
    define service {
        use                   generic-service
        service_description   HelloWorldService_NumCheckedApplicationFaults
        display_name          HelloWorldService_NumCheckedApplicationFaults
        check_interval        1
        host_name             tesb_cxf
        check_command         check_jmx4perl_cxf!NumCheckedApplicationFaults!HelloWorldServicePort!{http://www.talend.org/service/}HelloWorldService
        }
    
    define service {
        use                   generic-service
        service_description   HelloWorldService_NumLogicalRuntimeFaults
        display_name          HelloWorldService_NumLogicalRuntimeFaults
        check_interval        1
        host_name             tesb_cxf
        check_command         check_jmx4perl_cxf!NumLogicalRuntimeFaults!HelloWorldServicePort!{http://www.talend.org/service/}HelloWorldService
        }
    
    define service {
        use                   generic-service
        service_description   HelloWorldService_NumRuntimeFaults
        display_name          HelloWorldService_NumRuntimeFaults
        check_interval        1
        host_name             tesb_cxf
        check_command         check_jmx4perl_cxf!NumRuntimeFaults!HelloWorldServicePort!{http://www.talend.org/service/}HelloWorldService
        }
    
    define service {
        use                   generic-service
        service_description   HelloWorldService_NumUnCheckedApplicationFaults
        display_name          HelloWorldService_NumUnCheckedApplicationFaults
        check_interval        1
        host_name             tesb_cxf
        check_command         check_jmx4perl_cxf!NumUnCheckedApplicationFaults!HelloWorldServicePort!{http://www.talend.org/service/}HelloWorldService
        }
    
    define service {
        use                   generic-service
        service_description   HelloWorldService_OP_ANY_FAULTS
        display_name          HelloWorldService_OP_ANY_FAULTS
        check_interval        1
        host_name             tesb_cxf
        check_command         check_jmx4perl_cxf!AnyFaults!HelloWorldServicePort!{http://www.talend.org/service/}HelloWorldService
        }
    
    ###############################################################################
    ###############################################################################

     

  3. Define macros in the /etc/nagios3/resource.cfg that will be used by jmx_commands.cfg. Paths may vary depending on Linux and the version of Nagios.

    # set the path which jmx4perl plugin installed
    $USER5$=/usr/local/src/jmx4perl/scripts
    # set the path to where to find configuration files
    $USER6$=/etc/nagioss/etc/objects

    image.png

     

  4. Add the following lines (for the command configuration files) to the /etcl/nagios3/etc/nagios.cfg file:

    cfg_file=/etc/nagios3/etc/objects/jmx_commands.cfg
    cfg_file=/etc/nagios3/etc/objects/camel_host.cfg
    cfg_file=/etc/nagios3/etc/objects/cxf_host.cfg
    cfg_file=/etc/nagios3/etc/objects/HelloWorldService.cfg

     

  5. After all configuration is completed, restart the Nagios service.

    root@talend-monitor-test:/etc/nagios3/etc/objects# sudo service nagios3  restart

 

Monitoring the Service

Refresh the Nagios page, and the status of Cxf and Camel should be available. Nagios provides different views.

image.png

 

Nagios provides detailed status where it shows different statistics of the host.

image.png

 

Detailed view of each host where it shows Status, if the host is Flapping, and so on.

image.png

 

All the services on a particular host. In this example, you deployed the HelloWorldService, and below you can see its status.

image.png

 

Talend provides support for Camel Route monitoring.

image.png

 

Talend provides monitoring with any vendor who complies with the JMX protocol for monitoring. Nagios is the de-facto standard in the open-source community, as it supports integration with other graphical tools for customized dashboards, for better visibility and monitoring.

Version history
Revision #:
21 of 21
Last update:
‎10-13-2017 02:04 PM
Updated by:
 
Labels (2)
Contributors