Improve Camel flow monitoring

Problem Description

  • There is no request/exchange specific logging information out-of-the-box from Camel without increasing the log level to DEBUG

  • It must be possible to monitor requests without having to add a lot of manual log statements

 

Solution

Overview of Best Practice

Camel provides two ways to gather information about every message (Camel Exchange) being processed:

  • Camel Trace provides detailed information on messages being processed within the route. Every process step triggers an event, and you can configure what content should be added to the event (content, headers, properties, custom data).
  • Camel Events are triggered when an exchange is created, completed, failed, and so on. You can configure what content should be added to the event. For a list of Camel events, see the org.apache.camel.management.event page.

 

Apache Decanter is a flexible monitoring solution for Talend ESB (Apache Karaf) that allows you to gather monitoring information by using their collector architecture, like JMX, which is out-of-scope for this best practice.

 

Decanter provides the collector Camel Tracer & Notifier, which is notified whenever a Camel Event or Trace occurs. Decanter creates an OSGi Event, which can then be fed to any kind of appender to store the monitoring data centrally in Elasticsearch or MongoDB, or you can create a log event with the log appender. This solution will not work with Spring Boot, because Apache Decanter uses the OSGi EventAdmin service to bridge between the Camel events and the Decanter appenders.

 

Installation

Log in to the Talend ESB shell and install Decanter:

feature:repo-add decanter 2.0.0
feature:install decanter-appender-kafka decanter-collector-camel decanter-collector-eventadmin

 

Configuration

The configuration must be done with the Spring configuration file. You must configure several beans provided by Camel and Decanter to build the bridge between the two, and enable Camel Trace and Event features.

 

<!--
  Example property definition
-->

<osgix:cm-properties id="cmProps" persistent-id="com.example.myapplication>">
  <prop key="application">myapplication</prop>
  <prop key="camel.context.trace">false</prop>
  <prop key="camel.context.useMDCLogging">false</prop>
  <prop key="camel.trace.enabled">false</prop>
  <prop key="camel.trace.includeBody">false</prop>
  <prop key="camel.trace.includeProperties">true</prop>
  <prop key="camel.trace.includeHeaders">true</prop>
  <prop key="camel.event.includeBody">false</prop>
  <prop key="camel.event.includeProperties">true</prop>
  <prop key="camel.event.includeHeaders">true</prop>
  <prop key="camel.event.verbose">false</prop>    
</osgix:cm-properties>

<osgi:reference id="eventAdmin" interface="org.osgi.service.event.EventAdmin" />

<!--   Decanter bean definitions   Documentation: https://karaf.apache.org/manual/decanter/latest-2/#_camel_tracer_notifier --> <bean id="decanterEventNotifier" class="org.apache.karaf.decanter.collector.camel.DecanterEventNotifier">   <property name="eventAdmin" ref="eventAdmin"/>   <!-- Optional to add custom properties to the event -->   <!--<property name="extender" ref="customExtender" />-->   <property name="includeBody" value="${camel.event.includeBody}"/>   <property name="includeProperties" value="${camel.event.includeProperties}"/>   <property name="includeHeaders" value="${camel.event.includeHeaders}"/>   <property name="ignoreCamelContextEvents" value="false" />   <property name="ignoreRouteEvents" value="false" />   <property name="ignoreServiceEvents" value="false" />   <property name="ignoreExchangeEvents" value="false" />    <property name="ignoreExchangeCreatedEvent" value="#{!${camel.event.verbose}}" />   <property name="ignoreExchangeCompletedEvent" value="false" />   <property name="ignoreExchangeFailedEvents" value="false" />   <property name="ignoreExchangeRedeliveryEvents" value="false" />   <property name="ignoreExchangeSendingEvents" value="#{!${camel.event.verbose}}" />   <property name="ignoreExchangeSentEvents" value="#{!${camel.event.verbose}}" /> </bean> <bean id="traceHandler" class="org.apache.karaf.decanter.collector.camel.DecanterTraceEventHandler">   <property name="eventAdmin" ref="eventAdmin"/>   <!-- Optional to add custom properties to the event -->   <!--<property name="extender" ref="customExtender" />-->   <property name="includeBody" value="${camel.trace.includeBody}"/>   <property name="includeProperties" value="${camel.trace.includeProperties}"/>   <property name="includeHeaders" value="${camel.trace.includeHeaders}"/> </bean> <!--   Camel Trace documentation   Documentation: http://camel.apache.org/tracer.html --> <bean id="tracer" class="org.apache.camel.processor.interceptor.Tracer">   <property name="traceHandler" ref="traceHandler"/>   <property name="enabled" value="${camel.trace.enabled}"/>   <property name="traceOutExchanges" value="true"/>   <property name="logStackTrace" value="true"/>   <property name="logLevel" value="OFF"/> </bean> <camelContext xmlns="http://camel.apache.org/schema/spring"   trace="{{camel.trace.enabled}}"   useMDCLogging="{{camel.context.useMDCLogging}} "> </camelContext>

 

Add custom context data to every monitoring event

The two Decanter classes, DecanterEventNotifier and DecanterTraceEventHandler, provide a property to set a custom extender class, which implements the interface org.apache.karaf.decanter.collector.camel.DecanterCamelEventExtender. You can add properties to the passed map, and have full access to the Camel Exchange object.

 

For more information, see the Apache Karaf Decanter documentation, Camel Tracer & Notifier page.

 

Example: monitoring in Elasticsearch

Camel Event

{
  "hostName": "MYHOST",
  "customContext": {
    "customerId": "61601601",
    "orderId": "300018631"
  },
  "type": "camelEvent",
  "topic_type": "camel",
  "shortExchangeId": "ID-MYHOST-35588-1520542853557-2-2",
  "routeId": "myroute",
  "inHeaders": {},
  "cap": "tesb",
  "fromEndpointUri": "cxf://bean:proxyOrderEndpoint",
  "@version": "1",
  "topic_name": "event",
  "inBodyType": "org.apache.camel.converter.stream.InputStreamCache",
  "outHeaders": {},
  "@indexer": "myhost",
  "camelContextName": "mycamel-context",
  "exchangePattern": "InOut",
  "eventType": "org.apache.camel.management.event.ExchangeCompletedEvent",
  "tags": [
    "decanter"
  ],
  "exchangeId": "ID-MYHOST-35588-1520542853557-2-2",
  "@timestamp": "2018-03-09T13:57:28.228Z",
  "application": "orderApp",
  "outBodyType": "org.apache.camel.converter.stream.InputStreamCache",
  "topic": "decanter/collect/camel/event",
  "hostAddress": "192.157.11.11",
  "category": "trace",
  "properties": {
    "CamelCharsetName": "UTF-8",
    "mtom-enabled": "false",
    "CamelSkipGzipEncoding": true,
    "CamelToEndpoint": "cxf://bean:OrderEndpoint"
  }
}

 

Camel Trace

{
  "hostName": "MYHOST",
  "type": "camelTracer",
  "topic_type": "camel",
  "shortExchangeId": "ID-MYHOST-46763-1518450121529-21-872",
  "routeId": "ws-route",
  "inHeaders": {
    "CamelHttpResponseCode": 202,
    "Accept": "*/*",
    "Server": "Jetty(9.2.21.v20170120)",
    "CamelHttpCharacterEncoding": "UTF-8",
    "operationName": "do",
    "Date": "Tue, 20 Feb 2018 06:57:41 GMT",
    "operationNamespace": "http://webservices.test/OrderService",
    "CamelFileNameProduced": "work/order/ID-MYHOST-46763-1518450121529-21-871",
    "messageType": "Health",
    "breadcrumbId": "ID-MYHOST-46763-1518450121529-21-871",
    "CamelHttpMethod": "POST",
    "CamelAcceptContentType": "*/*",
    "CamelHttpResponseText": "Accepted",
    "Content-Length": "0",
    "CamelHttpUri": "/services/orderService",
    "flowid": "d08f54bd-44a6-48ae-94d6-099763159402",
    "CamelHttpPath": ""
  },
  "cap": "tesb",
  "fromEndpointUri": "cxf://bean:proxyOrderEndpoint",
  "@version": "1",
  "topic_name": "tracer",
  "inBodyType": "String",
  "outHeaders": {
    "CamelHttpResponseCode": 202,
    "Accept": "*/*",
    "Server": "Jetty(9.2.21.v20170120)",
    "CamelHttpCharacterEncoding": "UTF-8",
    "operationName": "do",
    "Date": "Tue, 20 Feb 2018 06:57:41 GMT",
    "operationNamespace": "http://webservices.test/OrderService",
    "CamelFileNameProduced": "work/ID-MYHOST-46763-1518450121529-21-871",
    "messageType": "ORDER",
    "breadcrumbId": "ID-MYHOST-46763-1518450121529-21-871",
    "CamelHttpMethod": "POST",
    "CamelAcceptContentType": "*/*",
    "CamelHttpResponseText": "Accepted",
    "Content-Length": "0",
    "CamelHttpUri": "/services/orderService",
    "flowid": "d08f54bd-44a6-48ae-94d6-099763159402",
    "CamelHttpPath": ""
  },
  "@indexer": "myhost",
  "camelContextName": "mycamel-context",
  "exchangePattern": "InOut",
  "outBody": "<removed>",
  "toNode": "setBody[Simple: OK]",
  "tags": [
    "decanter"
  ],
  "exchangeId": "ID-MYHOST-46763-1518450121529-21-872",
  "@timestamp": "2018-02-20T06:57:41.131Z",
  "application": "orderApp",
  "outBodyType": "String",
  "topic": "decanter/collect/camel/tracer",
  "hostAddress": "192.157.11.11",
  "category": "trace",
  "nodeId": "to616",
  "previousNode": "",
  "inBody": "<removed>",
  "properties": {
    "CamelCharsetName": "UTF-8",
    "mtom-enabled": "false",
    "CamelSkipGzipEncoding": true,
    "CamelToEndpoint": "http://localhost:10001/order?bridgeEndpoint=true"
  }
}
Version history
Revision #:
6 of 6
Last update:
‎09-29-2018 12:17 AM
Updated by: