Ten Stars

ESB runtime (community) - Apache Decanter

Have a question if anybody have experience:

 

currently I use with Talend runtime - Karaf web-console + hawtio plugin 

 

Routes and Jobs send alerts when some errors happens, but because not all errors could be handled inside route it self - for example route not start by connection timeout or other reasons, I use live pings algorithm as addition layer,

this combination cover most of problem, but I looking for additional ways.

 

after research and test I found - Apache Decanter could be solution, but:

- default repository not include sla-email alerter

- current version of decanter appenders not compatible with external ELK 5.* (it not a problem if not previous point)

- new versions of decanter (1.3.0) not compatible (or I do not all properly) with Talend version of Karaf - 4.0.7

 

Would be interesting to share with community:

- is anybody have success experience with installing decanter with sla-alert module?

- who and what use for monitoring and alarming/alerting? (alarms and alerts more interesting than monitoring)

Best regards, Vlad

-----------
14 REPLIES
Moderator

Re: ESB runtime (community) - Apache Decanter

Hello Vald,

We have redirected your issue to talend ESB experts and then come back to you as soon as we can.

Thanks for your time.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Employee

Re: ESB runtime (community) - Apache Decanter

Hi,

 

some details about Decanter and your question:

- the Decanter features repo includes decanter-sla-email feature. You have to install this one.

- if you use the decanter-appender-elasticsearch-rest, it can work and send the data to an external Elasticsearch instance, including version 5.x.

- Decanter is compatible with any Karaf version, including the one in Talend. What's the issue ?

 

Regards

JB

Ten Stars

Re: ESB runtime (community) - Apache Decanter

Hi Jb!

thank You for response. I not great expert in this question, but how it look from my side:

 

Talend 6.3.1 ESB (Community) 

feature:repo-add decanter
karaf@trun()> feature:list | grep decanter
decanter-common                         | 3.0.0.SNAPSHOT   |          | Uninstalled | karaf-decanter-3.0.0-SNAPSHOT | Karaf Decanter API
decanter-simple-scheduler               | 3.0.0.SNAPSHOT   |          | Uninstalled | karaf-decanter-3.0.0-SNAPSHOT | Karaf Decanter Simple Scheduler
decanter-collector-log                  | 3.0.0.SNAPSHOT   |          | Uninstalled | karaf-decanter-3.0.0-SNAPSHOT | Karaf Decanter Log Messages Collector
decanter-collector-jmx                  | 3.0.0.SNAPSHOT   |          | Uninstalled | karaf-decanter-3.0.0-SNAPSHOT | Karaf Decanter JMX Collector
decanter-appender-log                   | 3.0.0.SNAPSHOT   |          | Uninstalled | karaf-decanter-3.0.0-SNAPSHOT | Karaf Decanter Log Appender
decanter-appender-elasticsearch         | 3.0.0.SNAPSHOT   |          | Uninstalled | karaf-decanter-3.0.0-SNAPSHOT | Karaf Decanter Elasticsearch Appender
elasticsearch                           | 1.3.4            |          | Uninstalled | karaf-decanter-3.0.0-SNAPSHOT | Embedded Elasticsearch node
kibana                                  | 3.1.1            |          | Uninstalled | karaf-decanter-3.0.0-SNAPSHOT | Embedded Kibana dashboard
karaf@trun()> 

as You can see - no sla modules

if we change command:

feature:repo-add decanter 1.3.0

it will be much more features ... but

after this - any attempt to instal kibana or some other modules from repository (not all but few important) - freeze and not response after restart, still not work. Tested on 3 different machines, Mac and Win.
internal Kibana not start,

If install only collector and appenders, REST appender - do not connect to external Elasticsearch (error message - client must be at least version 2.0)

 

Finally I downgrade external ELK (it is working solution, but not ideal because other services use 5.*)

  • Elasticsearch to 1.7.6
  • logstash to 2.4.1
  • Kibana to 4.1.11

after this install logs collector and appender and email-sla from 1.3.0 repository, both appenders start work, as well as log4j from Studio start log to logstash


now play around email-sla and ELK watcher, it new for me, so this part in progress.

 

main idea of all this games, have alarms for some errors from routes which we do not have now.

 

In DI part I already use - email, slack, snmp, OpsGenie alarms (different levels), in routes for internal errors - also all work fine

 

But when route not start because external server not answer - I not found method to catch this error from route. It die and not send any notice, only error in log

it was asked before - https://www.talendforge.org/forum/viewtopic.php?id=56642


with monitoring part - all fine, but after couple years with good alarms from DI Jobs, was surprised when can not realise same in routes and try to resolve

Best regards, Vlad

 

-----------
Eight Stars

Re: ESB runtime (community) - Apache Decanter

Hi,

 

For now, I would like to use decanter only to send alerts when an ERROR line is appended in tesb.log.

 

I ran:
feature:repo-add decanter
feature:install decanter-sla-log
feature:install decanter-sla-email
feature:install decanter-collector-log

 

I amended the file org.apache.karaf.decanter.sla.checker.cfg:
# For instance, if you want to create a critical (error) alert if the loggerLevel is ERROR, you can do:
loggerLevel.error=match:ERROR

 

These features are installed:
decanter-common | 1.4.0 | | Started | karaf-decanter-1.4.0 | Karaf Decanter API
decanter-collector-log | 1.4.0 | x | Started | karaf-decanter-1.4.0 | Karaf Decanter Log Messages Collector
decanter-sla-core | 1.4.0 | | Started | karaf-decanter-1.4.0 | Karaf Decanter SLA support core
decanter-sla | 1.4.0 | | Started | karaf-decanter-1.4.0 | Karaf Decanter SLA support
decanter-sla-log | 1.4.0 | x | Started | karaf-decanter-1.4.0 | Karaf Decanter SLA log alerter
decanter-sla-email-core | 1.4.0 | | Started | karaf-decanter-1.4.0 | Karaf Decanter SLA email alerter core
decanter-sla-email | 1.4.0 | x | Started | karaf-decanter-1.4.0 | Karaf Decanter SLA email alerter

 

Then I deployed a route that create an error log message:
2017-11-09 11:24:29,273 | ERROR | :/Temp/EXCEPTION | cLog_2 | 158 - org.apache.camel.camel-core - 2.16.3 | Test Decanter !!!!!!!!!!!!!!!!!

 

I was expecting to see a DECANTER SLA ALERT message in log. But nothing happens...

 

What did I misconfigure ?

 

Thanks for your help.

 

Regards.

 

Eric

Employee

Re: ESB runtime (community) - Apache Decanter

I bet that the property name (in the map) doesn't match error.

How do you create the log message ?

Did you check in the backend the property name ?

Regards
JB
Eight Stars

Re: ESB runtime (community) - Apache Decanter

Hi JB,

Thanks for your quick answer.

 

Are you talking about properties in the route ?

 

If yes, the properties are:
{CamelBatchSize=2,
C:\Temp\EXCEPTION\CASE_ROUTE_ID.txt-CamelFileLockFileAcquired=true,
C:\Temp\EXCEPTION\CASE_ROUTE_ID.txt-CamelFileLockFileName=C:\Temp\EXCEPTION\CASE_ROUTE_ID.txt.camelLock,
CamelMessageHistory=[DefaultMessageHistory[routeId=TEST_EXCEPTION_cFile_1,
node=TEST_EXCEPTION_cLog_2],
DefaultMessageHistory[routeId=TEST_EXCEPTION_cFile_1,
node=TEST_EXCEPTION_cProcessor_1]],
CamelBatchComplete=false,
CamelBatchIndex=0,
CamelFileExchangeFile=GenericFile[C:\Temp\EXCEPTION\CASE_ROUTE_ID.txt],
CamelCreatedTimestamp=Thu Nov 09 12:14:39 CET 2017}

 

The log message is created with cLog component. Should I log it with log endpoint (In a cmessageendpoint component) ? Or should I trigger an exception as a test ?

 

NB: I didn't install kibana neither elasticsearch. None of them is required for my need. Am I wrong ?

 

regards.

 

Eric

Employee

Re: ESB runtime (community) - Apache Decanter

OK, I think I know the problem. I guess you are using the default
etc/org.ops4j.pax.loggging.cfg provided in the Talend ESB distribution.

You have to enable any Pax Logging appender (including the Decanter one).

To do so, in etc/org.ops4j.pax.logging.cfg, you have to change the rootLogger
configuration like this:

log4j.rootLogger=INFO, out, osgi:*

(currently, you probably have log4j.rootLogger = INFO, out, osgi:VmLogAppender).

Then, the Decanter pax appender will catch the log messages.

Regards
JB
Eight Stars

Re: ESB runtime (community) - Apache Decanter

Hi,

 

Indeed the configuration was:

log4j.rootLogger=INFO, out, camel-mdc, osgi:VmLogAppender, osgi:eventloglistener

 

Now it is:

log4j.rootLogger=INFO, out, camel-mdc, osgi:*

 

But it doesn't help...

 

On your blog, I can read "Decanter Alerters are a special kind of appenders, consuming events from the OSGi EventAdmin decanter/alert/* topics.". How can I check I get events at OSGi EventAdmin level ?

 

Eric

Employee

Re: ESB runtime (community) - Apache Decanter

I just try and it works for me.

Can you share the cLog, especially, how do you define the log level ?

Did you restart the pax logging bundles (or the whole Talend runtime) after
changing etc/org.ops4j.pax.logging.cfg ?

There's no easy way to see the internal events (it's possible in debug mode, but
not easy for end users).
If you use a backend like Elasticsearch, you can use Kibana to see the events
sent by Decanter.

Eight Stars

Re: ESB runtime (community) - Apache Decanter

Hi,

 

I uploaded:

-My route

-The kar file

-The three cfg files

 

Thanks again for your help.

 

Eric

Employee

Re: ESB runtime (community) - Apache Decanter

Do you mind to send it to me on jbonofre@talend.com ?

Thanks.
Regards
JB
Eight Stars

Re: ESB runtime (community) - Apache Decanter

Sure, I'll do it right now.

 

Eric

Two Stars

Re: ESB runtime (community) - Apache Decanter

Hello, Did you get a solution for your problem ?

I am also trying to use Decanter as a monitoring solution at our end, but so far I am just checking what could be done and how we could implement it. 

Eight Stars

Re: ESB runtime (community) - Apache Decanter

Hi,

 

Yes, alert works with this condition:

level.error=match:ERROR

 

Eric