Redirecting Talend Cloud logs to a JMS

Overview

This article shows you how to redirect logs to another target but still get the Job execution status in Talend Cloud. In this article, the target is ActiveMQ, a JMS message service.

 

This article covers, testing your configuration by creating a Talend Job to generate the logs. Running the Job and redirecting the logs to Talend Cloud Management Console, then changing the configuration to redirect the logs to the JMS.

Picture1.png

The demo Job is available in the LogGenerator.zip file attached to this article.

 

Prerequisites

 

Creating a Job

  1. Create a Job using a tWarn component to display the log4j message, and a tRowGenerator component with a tLogrow component to display data directly in the console.

    Screen Shot 2019-02-25 at 12.14.03.png

     

  2. Run the Job to display the logs.

    Screen Shot 2019-02-25 at 12.14.56.png

     

  3. Publish the Job to Talend Cloud, by right-clicking the Job, then select Publish to Cloud.

    Screen Shot 2019-02-25 at 12.17.20.png

     

  4. If this isn't the first version, select the version you want from the Publish With Version field, then select Personal from the Workspace drop-down menu. Click Finish.

    Screen Shot 2019-02-25 at 12.22.20.png

     

  5. When the Job has finished uploading, click Open Job Task.

    Screen Shot 2019-02-25 at 12.18.45.png

     

Assigning a Runtime

  1. Notice the Task doesn't have an assigned Runtime.

    Screen Shot 2019-02-25 at 12.23.02.png

     

  2. On the Engine tab, you should see your engine, in this case, demo-mga.

    Screen Shot 2019-02-26 at 11.57.43.png

     

  3. Select the Task and click the pencil icon to edit the configuration.

    Screen Shot 2019-02-26 at 11.58.01.png

  4. In step 2, Go Live, select the Runtime and the Run type.

    Screen Shot 2019-02-26 at 11.59.54.png

     

  5. Click Go Live. You will be sent back to the Task View. Notice that the Job Status is Running.

    Screen Shot 2019-02-26 at 12.00.08.png

  6. Once the Task is complete, select the execution and click View Logs.

    Screen Shot 2019-02-26 at 12.42.44.png

  7. Notice that the logs are redirected to Talend Cloud. In the next steps, you will change the configuration to modify this behavior.

    Screen Shot 2019-02-26 at 12.43.43.png

     

Redirecting the logs

  1. Stop the Remote Engine service.

  2. Edit the remote-engine-installation/etc/org.talend.eventlogging.sender.jms.cfg file in your Remote Engine installation folder.

    The configuration should look similar to this:

    # JMS connection url
    sender.destination.jms.url = failover:(https://log.eu.cloud.talend.com:443)
    
    # Name of JMS queue
    sender.destination.jms.queue=event.logging.server
    
    # JMS Username
    sender.destination.jms.username = ca677a20-086b-472e-8811-6x48fbcd13d5_kIt4srr50GAyZ9pwr
    
    # JMS Password
    sender.destination.jms.password = WxLgy6pBHadz4xq0HvASe74
    
    sender.destination.jms.pool.maxConnections = 8
  3. Update the sender.destination.jms.url parameter to your JMS broker, in this case, localhost.

    # JMS connection url
    sender.destination.jms.url = failover:(tcp://localhost:61616)
    
    # Name of JMS queue
    sender.destination.jms.queue=event.logging.server
    
    # JMS Username
    sender.destination.jms.username = ca677a20-086b-472e-8s811-6b48dbcd1q3qad5_kIt4rr50GAyZ9pwr
    
    # JMS Password
    sender.destination.jms.password = WxLgy6pBH4xq0HvASe74
    
    sender.destination.jms.pool.maxConnections = 8
  4. Restart the Remote Engine service.

  5. Log in to the ActiveMQ admin console; you should see a new queue, event.logging.server.

    Screen Shot 2019-02-26 at 12.46.53.png

     

  6. Go back to Talend Cloud Management Console and rerun the Task.

    Screen Shot 2019-02-26 at 12.48.03.png

     

  7. Once it is successfully executed, check the logs to validate no logs are displayed.

    Screen Shot 2019-02-26 at 12.48.15.png

  8. In the ActiveMQ console, you should see several messages enqueued.

    Screen Shot 2019-02-26 at 12.48.27.png

  9. Open one to validate that it is available.

    Screen Shot 2019-02-26 at 12.48.42.png

     

Conclusion

In this example, you learned how to edit the Remote Engine configuration to redirect Task logs in to a JMS broker. Once logs are available in the JMS, you can have a pipeline to read and process them to another system.

Version history
Revision #:
13 of 13
Last update:
‎04-23-2019 06:20 AM
Updated by: