Continuous Integration with Talend ESB Route Builder

Overview

This article describes how to perform all the steps required for a continuous integration test of your Apache Camel-based routes designed within Talend ESB Studio using the Continuous Integration Testing Framework:

  1. Automated compilation of source artifacts
  2. Automated Unit-Test on each artifact
  3. Automated deployment into Test-Environment
  4. Run automated Integration-Tests

 

Environment

This procedure applies to the Talend solutions that include ESB, starting from version 5.1.0.

 

Procedure

Automating the compilation of source artifacts

You cannot directly build your Camel Routes based on artifacts stored in your SVN repository. Mapping from Studio artifacts to Java Code is only possible in the Studio. Therefore you need to generate your Java Code in the Talend Studio first, and then deploy those artifacts to Archiva. However, since version 5.2 you can generate your Java Code based on Studio artifacts using the Talend Commandline.

 

Automating the Unit-Test on each artifact

Usually an SVN Update triggers a recompilation of these sources in your Continuous Integration Framework, and Maven will automatically run all JUnit Tests attached to your code. However, this does not work with ESB Routes designed in Talend Studio right now.

 

One reason is that SVN commits happen quite often, each time someone makes changes to a route and tries to run it. Thus SVN Update events are not a proper trigger for running a Unit-Test, since an SVN commit does not mean the development process is completed. Therefore your Continuous Integration Framework should not listen for SVN updates, but rather for Archiva updates, because if a developer is done with designing a route, he/she will publish his/her route to an Archiva snapshot release repository.

 

Since only the artifacts published to Archiva are "mavenized" but not the .Java project inside Studio, you cannot simply append your Unit-Test to the .Java project. Code-changes within the .Java project could also get overridden easily by the code generator. Therefore it is recommended to create a separate Maven-Test-Project. This Maven-Test-Project needs to refer to the Maven Artifact generated by Studio. Look at the attached Maven sample (pom.xml).

 

By default, Maven dependencies are only downloaded once from remote repositories, and then stored/cached in a local file system. Normally you can force Maven to download changed (snapshot) artifacts with the same artifact-version each time from a remote repository, but Maven will not detect a newer version of the published route in the Archiva repository and thus will use an old version stored locally. So changes in your route will not be reflected in your next test-run. To avoid testing the old versions deployed first from Studio, you need to remove all matching directories or files from your local Maven repository. You can also use a Maven plugin to delete these folders each time you type mvn clean. Refer to the attached sample pom.xml for further details.

 

Since the Route deployment and Unit-Tests are separate Maven projects, you need to link these projects in your integration testing framework, so that your test will run each time an update is applied to your Camel route project.

 

To learn more about Unit-Testing, look at the attached HelloWorldTest.java.zip.

 

Handling 3rd Party dependencies

Unfortunately, the Maven project file generated by Studio does not contain any dependencies (even though there are a couple). So you need to discover all dependencies used within the route yourself and add these to your Maven pom.xml Test project.

 

Automate deployment into the Test-Environment

To make a real integration test, it is not sufficient to simply test each route individually. The goal of an integration test is to verify that routes are working in conjunction with each other in a production-like environment with real backend-systems.

Deploying routes to Talend ESB Runtime can be done through JMX or using Talend Commandline. See the attached deploy-routes-and-tests.zip to see how this could be done. Remember to use a new Talend ESB runtime container for deployment each time, to ensure reproducible results.

 

Run automated Integration Tests

After deployment of your routes into Talend ESB Runtime, you need to run further Unit-Test focusing on integration tests.

 

There are several options to write Integration Tests. For example, you can deploy them into your runtime container, and use JMX notifications or writing/checking log files to evaluate your test results. Camel provides a rich Testing Framework that could also be used in a normal JUnit Test Case to send messages to JMS queues or create sample files in a local test system.

 

Summary

Best practices can be summarized as follows:

  • Write your Test Cases in a separate Maven project linked to your RouteBuilder project.
  • Use Archiva updates as triggers for your tests instead of SVN commits.
  • Connect Unit-Test Project with Route Builder project.
  • Clear your local Maven repository (from routing projects only).
  • Use Camel Mock components in your Unit-Tests.
  • Test in production-like environments.
  • Write Unit-Test for real-world integration testing.

 

Running the Hello World Sample

This article provides a Hello world sample. Follow these steps to run it:

  1. Download HelloWorld.zip (attached) and import the Hello World project into Studio.
  2. Deploy Studio Project to Archiva.
  3. Download HelloWorldTest.zip (attached) and import the HelloWorldTest Project (you can use a normal Eclipse or the Java Perspective in Studio).
  4. Adjust pom.xml to match your deployed route artifact name.
  5. Run: mvn clean compile in your test project.
  6. Run: mvn test or mvn install to run your unit tests.
Version history
Revision #:
3 of 3
Last update:
‎06-15-2017 07:11 PM
Updated by:
 
Labels (1)
Tags (1)