Talend 7.1.1 supports Continuous Integration and Continuous Deployment (CI/CD) of data services and routes, and you can use CI to build Microservices or Talend Runtime (OSGi) artifacts. If you implemented CI/CD for standard DI/BD Jobs, you can use the same steps for ESB Data Services and Routes.
This article shows you how to automatically build, test, package, and deploy data services, routes, and microservices using Talend CommandLine, CI Builder Maven plugin, Jenkins, and Nexus.
Sources for the project are attached to this article.
Install and configure the servers:
Create two VMs and install Ubuntu Linux 18.04.
Name the servers ciserver and adminserver respectively.
In the /etc/hosts file, map the domain name of the server to a given IP address.
sudo vi /etc/hosts 192.168.238.4 ciserver 192.168.238.5 adminserver
On the ciserver, install the following:
On the adminserver, install the following:
Steps in this article can be executed on any Linux OS.
For information on installing Talend Software modules on the adminserver, see Talend Data Services Platform Installation Guide for Linux on the Talend Help Center.
Configure Jenkins for CI/CD
Create a Job in Jenkins that builds OSGi bundles, routes, and microservices, then publish it to Nexus
Create a pipeline Job that builds OSGi bundles, routes, and microservices, then publish it to Nexus
Note: Talend CI Builder is now embedded in Studio as a plug-in and doesn't require a manual installation.
Log in to the CI server and start CommandLine, doing this initializes your commandline local Maven repository and installs the CI Builder plug-in automatically.
Jenkins needs additional tools and configurations to automate the build of Jobs and routes, to execute unit tests, and to package and deploy the artifacts to a repository. To accomplish this, you will do the following:
Use Manage Plugins to install additional plug-ins
Use Global Tool Configuration to install or configure required tools, for example, Maven and Git
Configure System parameters, for example, PATH
Configure Credentials, for example, Git
Use Managed files to configure Maven settings to deploy Jobs
In the Manage Jenkins view, select Manage Plugins.
Select the Available tab.
Select and install the required plug-ins, for example:
In the Manage Jenkins view, select Global Tool Configuration to configure the tools required for your Maven build.
Click Add JDK and provide the location of your JAVA_HOME.
Click Add Git and add JGit as a client.
Click Add Maven, provide the path to the existing MAVEN_HOME or install it automatically.
In the Manage Jenkins view, select Configure System and configure your system variables.
Define a global property PATH+extra for locating Java executable.
On the Global properties section, click Environmental variables.
Use the Jenkins Credentials Provider to save your Git repository credentials.
Navigate to Jenkins > Credentials > System > Global credentials.
Select Add Credentials, fill in your Git Username, Password, ID, and a Description. Click OK.
The Config File Provider plug-in provides the ability to add configuration files loaded through the Jenkins UI that will be copied to the Job's workspace.
Here you need to create and configure a Maven settings file that contains the path to your local Maven repository and remote repository definitions with credentials.
Note: Repository settings are configured at two levels, that is, user local settings or global Maven settings.
In the Manage Jenkins view, select Managed files.
Click Add a new Config, select Maven settings.xml, and provide a valid ID for the settings file. Click Submit.
Configure the local and remote repositories as shown below, then click Submit.
Note: The ID and Name of the settings file is referenced in your Jenkins Job during the Build phase. Use the settings file, ci_maven_settings.xml, attached to this article.
Create a new TAC project.
Grant write access to authorized users.
Log in to Talend Studio.
Import the ESB Demo Jobs by selecting Import on the Studio toolbar.
Open the DemoREST Job service. Click the Job tab and observe that Deployment > Build Type is set to OSGi Bundle.
Open the TestDemoRest Job. Click the Test Cases tab, then right-click Customer2 to run the test, and verify the test results.
Open the DemoResourceRoute route. Click the Job tab and observe that the Deployment > Build Type is set to Route.
Create a microservice by right-clicking DemoRESTRoute, then click Duplicate, rename the route to MS_DemoRESTRoute.
You can run a route as microservice by changing the Deployment > Built Type.
Click the Job tab and change the Deployment > Build Type to RouteMicroservice.
Test the Job, route, and microservice in Studio.
In the previous section, you prepared a data service, route, and microservice in Studio.
Before you check your changes into SCM, run the Jobs and routes in Studio, and verify that all the tests are working, then push the changes to the Git repository.
In this section, you create a Jenkins Job and verify the build and test execution results on a CI server.
Click New Item.
In the Enter an item name text box, name the Job, ESB_DeploymentType. Select Maven project, then click OK.
On the General tab, enter the Job Description.
Click Source Code Management, choose Git or Multiple SCMs, provide the Git repository URL and branch specifier.
For Git credentials, select the credentials with id bitbucket, by using the down arrow to the right of Credentials.
Click Build Triggers and configure when Jenkins should automatically run the Job.
Note: An alternate option is to build every two hours based on the size and complexity of the project.
Click Build Environment, then select the Delete workspace before build starts check box.
Note: The project name is in uppercase.
For Goals and options, fill in the Maven goal and additional CLI options:
deploy -X -pl jobs/process/demorest_0.1,jobs/routes/demoresourceroute_0.1,jobs/routes/ms_demorestroute_0.1 -fae -amd
Note: modify the options as per your convenience, for example:
Use the -ff option to stop the build at the first failure.
Remove the -pl option for a full build.
Maven command line options:
-pl: Comma-delimited list of projects to build, or remove the -pl option for complete build
-fae: Only fail the build at the end, allow all non-impacted builds to continue
-ff: Stop at first failure in reactorized builds
-amd: Using -pl if project list is specified, then also build projects required by the list
-X: Debug log output
For Maven_OPTS, fill in the location of your local CommandLine, and remote Nexus repository URLs.
-DaltDeploymentRepository=releases-ci::default::http://adminserver:8081/repository/releases-ci/ -Dproject.distributionManagement.repository.url=http://adminserver:8081/repository/releases-ci/ -Dproject.distributionManagement.repository.id=releases-ci
Click Build Settings, optionally for unstable builds, and configure E-mail Notification.
Save the changes, by clicking Apply, then click Save, to save the Job.
Run the Job and verify the results.
Review the test results:
Verify that the artifacts are published to Nexus, by logging in to the Nexus console at, http://adminserver:8081/
The Pipeline plug-in allows you to express your build process as a script and implement complex workflows. The Pipeline Maven plug-in provides Maven integration by using the withMaven step, which configures a Maven environment to use within a pipeline Job.
The Blue Ocean plug-in provides sophisticated visualizations of the build and pipeline status with dashboards.
Click New Item, enter the name of the pipeline, for example, Pipeline_ESB_Deployments, then click Pipeline and click OK.
In the Description field, provide the purpose of this pipeline.
Click Build Triggers, select Poll SCM, for example, every night.
Click Pipeline, then select pipeline script from the Definition drop-down menu. Copy the contents of the Pipeline_Nexus_deploy_groovy.txt file, attached to this article, into the script area.
Click Build now, and run the pipeline.
Click Full Stage View to see the pipeline stages.
Select the Open Blue Ocean pipeline and visualize the dashboards.
Select Pipeline_ESB_Deployments, then click Run.
Observe the pipeline status is blue while the Job is running.
After successful execution, the pipeline status turns to green.