Continuous Integration and Continuous Deployment of data services, routes, and microservices

Overview

Talend 7.1.1 supports Continuous Integration and Continuous Deployment (CI/CD) of data services and routes, and you can use CI to build Microservices or Talend Runtime (OSGi) artifacts. If you implemented CI/CD for standard DI/BD Jobs, you can use the same steps for ESB Data Services and Routes.

 

This article shows you how to automatically build, test, package, and deploy data services, routes, and microservices using Talend CommandLine, CI Builder Maven plugin, Jenkins, and Nexus.

 

Sources for the project are attached to this article.

 

Prerequisites

Install and configure the servers:

  1. Create two VMs and install Ubuntu Linux 18.04.

  2. Name the servers ciserver and adminserver respectively.

    In the /etc/hosts file, map the domain name of the server to a given IP address.

    sudo vi /etc/hosts 
    192.168.238.4 ciserver
    192.168.238.5 adminserver
  3. On the ciserver, install the following:

    • OpenJDK Zulu-8
    • Jenkins
    • Talend CommandLine
    • Latest patches for Talend CommandLine and Studio
  4. On the adminserver, install the following:

    • OpenJDK Zulu-8
    • Nexus
    • Talend Administration Center (Optional)
    • Talend Studio
    • Install Git and create a repository, for example, CI_ESB_DEMOS.

Note:

 

Objectives and logical architecture

  • Configure Jenkins for CI/CD

  • Create a Job in Jenkins that builds OSGi bundles, routes, and microservices, then publish it to Nexus

  • Create a pipeline Job that builds OSGi bundles, routes, and microservices, then publish it to Nexus

    KB1-Architectuere2.jpg

     

Installing Talend CI Builder

Note: Talend CI Builder is now embedded in Studio as a plug-in and doesn't require a manual installation.

 

Log in to the CI server and start CommandLine, doing this initializes your commandline local Maven repository and installs the CI Builder plug-in automatically.

CI_Builder_install.jpg

 

Configuring Jenkins for CI/CD

Jenkins needs additional tools and configurations to automate the build of Jobs and routes, to execute unit tests, and to package and deploy the artifacts to a repository. To accomplish this, you will do the following:

Manage Plugins

  1. In the Manage Jenkins view, select Manage Plugins.

  2. Select the Available tab.

  3. Select and install the required plug-ins, for example:

    • Maven Integration
    • Config File Provider
    • Green Balls
    • Bitbucket (Note: If you choose a different Git provider then the Bitbucket plug-in is optional.)
    • Multiple SCMs
    • Pipeline
    • Pipeline Maven Integration
    • Blue Ocean

    Jenkins_Plugins_inst.jpg

     

Global Tool Configuration

  1. In the Manage Jenkins view, select Global Tool Configuration to configure the tools required for your Maven build.

  2. Click Add JDK and provide the location of your JAVA_HOME.

    Java_home_global_settings.jpg

     

  3. Click Add Git and add JGit as a client.

    Config_JGIT.jpg

     

  4. Click Add Maven, provide the path to the existing MAVEN_HOME or install it automatically.

    Mvn.jpg

     

Configure System

  1. In the Manage Jenkins view, select Configure System and configure your system variables.

  2. Define a global property PATH+extra for locating Java executable.

    1. On the Global properties section, click Environmental variables.

    2. Click Add and fill in the Name and Value properties.

    ConfigureJava.jpg

     

Configure Credentials

Use the Jenkins Credentials Provider to save your Git repository credentials.

  1. Navigate to Jenkins > Credentials > System > Global credentials.

    crdentails_navigation.jpg

     

  2. Select Add Credentials, fill in your Git Username, Password, ID, and a Description. Click OK.

    credentials_save.jpg

     

Managed files

The Config File Provider plug-in provides the ability to add configuration files loaded through the Jenkins UI that will be copied to the Job's workspace.

 

Here you need to create and configure a Maven settings file that contains the path to your local Maven repository and remote repository definitions with credentials.

 

Note: Repository settings are configured at two levels, that is, user local settings or global Maven settings.

 

  1. In the Manage Jenkins view, select Managed files.

    Managed_files_big.jpg

     

  2. Click Add a new Config, select Maven settings.xml, and provide a valid ID for the settings file. Click Submit.

    AddNewConfig.jpg

     

  3. Configure the local and remote repositories as shown below, then click Submit.

    Note: The ID and Name of the settings file is referenced in your Jenkins Job during the Build phase. Use the settings file, ci_maven_settings.xml, attached to this article.

    settings_xml_selected.jpg

     

Preparing a Job, route, and microservice in Studio

  1. Create a new TAC project.

    Ci_esb_demos_proj.jpg

     

  2. Grant write access to authorized users.

    Project_Auth.jpg

     

  3. Log in to Talend Studio.

  4. Import the ESB Demo Jobs by selecting Import on the Studio toolbar.

  5. Open the DemoREST Job service. Click the Job tab and observe that Deployment > Build Type is set to OSGi Bundle.

    osgi_Bundle.jpg

     

  6. Open the TestDemoRest Job. Click the Test Cases tab, then right-click Customer2 to run the test, and verify the test results.

    RunTest.jpg

     

  7. Open the DemoResourceRoute route. Click the Job tab and observe that the Deployment > Build Type is set to Route.

    DemoResourceRoute.jpg

     

  8. Create a microservice by right-clicking DemoRESTRoute, then click Duplicate, rename the route to MS_DemoRESTRoute.

    You can run a route as microservice by changing the Deployment > Built Type.

  9. Click the Job tab and change the Deployment > Build Type to RouteMicroservice.

    RouteMicroservice.jpg

     

  10. Test the Job, route, and microservice in Studio.

 

Creating a Jenkins Job and publishing artifacts to Nexus

In the previous section, you prepared a data service, route, and microservice in Studio.

Before you check your changes into SCM, run the Jobs and routes in Studio, and verify that all the tests are working, then push the changes to the Git repository.

In this section, you create a Jenkins Job and verify the build and test execution results on a CI server.

 

  1. Click New Item.

  2. In the Enter an item name text box, name the Job, ESB_DeploymentType. Select Maven project, then click OK.

    Jenkins_Job.jpg

     

  3. On the General tab, enter the Job Description.

    Description.jpg

     

  4. Click Source Code Management, choose Git or Multiple SCMs, provide the Git repository URL and branch specifier.

    gIT_REPO.jpg

     

  5. For Git credentials, select the credentials with id bitbucket, by using the down arrow to the right of Credentials.

    proj_scm_crede.jpg

     

  6. Click Build Triggers and configure when Jenkins should automatically run the Job.

    Note: An alternate option is to build every two hours based on the size and complexity of the project.

    BuildTriggers.jpg

     

  7. Click Build Environment, then select the Delete workspace before build starts check box.

    BuildEnvironment.jpg

    BuildEnvironment.jpg

     

  8. Click Build:

    1. For root pom, fill in the relative URL of your pom.xml project:
      CI_ESB_DEMOS/poms/pom.xml

      Note: The project name is in uppercase.

    2. For Goals and options, fill in the Maven goal and additional CLI options:

      deploy -X -pl jobs/process/demorest_0.1,jobs/routes/demoresourceroute_0.1,jobs/routes/ms_demorestroute_0.1 -fae -amd

      Note: modify the options as per your convenience, for example:

      1. Use the -ff option to stop the build at the first failure.

      2. Remove the -pl option for a full build.

      Maven command line options:

      -pl: Comma-delimited list of projects to build, or remove the -pl option for complete build

      -fae: Only fail the build at the end, allow all non-impacted builds to continue

      -ff: Stop at first failure in reactorized builds

      -amd: Using -pl if project list is specified, then also build projects required by the list

      -X: Debug log output

    3. For Maven_OPTS, fill in the location of your local CommandLine, and remote Nexus repository URLs.

      -Dgeneration.type=local
      -Dproduct.path=/opt/talend/Talend-7.1.1/cmdLine/cmdline/studio
      -Xmx3076m
      -DaltDeploymentRepository=releases-ci::default::http://adminserver:8081/repository/releases-ci/ -Dproject.distributionManagement.repository.url=http://adminserver:8081/repository/releases-ci/ -Dproject.distributionManagement.repository.id=releases-ci

      Build.jpg

       

  9. Click Build Settings, optionally for unstable builds, and configure E-mail Notification.

    Email_Notification.jpg

     

  10. Save the changes, by clicking Apply, then click Save, to save the Job.

  11. Run the Job and verify the results.

    1. Review the test results:

      TestResults.jpg

       

    2. Verify that the artifacts are published to Nexus, by logging in to the Nexus console at, http://adminserver:8081/

      Nexus_publish.jpg

 

Creating a build pipeline and publishing artifacts to Nexus

The Pipeline plug-in allows you to express your build process as a script and implement complex workflows. The Pipeline Maven plug-in provides Maven integration by using the withMaven step, which configures a Maven environment to use within a pipeline Job.

The Blue Ocean plug-in provides sophisticated visualizations of the build and pipeline status with dashboards.

 

Create a pipeline project in Jenkins

  1. Click New Item, enter the name of the pipeline, for example, Pipeline_ESB_Deployments, then click Pipeline and click OK.

    new_Pipeline.jpg

     

  2. In the Description field, provide the purpose of this pipeline.

    Description.jpg

     

  3. Click Build Triggers, select Poll SCM, for example, every night.

    BuildTriggers.jpg

     

  4. Click Pipeline, then select pipeline script from the Definition drop-down menu. Copy the contents of the Pipeline_Nexus_deploy_groovy.txt file, attached to this article, into the script area.

    pipeline_script.jpg

  5. Click Build now, and run the pipeline.

  6. Click Full Stage View to see the pipeline stages.

    Stage_view.jpg

     

  7. Select the Open Blue Ocean pipeline and visualize the dashboards.

    OpenBlueOcean.jpg

     

  8. Select Pipeline_ESB_Deployments, then click Run.

    Blue_ocean.jpg

     

  9. Observe the pipeline status is blue while the Job is running.

    Blue_ocean_result.jpg

     

  10. After successful execution, the pipeline status turns to green.

    pipeline_success.jpg

Version history
Revision #:
49 of 49
Last update:
‎02-25-2019 02:20 AM
Updated by:
 
Comments
Five Stars

I am trying to build job using SVN Repository. 

In Step 8 Click Build:

I have written 

following 

Root POM : tdo_repo/poms/jobs/process/jb_random_0.1/pom.xml

GOAL Setting : deploy -pl tdo_repo/poms/jobs/process/jb_random_0.1 -fae -am

 

But when I am building my job its give me following error in Jenkins 

-------------------------------------------------------------------------------------------

[INFO] Scanning for projects...
[ERROR] [ERROR] Could not find the selected project in the reactor: tdo_repo/poms/jobs/process/jb_random_0.1 @
[ERROR] Could not find the selected project in the reactor: tdo_repo/poms/jobs/process/jb_random_0.1 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MavenExecutionException
channel stopped
Finished: FAILURE

Employee Ram
Employee

@birari  Can you try changing the location of ROOT POM  

Root POM : tdo_repo/poms/pom.xml

Five Stars

I tried to changing the mentioned pom file and I got below error, my file is placed in correct location.

Parsing POMs
using settings config with name maven-file
Replacing all maven server entries not found in credentials list is true
ERROR: Failed to parse POMs
org.apache.maven.project.ProjectBuildingException: Some problems were encountered while processing the POMs:
[ERROR] Child module /var/lib/jenkins/workspace/jb_deploy/tdo_repo/poms/jobs/process/jb_random_tac_0.1 of /var/lib/jenkins/workspace/jb_deploy/tdo_repo/poms/pom.xml does not exist @

at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:364)
at hudson.maven.MavenEmbedder.buildProjects(MavenEmbedder.java:361)
at hudson.maven.MavenEmbedder.readProjects(MavenEmbedder.java:331)
at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1328)
at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1125)
at hudson.FilePath.act(FilePath.java:1078)
at hudson.FilePath.act(FilePath.java:1061)
at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.parsePoms(MavenModuleSetBuild.java:986)
at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:691)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
at hudson.model.Run.execute(Run.java:1810)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:543)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Finished: FAILURE

Five Stars

Do you have pipeline script for SVN.

Community Manager

Hi @birari,

Please ask your question on the appropriate Community discussion forum, where you will get help from Talend staff and other users. If you need further assistance, please contact Talend Support.

Thank you,

Donna