Continuous Integration with Talend CI Builder 6.3

Overview

This article explains how to set up Continuous Integration, Continuous Testing, and Continuous Deployment using Talend CI-Builder. It covers the required preparation in your environment before installation, the configuration and file system folder structures suggested by Talend, and the steps to automate the build process using Jenkins and Maven.

 

Note: You can use any CI tool that supports Maven. This article shows how to use Jenkins.

 

Preparing your environment

 

Prerequisites

The following Talend environment must be installed. For details, see these pages on the Talend Help Center.

  • Talend Administration Center

    Set up with a dedicated command line and port (minimum version 6.1.1). For more information, see Installing and configuring Talend Administration Center.

  • Talend Studio

    See Installing your Talend Studio.

  • Nexus

    The Nexus repository is configured, the server is accessible from the CI server, and the interaction between Nexus, TAC, and Studio is fully working. Any HTTP or HTTPS proxies that are present in the environment must be accounted for, and you should ensure that Talend environment communications are not going over a proxy unless it is necessary. See Installing and configuring the Nexus artifact repository under Installing and Configuring Talend server modules.

  • Subversion or Gitlab

    SVN or GIT must be integrated with TAC and Studio, and must be tested and fully functional.

  • Projects

    Projects, reference projects, and project authorizations in TAC are set up, with users having appropriate access/authorization.

  • Talend Jobs

    Jobs containing test cases for Continuous Testing. To familiarize yourself with the test cases, review these videos:

     

     

     

     

    ci.png

 

Minimum System Requirements

Component Processor RAM Disk Space Operating System
Talend Administration Center + Talend Activity Monitoring Console Web application 2 CPU 8 GB 1 GB Linux, Windows, MacOS
CommandLine 2 CPU 5 GB 3 GB  
JobServer 1 CPU 1 GB 20 MB  
Studio 2 CPU 4 GB 3 GB  
Talend Runtime 1 CPU 4 GB 400 MB  

 

Minimum Temporary Desk Space Requirements

Product Operating System Disk Space
Installer Windows or UNIX 1 GB
  Windows 2 GB
  UNIX 605 MB

 

Patch Installation

 

Option 1

If the TAC server has internet access, and the software update configuration is correct in TAC, you can automatically get the patch from Talend on the software update page. Once you choose to download the patch, it should be made available for all Studios and Command Line applications.

 

configure_update.png

 

For more information, see Configuring the Software Update repository in Talend Administration Center.

 

Option 2

If the TAC server has no internet access, follow the instructions in Uploading a patch in the local Nexus to manually download the patch and upload it to your local Nexus.

 

File System Folder Structure

Having a dedicated CI server is recommended. To run CI in the server, it will need dedicated storage space, so create the folder structure as shown below:

ci_folder.png

  • commandline_workspace: allows Jenkins, CommandLine, and CI Builder to utilize and share information. If you have CommandLine currently installed, you can use the existing CommandLine folder and copy it over instead of creating a new commandline_workspace.

  • JENKINS_HOME: stores the Jenkins configuration (system and projects). Jenkins_Home is also the name of the environment variable.

  • maven_repo: this folder is optional. Create it only if you need Maven to use the CommandLine Maven artifact repository.

  • target: stores a copy of the Java code, compiled artifacts, and test results.

 

Users

To prevent file locking and permissions issues, all components should be run using the same user, for example talenduser on UNIX or Local System Account on Windows. If you choose to change the default users, be very precise with file system permissions for the folders, including their children.

 

commandline_logon.png

 

Installation and Configuration

This process covers setup on Windows Server 2008, but most of the steps are identical in a UNIX-based machine. This assumes that the server is properly configured with a Java 8 JDK, the environment connectivity is working correctly, and your JAVA_HOME environment variable is set.

 

CommandLine

  1. Using the Talend installer, install CommandLine and its Server Service to wherever you will later install Jenkins and Maven. Alternatively, you can use an existing CommandLine install, or turn on Studio's CommandLine, instead of installing a new copy.

    Note: The installer may attempt to start the Windows service for CommandLine on completion.

  2. Start CommandLine and install the patch as described under Patch Installation above.
  3. Stop the CommandLine service.

 

Windows Service

  1. Browse to your CommandLineHome/TalendServices/conf directory:

    cmdline_conf.png

     

  2. Edit the wrapper.conf file and find the following section:

    #******************************************************************** 
    
    # genConfig: further Properties generated by genConfig 
    
    #********************************************************************
  3. Alter the parameter wrapper.app.parameter.7 to point to your commandline_workspace folder:

    genConfig.png

 

UNIX Service

When CommandLine is used with CI, it will not connect to TAC and does not find the location of the Nexus libraries repository (which by default is talend-custom-libs). Connecting to TAC is required to compile and build the Talend jobs, so to integrate Nexus, you must set the location of the Nexus libraries repository for CommandLine and add the other parameters listed below to the Talend_Root/CommandLineHome/studio/configuration/config.ini file.

 

  1.  Navigate to and open the config.ini file:

    config_ini.png

  2. Add the following parameters to the file:

    nexus.url=http://localhost:8081/nexus
    nexus.user=admin
    nexus.password=Talend123
    nexus.lib.repo=talend-custom-libs

     

    Note: If your user and password are the defaults shown above you don't need to include them in the file, though if you changed the defaults, specify the appropriate values.

    config_ini2.png

     

  3. Start the CommandLine service.

    Important: Do not use this CommandLine for anything other than Continuous Integration. It is recommended that the configuration folder of CommandLine remains in its default location.

 

Continuous Integration CommandLine Server

This server is dedicated for CI use only, and the CI CommandLine cannot be used for normal CommandLine operations such as Audit, TAC publisher, TAC Job Conductor generation, Studio remote generation, and scripting.

Important: Verify that Studio is not being used during installation.

 

The CI Server will host the following components:

  • CommandLine
  • Jenkins: Any CI tool that can use Maven should be compatible
  • Maven
  • CI Builder plugin

Optional: This server can be a logical place to install SVN, GIT, and/or Nexus.

Example diagram from v6 Reference architecture 1.4

ci_server_diagram.png

 

Maven – Windows Installation

 

  1. Download Maven from Downloading Apache Maven site (binary zip archive).

  2. Extract the downloaded zip file.

    maven_zip.png

     

    Option 1 - Set the location of the Maven .m2 repository

    The default location of Maven is ~/.m2 (user home, .m2 directory). With this setup, Maven will use CommandLine’s .m2 repository, as it is the simplest way of ensuring that Maven has the correct Talend libraries.

    1. Browse to MAVEN_ROOT_INSTALL_DIR/conf/settings.xml.

    2. Add the following property:

      <localRepository>E:/Talend/6.3.1/cmdline/studio/configuration/.m2/repository</localRepository>

      Where the path specified is the path to the CommandLine .m2 repository:

      cmdline_m2dir.png

       

    3. Navigate to the <servers> section of the same file and add an entry for the Nexus credentials:

      <server>
         <id>talendNexus</id>
         <username>admin</username>
         <password>Talend123</password>
      </server>

      server.png

       

    4. You can obfuscate the password if required. Review the instructions on the Maven site: Password Encryption.

     

    Option 2 - Move the path of the local repository

    To move the path of the local repository, you need to set the path in settings.xml.

    1. Set up a <server> set of credentials for Nexus as described in Option 1. It is possible to have multiple profiles here if you need to use different users for accessing the libraries repository and the deployment repository:

      <server>
         <id>talendNexusDeploy</id>
         <username>deploy</username>
         <password>deployPWD</password>
      </server>
      <server>
         <id>talendNexusLibs</id>
         <username>talend-custom-libs-admin</username>
         <password>talend-custom-libs-admin</password>
      </server>
    2. Locate the <profiles> section and add your Talend libraries repository as a new profile:

      <profile>
         <id>talendNexusLibs</id>
         <activation>
          <property>
           <name>talendNexusLibs</name>
           <value/>
          </property>
         </activation>
         <repositories>
          <repository>
           <id>nexus-repository</id>
          <url>http://win-1k2cbe6vvuj:8081/nexus/content/repositories/talend-custom-libs</url>
           <releases>
            <enabled>true</enabled>
           </releases>
           <snapshots>
            <enabled>true</enabled>
           </snapshots>
          </repository>
         </repositories>
      </profile>

      The <id> tag should have the same <id> as the <server> section, with the correct credentials of your repository.

    3. Add an <activeprofile> to the <activeprofiles> section:

      <activeProfiles>
       <activeProfile>talendNexusLibs</activeProfile>
      </activeProfiles>

      The file should look like this:

      active_profile.png

    Now Maven will be able to access the Talend Nexus for Talend libraries.

 

Continuous Integration Builder

  1. Download Talend CI Builder as described in your Talend license email.

     

  2. Unzip the downloaded file into your root CI folder. This creates ci.builder-version.jar and .pom.

    ci_folder.png

     

  3. Open a command prompt and browse to the directory where you installed Maven /bin.

    maven_bin.png

     

  4. Run the following command:
    mvn install:install-file -Dfile=CI-Builder_install_dir\ci.builder-6.3.1.jar -DpomFile=CI-Builder_install_dir\ci.builder-6.3.1.pom

    This will install CI Builder into Maven, and the supplied pom file will list the dependencies of CI Builder. It will then download these dependencies as part of the install process.

    Depending on your setup, the machine will need internet access to download these libraries, or you may have configured a repository for Maven (using Option 2) that contains these libraries.

     

  5. Create a new file in your CI Root folder called pom.xml, and copy the following contents into it.

    <?xml version="1.0" encoding="UTF-8"?>
    <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> 
        <modelVersion>4.0.0</modelVersion> 
        <groupId>org.talend</groupId> 
        <artifactId>buildsources</artifactId> 
        <version>0.1</version> 
        <packaging>pom</packaging> 
        <build> 
          <plugins> 
          <plugin> 
            <groupId>org.talend</groupId> 
            <artifactId>ci.builder</artifactId> 
            <version>6.3.1</version> 
            <executions> 
              <execution> 
                <phase>validate</phase> 
                  <goals> 
                    <goal>generate</goal> 
                  </goals> 
              </execution> 
            </executions> 
          </plugin> 
          </plugins> 
        </build> 
    </project>

     

    A Project Object Model or POM is the fundamental unit of work in Maven. It is an XML file that contains information about the project and configuration details used by Maven to build the project. It contains default values for most projects. Some of the configuration that can be specified in the POM are the project dependencies, the plugins or goals that can be executed, the build profiles, and so on. Other information such as the project version, description, developers, mailing lists and such can also be specified. This pom file defines the goals that the first stage of the CI process will attempt to achieve when executed from Jenkins.

 

Nexus

To segregate the CI Jobs, create a repository called releases_ci. The Nexus admin user was created to access this repository for deployment actions.

The recommended best practice is to set up suitable users and groups.

 

As an option for the artifacts to be generated by CI Builder and pushed to a Nexus repository, it is recommended to set up a unique repository for this purpose as shown below:

 

releases_ci.png

 

Jenkins - Windows Installation

  1. Download the Jenkins Windows installer from the Jenkins site: https://jenkins.io.
  2. Follow the installation wizard prompts:

    1. Verify the Jenkins port does not conflict with any Talend ports.
    2. Example is installed to E:\Jenkins.
    3. Another option is to deploy it on Tomcat as a war file in the webapps directory.
  3. Stop the Windows service if it started automatically.
  4. By default, the Jenkins directory is located in ~/.jenkins under the user that is running the Jenkins home directory, and where all the configuration files are stored. If running it as a Windows service, it is located under the Windows system folders in Users/user/.jenkins.

    1. To run the service as a different user, that same user would also need to run CommandLine.

    2. You can set a different path for JENKINS_HOME by editing the file JENKINS_INSTALL_ROOT/Jenkins.xml.

      Note: If this was deployed as a war file, you will not have a Jenkins.xml file.

      jenkins_home.png

       

  5. Start the Jenkins service and check that it can connect to the web UI (for example, http://localhost:8380).

    jenkins_localhost.png

    For more information about administering Jenkins, see the Administering Jenkins site.

 

Jenkins Configuration

Before continuing, you should have CommandLine, Jenkins and the CI Builder plugin installed into Maven to finish configuring Jenkins.

Navigate to Jenkins System settings

  1. Click Manage Jenkins > Manage Plugins.

    manage_jenkins.png

     

  2. Install the Maven Integration Plugin, and either the Subversion Plugin or the Gitlab Plugin, and verify the plugins are enabled.

    maven_plugin.png

     

  3. Click Manage Jenkins > Configure System.

    manage_jenkins2.png

     

  4. The home directory should reflect how you configured it previously (while installing Jenkins).

    jenkins_home2.png

     

  5. In the Global Tool Configuration, under JDK installations, add the path to the Java 8 JDK installation on your CI Server.

    jdk.png

     

  6. Under Maven installations, add the path to the Maven installation on your CI server.

     maven_install.png

     

  7. Save and close System Config.

 

Process

The Process of Continuous Integration, Continuous Testing, and Continuous Deployment begins once a job is committed to the central repository.

process.png

 

The build process uses Jenkins and Maven to take the completed Job and generate Java source code. It also compiles the Job into a Java executable and then publishes it to Nexus. Once the Job is published to Nexus, it can be deployed into any environment in a self-contained executable file. The following sections explain the steps to automate this process using Jenkins Jobs. There are three Jenkins Jobs, as shown below, to automate the build process:

jobs.png

 

Build

Building is the process of generating Java source code and Java executable binaries.

 

  1. Create a new Job by clicking Jenkins > New Item and adding a new Maven project named BuildSources.

    buildsources.png

     

  2. Configure the Source Code Management tab.

    • If using Git, provide the Git repository and other details. Configure the repository URL to point to a specific Tag, Branch, and Trunk.

      git.png

    • If using SVN, configure the repository URL to point to a specific Tag, Branch, and Trunk on SVN.

      source_code_mgmt.png

      Set the local module directory to the same as the project name (multiple branches of the same projects might require unique names).

      This will allow Jenkins to perform an SVN checkout to commandline_workspace that replicates the file system structure that would be created if Studio or CommandLine performed a check out of the project.

      Note: You will be prompted to enter credentials to communicate with SVN.

      Examples of the folder structure comparing Studio to CI:

      comparison1.png

      comparison2.png 

       

  3. On the Build Triggers tab, specify the trigger action according to the project's needs. In this example, the build will trigger as soon as a change is pushed to Gitlab.

    trigger.png

     

  4. On the Build tab, set the build options:

    build_2.png

    1. Root POM: the pom you created earlier (see the CI Builder section).
    2. Goals and options: in this scenario, you will use the options of Maven and CI Plugin for generate. It is possible to add a -X flag here to get debug information.
    3. MAVEN_OPTS: these parameters are passed to CI Builder to command it to call CommandLine.
      • Workspace: the same workspace configured in the CommandLine section.
      • User: not a TAC user; the CI builder does not communicate with TAC. This is a ‘substitute’ user that allows CommandLine to log onto a local project.
      • Target directory: the target directory, where the results of this stage will be copied.
      • In addition to these parameters, it is possible to filter what CommandLine builds. Behind the scenes, CI builder is calling the CommandLine command buildProjectSources, which can take filter parameters:

        build_params.png

         

        The filter option is:

        -DitemFilter=(filter expression)

        For example:

        -DitemFilter=(label=xxx)

        Example: Force CI Builder to build only specific jobs from a project, such as jobs in a subfolder called Integration:

        -DitemFilter=(path=Integration)

        Example: Build a specific job called testjob:

        -DitemFilter=(label=testJob)

        For more info on filters, use the CommandLine -help command.

    4. Set the project to use your custom workspace (this assumes the commandline_workspace on the Jenkins machine will be used only for CI Builder).

    5. Add a post-build step for this project to build the RunTests project, which doesn't exist yet. You will create this project in the Testing section below.

      Note: There is no automatic check on this field, so if you enter a project that doesn't exist yet, be sure to create it with the same name and capitalization as what you enter here.

      build_other_runtests.png

       

  5. Save and close.

 

Testing the Configuration

To test your configuration, you will need to build the project to generate the Java source code from the Talend metadata. You can view the console log to monitor progress.

  1. Click Jenkins > New Item, then add a new Maven project called RunTests.
  2. Configure it as follows:

    config_runtests1.png

    config_runtests2.png

    post_xDeploy.png

     

    Note: The xDeploy project doesn't exist yet, but as before, you can add it as a post-build step now and create it later (in the Deploy section).

  3. Run the project to test the configuration.

  4. View your test results on the Test Result page.

    testresult.png

 

Deploy

The final step in Continuous Integration is deploying the compiled code to the repository.

  1. For Jenkins to deploy artifacts to the releases_ci Nexus repository, you need to create another project. In this example, it is named xDeploy so it is alphabetized as the third project in the list (see the Process section). Only the Build fields need to be configured:

    build_xdeploy.png

     

    1. Root POM: use the same target pom.xml you generated earlier (see the CI Builder section).

    2. Goals and options: the action is a built-in Maven goal that deploys to an artifact repository.
    3. MAVEN_OPTS: set this parameter as the deployment repository.
    4. Username and Password: The entries after talendNexus in the screenshot below refer to the set of credentials for Nexus defined in the Maven settings.xml file (see the Maven - Option 1 section).

      talendNexus.png

       

      Note: The version number of the artifact that is published to Nexus will be the Talend version number (that is, the one next to each Job in the Studio). Currently there is no option to control artifact version numbers independently of this version number.

Summary

This article introduced Continuous Integration, Continuous Testing, and Continuous Deployment using Talend CI-Builder, and explained the steps required for you to successfully configure and deploy them.

Version history
Revision #:
5 of 5
Last update:
‎08-14-2017 12:10 PM
Updated by:
 
Labels (2)