This article explains how to set up Continuous Integration, Continuous Testing, and Continuous Deployment using Talend CI-Builder. It covers the required preparation in your environment before installation, the configuration and file system folder structures suggested by Talend, and the steps to automate the build process using Jenkins and Maven.
Note: You can use any CI tool that supports Maven. This article shows how to use Jenkins.
The following Talend environment must be installed. For details, see these pages on the Talend Help Center.
Talend Administration Center
Set up with a dedicated command line and port (minimum version 6.1.1). For more information, see Installing and configuring Talend Administration Center.
The Nexus repository is configured, the server is accessible from the CI server, and the interaction between Nexus, TAC, and Studio is fully working. Any HTTP or HTTPS proxies that are present in the environment must be accounted for, and you should ensure that Talend environment communications are not going over a proxy unless it is necessary. See Installing and configuring the Nexus artifact repository under Installing and Configuring Talend server modules.
Subversion or Gitlab
SVN or GIT must be integrated with TAC and Studio, and must be tested and fully functional.
Projects, reference projects, and project authorizations in TAC are set up, with users having appropriate access/authorization.
Jobs containing test cases for Continuous Testing. To familiarize yourself with the test cases, review these videos:
|Component||Processor||RAM||Disk Space||Operating System|
|Talend Administration Center + Talend Activity Monitoring Console Web application||2 CPU||8 GB||1 GB||Linux, Windows, MacOS|
|CommandLine||2 CPU||5 GB||3 GB|
|JobServer||1 CPU||1 GB||20 MB|
|Studio||2 CPU||4 GB||3 GB|
|Talend Runtime||1 CPU||4 GB||400 MB|
|Product||Operating System||Disk Space|
|Installer||Windows or UNIX||1 GB|
If the TAC server has internet access, and the software update configuration is correct in TAC, you can automatically get the patch from Talend on the software update page. Once you choose to download the patch, it should be made available for all Studios and Command Line applications.
For more information, see Configuring the Software Update repository in Talend Administration Center.
If the TAC server has no internet access, follow the instructions in Uploading a patch in the local Nexus to manually download the patch and upload it to your local Nexus.
Having a dedicated CI server is recommended. To run CI in the server, it will need dedicated storage space, so create the folder structure as shown below:
commandline_workspace: allows Jenkins, CommandLine, and CI Builder to utilize and share information. If you have CommandLine currently installed, you can use the existing CommandLine folder and copy it over instead of creating a new commandline_workspace.
JENKINS_HOME: stores the Jenkins configuration (system and projects). Jenkins_Home is also the name of the environment variable.
maven_repo: this folder is optional. Create it only if you need Maven to use the CommandLine Maven artifact repository.
target: stores a copy of the Java code, compiled artifacts, and test results.
To prevent file locking and permissions issues, all components should be run using the same user, for example talenduser on UNIX or Local System Account on Windows. If you choose to change the default users, be very precise with file system permissions for the folders, including their children.
This process covers setup on Windows Server 2008, but most of the steps are identical in a UNIX-based machine. This assumes that the server is properly configured with a Java 8 JDK, the environment connectivity is working correctly, and your JAVA_HOME environment variable is set.
Using the Talend installer, install CommandLine and its Server Service to wherever you will later install Jenkins and Maven. Alternatively, you can use an existing CommandLine install, or turn on Studio's CommandLine, instead of installing a new copy.
Note: The installer may attempt to start the Windows service for CommandLine on completion.
Browse to your CommandLineHome/TalendServices/conf directory:
Edit the wrapper.conf file and find the following section:
#******************************************************************** # genConfig: further Properties generated by genConfig #********************************************************************
Alter the parameter wrapper.app.parameter.7 to point to your commandline_workspace folder:
When CommandLine is used with CI, it will not connect to TAC and does not find the location of the Nexus libraries repository (which by default is talend-custom-libs). Connecting to TAC is required to compile and build the Talend jobs, so to integrate Nexus, you must set the location of the Nexus libraries repository for CommandLine and add the other parameters listed below to the Talend_Root/CommandLineHome/studio/configuration/config.ini file.
Navigate to and open the config.ini file:
Add the following parameters to the file:
nexus.url=http://localhost:8081/nexus nexus.user=admin nexus.password=Talend123 nexus.lib.repo=talend-custom-libs
Note: If your user and password are the defaults shown above you don't need to include them in the file, though if you changed the defaults, specify the appropriate values.
Start the CommandLine service.
Important: Do not use this CommandLine for anything other than Continuous Integration. It is recommended that the configuration folder of CommandLine remains in its default location.
This server is dedicated for CI use only, and the CI CommandLine cannot be used for normal CommandLine operations such as Audit, TAC publisher, TAC Job Conductor generation, Studio remote generation, and scripting.
Important: Verify that Studio is not being used during installation.
The CI Server will host the following components:
Optional: This server can be a logical place to install SVN, GIT, and/or Nexus.
Download Maven from Downloading Apache Maven site (binary zip archive).
Extract the downloaded zip file.
The default location of Maven is ~/.m2 (user home, .m2 directory). With this setup, Maven will use CommandLine’s .m2 repository, as it is the simplest way of ensuring that Maven has the correct Talend libraries.
Browse to MAVEN_ROOT_INSTALL_DIR/conf/settings.xml.
Add the following property:
Where the path specified is the path to the CommandLine .m2 repository:
Navigate to the <servers> section of the same file and add an entry for the Nexus credentials:
<server> <id>talendNexus</id> <username>admin</username> <password>Talend123</password> </server>
You can obfuscate the password if required. Review the instructions on the Maven site: Password Encryption.
To move the path of the local repository, you need to set the path in settings.xml.
Set up a <server> set of credentials for Nexus as described in Option 1. It is possible to have multiple profiles here if you need to use different users for accessing the libraries repository and the deployment repository:
<server> <id>talendNexusDeploy</id> <username>deploy</username> <password>deployPWD</password> </server> <server> <id>talendNexusLibs</id> <username>talend-custom-libs-admin</username> <password>talend-custom-libs-admin</password> </server>
Locate the <profiles> section and add your Talend libraries repository as a new profile:
<profile> <id>talendNexusLibs</id> <activation> <property> <name>talendNexusLibs</name> <value/> </property> </activation> <repositories> <repository> <id>nexus-repository</id> <url>http://win-1k2cbe6vvuj:8081/nexus/content/repositories/talend-custom-libs</url> <releases> <enabled>true</enabled> </releases> <snapshots> <enabled>true</enabled> </snapshots> </repository> </repositories> </profile>
The <id> tag should have the same <id> as the <server> section, with the correct credentials of your repository.
Add an <activeprofile> to the <activeprofiles> section:
<activeProfiles> <activeProfile>talendNexusLibs</activeProfile> </activeProfiles>
The file should look like this:
Now Maven will be able to access the Talend Nexus for Talend libraries.
Download Talend CI Builder as described in your Talend license email.
Unzip the downloaded file into your root CI folder. This creates ci.builder-version.jar and .pom.
Open a command prompt and browse to the directory where you installed Maven /bin.
mvn install:install-file -Dfile=CI-Builder_install_dir\ci.builder-6.3.1.jar -DpomFile=CI-Builder_install_dir\ci.builder-6.3.1.pom
This will install CI Builder into Maven, and the supplied pom file will list the dependencies of CI Builder. It will then download these dependencies as part of the install process.
Depending on your setup, the machine will need internet access to download these libraries, or you may have configured a repository for Maven (using Option 2) that contains these libraries.
Create a new file in your CI Root folder called pom.xml, and copy the following contents into it.
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>org.talend</groupId> <artifactId>buildsources</artifactId> <version>0.1</version> <packaging>pom</packaging> <build> <plugins> <plugin> <groupId>org.talend</groupId> <artifactId>ci.builder</artifactId> <version>6.3.1</version> <executions> <execution> <phase>validate</phase> <goals> <goal>generate</goal> </goals> </execution> </executions> </plugin> </plugins> </build> </project>
A Project Object Model or POM is the fundamental unit of work in Maven. It is an XML file that contains information about the project and configuration details used by Maven to build the project. It contains default values for most projects. Some of the configuration that can be specified in the POM are the project dependencies, the plugins or goals that can be executed, the build profiles, and so on. Other information such as the project version, description, developers, mailing lists and such can also be specified. This pom file defines the goals that the first stage of the CI process will attempt to achieve when executed from Jenkins.
To segregate the CI Jobs, create a repository called releases_ci. The Nexus admin user was created to access this repository for deployment actions.
The recommended best practice is to set up suitable users and groups.
As an option for the artifacts to be generated by CI Builder and pushed to a Nexus repository, it is recommended to set up a unique repository for this purpose as shown below:
Follow the installation wizard prompts:
By default, the Jenkins directory is located in ~/.jenkins under the user that is running the Jenkins home directory, and where all the configuration files are stored. If running it as a Windows service, it is located under the Windows system folders in Users/user/.jenkins.
To run the service as a different user, that same user would also need to run CommandLine.
You can set a different path for JENKINS_HOME by editing the file JENKINS_INSTALL_ROOT/Jenkins.xml.
Note: If this was deployed as a war file, you will not have a Jenkins.xml file.
Start the Jenkins service and check that it can connect to the web UI (for example, http://localhost:8380).
For more information about administering Jenkins, see the Administering Jenkins site.
Click Manage Jenkins > Manage Plugins.
Install the Maven Integration Plugin, and either the Subversion Plugin or the Gitlab Plugin, and verify the plugins are enabled.
Click Manage Jenkins > Configure System.
The home directory should reflect how you configured it previously (while installing Jenkins).
In the Global Tool Configuration, under JDK installations, add the path to the Java 8 JDK installation on your CI Server.
Under Maven installations, add the path to the Maven installation on your CI server.
The Process of Continuous Integration, Continuous Testing, and Continuous Deployment begins once a job is committed to the central repository.
The build process uses Jenkins and Maven to take the completed Job and generate Java source code. It also compiles the Job into a Java executable and then publishes it to Nexus. Once the Job is published to Nexus, it can be deployed into any environment in a self-contained executable file. The following sections explain the steps to automate this process using Jenkins Jobs. There are three Jenkins Jobs, as shown below, to automate the build process:
Building is the process of generating Java source code and Java executable binaries.
Create a new Job by clicking Jenkins > New Item and adding a new Maven project named BuildSources.
Configure the Source Code Management tab.
If using Git, provide the Git repository and other details. Configure the repository URL to point to a specific Tag, Branch, and Trunk.
If using SVN, configure the repository URL to point to a specific Tag, Branch, and Trunk on SVN.
Set the local module directory to the same as the project name (multiple branches of the same projects might require unique names).
This will allow Jenkins to perform an SVN checkout to commandline_workspace that replicates the file system structure that would be created if Studio or CommandLine performed a check out of the project.
Note: You will be prompted to enter credentials to communicate with SVN.
Examples of the folder structure comparing Studio to CI:
On the Build tab, set the build options:
In addition to these parameters, it is possible to filter what CommandLine builds. Behind the scenes, CI builder is calling the CommandLine command buildProjectSources, which can take filter parameters:
The filter option is:
Example: Force CI Builder to build only specific jobs from a project, such as jobs in a subfolder called Integration:
Example: Build a specific job called testjob:
For more info on filters, use the CommandLine -help command.
Set the project to use your custom workspace (this assumes the commandline_workspace on the Jenkins machine will be used only for CI Builder).
Add a post-build step for this project to build the RunTests project, which doesn't exist yet. You will create this project in the Testing section below.
Note: There is no automatic check on this field, so if you enter a project that doesn't exist yet, be sure to create it with the same name and capitalization as what you enter here.
To test your configuration, you will need to build the project to generate the Java source code from the Talend metadata. You can view the console log to monitor progress.
Configure it as follows:
Note: The xDeploy project doesn't exist yet, but as before, you can add it as a post-build step now and create it later (in the Deploy section).
Run the project to test the configuration.
View your test results on the Test Result page.
The final step in Continuous Integration is deploying the compiled code to the repository.
For Jenkins to deploy artifacts to the releases_ci Nexus repository, you need to create another project. In this example, it is named xDeploy so it is alphabetized as the third project in the list (see the Process section). Only the Build fields need to be configured:
Root POM: use the same target pom.xml you generated earlier (see the CI Builder section).
Username and Password: The entries after talendNexus in the screenshot below refer to the set of credentials for Nexus defined in the Maven settings.xml file (see the Maven - Option 1 section).
Note: The version number of the artifact that is published to Nexus will be the Talend version number (that is, the one next to each Job in the Studio). Currently there is no option to control artifact version numbers independently of this version number.
This article introduced Continuous Integration, Continuous Testing, and Continuous Deployment using Talend CI-Builder, and explained the steps required for you to successfully configure and deploy them.