Install, configure, and automate Talend 7 Continuous Integration with Jenkins

Overview

Talend Continuous Integration (CI) is now fully compliant with Maven standards, and Continuous Integration and Deployment (CI/CD) with Talend has never been easier.

 

This tutorial illustrates how to automate CI/CD with Talend 7 and Jenkins, by complementing current Talend documentation. For more information, see the Talend Software Development Life Cycle Best Practices Guide.

 

Requirements

 

Architecture

This example uses a Continuous Integration server (Jenkins) by leveraging Talend CI Builder, Bitbucket as a service for the code repository, and the Talend Nexus Artifact Repository.

 

You can continue to use TAC to publish Jobs to a Nexus repository using the Publisher page, but the Publisher page is deprecated in TAC, so Talend recommends using a Maven build.

 

Talend CI Builder is a Maven plugin, delivered by Talend, that transforms the Talend Job sources to Java classes using the Talend CommandLine application. This allows you to execute your tests in your own company Java factory.

 

  • The overall high-level architecture:

    image.png

     

  • Talend platform architecture and components on a Windows Server machine:

    image.png

     

  • CI server architecture and components on a Red Hat machine:

    image.png

     

 

Installing the Talend platform

 

Install TAC and CommandLine

  1. Download the Talend-Installer, the Dist file, and your License.

  2. Extract and store them in the same folder.

    image.png

     

  3. Double-click the installer.

  4. Choose Advanced Install, choose Custom, then browse to your License File.

    image.png

     

  5. Select Talend Administration Center and Talend Command Line. Click Next.

    image.png

     

  6. Select Talend Administration Center, Talend Command Line, and Talend Server Services. Click Next.

    image.png

     

  7. Choose Install an embedded tomcat8 server on the drop-down list. Keep the default Create TAC administrator user setting. Click Next.

    image.png

     

  8. Choose Embedded H2 database on the drop-down list and select Install Nexus server with TAC.

    image.png

     

  9. Keep the default Nexus Port.

    image.png

     

  10. Keep the default CommandLine port.

    image.png

     

  11. Select Install Talend Administration Center as a service and Install Talend Command Line as a service.

    image.png

     

  12. The Talend platform will install.

    image.png

     

 

Set up Bitbucket as a service

  1. Create a Bitbucket account or use your existing one.

  2. Click the plus [+] sign.

    image.png

     

  3. From the CREATE menu, select Repository.

    createrepbitbucket.PNG

     

  4. Create a new repository and give it a name. Select Git as the version control system, then click Create repository.

    image.png

     

 

Set up Talend Administration Center

  1. Go to the TAC web interface.

  2. Login with security@company.com and your password.

    image.png

     

  3. Create a new user. Complete your Git login with your Bitbucket login, click Validate, then click Save.

    image.png

     

  4. Log out, then log back in as the user you just created.

  5. Create a new project.

    image.png

     

  6. Navigate to your Bitbucket repository and find the Git URL.

    image.png

     

  7. Complete your project details. Check the connection to your Bitbucket repository, if it is OK, click Save.

    image.png

     

  8. Grant your user write privileges on the newly created project.

    image.png

     

 

Set up Talend Nexus

  1. Open the Nexus web interface.

  2. Login with the user admin and password Talend123.

  3. Navigate to Server Administration and configuration, click Repositories, then select Create repository.

    image.png

     

  4. Create a new repository, choose maven2 (hosted) and configure it, as shown below. Use this repository to deploy snapshot versions of your Talend Job.

    image.png

     

  5. Create a second repository, choose maven2 (hosted) and configure it, as shown below. Use this repository to deploy release version of your Talend Job.

    image.png

     

  6. Create a third repository, choose maven2 (hosted) and configure it, as shown below. This repository is used by Talend CI Builder and is defined later in the maven_user_settings.xml file.

    image.png

     

  7. When the installation is complete, open Studio.

 

Installing Talend Studio

 

Install Studio and connect to TAC

  1. On another machine, install Talend Studio using the installer.

  2. When the installation is complete, open Studio.

  3. Click Manage Connection, click the green plus [+] sign to create a Remote TAC connection. Enter the User name, User Password, and the Web-app Url. Click Check url to make sure the connection is working.

    image.png

     

  4. Select the connection you created. Click Finish.

    image.png

     

 

Create a sample Job

Create a simple Job that reads two flat files and uses a tMap component to join the files.

 

  1. Add two tFileInputDelimited components to the Designer.

    image.png

     

  2. Drag and drop a tMap component and connect your input files to the tMap. Use one as a source table and the other as a lookup table.

    image.png

     

  3. Use an ID common for both files and join them using the tMap interface.

    image.png

     

  4. Drag and drop a tFileOutputDelimited component and connect it at the output of the tMap component to store the results of the join into a file.

    image.png

     

  5. Run your Job and make sure that it runs successfully.

 

Create test cases

  1. Right-click the tMap component within the Job, then select Create Test Case.

    image.png

     

  2. Give it a name, then click Finish.

    image.png

     

    A test case Job has been generated automatically for you.

    image.png

     

  3. As the note in Step 2 indicates, you need sample data to complete the step. Create another Job with the files you used in the tFileInputDelimited components, but only extract 200 rows by using a tSampleRow component.

    image.png

     

  4. Create another Job with the sample files as input, connect the same tMap component as before, and connect a tOutputfiledelimited component at the output to get a sample output for your test cases.

    image.png

     

  5. After the sample files are generated, double-click the testcase. Select the TestCase tab and open the Default test case.

    image.png

     

  6. For each input_file and reference_file, click File Browse and select each previously generated sample file.

    image.png

     

  7. Right-click the Default test, select Run Instance, and make sure that your test runs successfully.

    image.png

    Your test cases are now set up, and you can automate them within the Jenkins pipeline.

     

  8. Before going to the next steps, make sure you have saved your Job and pushed the modification to your Bitbucket repository.

    image.png

     

 

Configuring the CI server

 

Install Maven

  1. Login to the Red Hat machine.

  2. From the Apache Maven Project web page, download Maven.

  3. Extract the ZIP archive to the directory where you want to install Maven.

  4. Open a terminal and add the M2_HOME environment variable by entering the following command:

    export M2_HOME=/opt/apache-maven/apache-maven-3.0.x
  5. Add the M2 environment variable by entering the following command:

    export M2=$M2_HOME/bin
  6. Add the M2 environment variable to your path by entering the following command:

    export PATH=$M2:$PATH
  7. Make sure that JAVA_HOME is set to the location of your JDK.

  8. Verify that Maven is installed successfully on your machine, by running the command:

    mvn --version

 

Install Git

  1. Ensure that your system is up-to-date with the latest version of packages by running the YUM package manager update command, as shown below:

     yum update
  2. Install Git by running the command:

     yum install git 
  3. Verify that Git is installed successfully on your machine, by running the command:

    git --version

 

Install and configure Talend CommandLine

  1. Ensure that the dist file is in the same folder as the Talend-Tools-Installer-YYYYYYYY_YYYY-VA.B.C-linux64-installer.run file.

  2. Make the Talend-Tools-Installer-YYYYYYYY_YYYY-VA.B.C-linux64-installer.run file executable, using the following command. If you want to install Talend server modules as services, execute this command with the super-user rights.

    chmod +x Talend-Tools-Installer-YYYYYYYY_YYYY-VA.B.C-linux64-installer.run
  3. Launch Talend Installer by running the command:

    ./Talend-Tools-Installer-YYYYYYYY_YYYY-VA.B.C-linux64-installer.run
  4. Accept the License Agreement, and choose the directory where you want your Talend product to be installed.

  5. Choose Advanced Install from the installation style list, and Custom from the installation type list.

  6. Add your license file and launch the installation.

  7. Install only Talend CommandLine with default port 8002.

  8. Start Talend CommandLine at least once to initialize its default Maven repository, then close it.

  9. Edit the commandlinePath/configuration/maven_user_settings.xml file and add the connection information to the Nexus repositories. In your case, Nexus is on a remote server where you installed the Talend platform, so replace localhost with the private EC2 IP address.

    <?xml version="1.0" encoding="UTF-8"?>
    <settings xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd" xmlns="http://maven.apache.org/SETTINGS/1.1.0"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
      <localRepository>/<.m2Path>/repository</localRepository>
    
    <servers>
        <!-- credentials to access the default releases/snapshots repositories -->
        <server>
            <id>releases</id>
            <username>admin</username>
            <password>Talend123</password>
        </server>
        <server>
            <id>snapshots</id>
            <username>admin</username>
            <password>Talend123</password>
        </server>
        <!-- credentials to access the repositories holding external jars -->  
        <server>
            <id>talend-custom-libs-release</id>
            <username>admin</username>
            <password>Talend123</password>
        </server>
        <server>
            <id>talend-custom-libs-snapshot</id>
            <username>admin</username>
            <password>Talend123</password>
        </server>
        
        <!-- credentials to access the repositories holding maven plugins -->
        <server> <!-- central (as proxy) -->
            <id>central</id>
            <username>admin</username>
            <password>Talend123</password>
        </server>
        <server>
            <id>thirdparty</id>
            <username>admin</username>
            <password>Talend123</password>
        </server>
        
    </servers>
    
    <mirrors/>
    <proxies/> <!-- http proxies, not maven proxy repositories -->
    
    <profiles>
        <profile>
            <id>talend-ci</id>
            <repositories>
                <repository>
                    <id>central</id>
                    <name>central</name>
                    <url>http://localhost:8081/repository/maven-central/</url>
                    <layout>default</layout>
                </repository>
                <repository>
                    <id>talend-custom-libs-release</id>
                    <name>talend-custom-libs-release</name>
                    <url>http://localhost:8081/repository/talend-custom-libs-release</url>
                    <layout>default</layout>
                    <releases>
                        <enabled>true</enabled>
                    </releases>
                    <snapshots>
                        <enabled>false</enabled>
                    </snapshots>
                </repository>
                <repository>
                    <id>talend-custom-libs-snapshot</id>
                    <name>talend-custom-libs-snapshot</name>
                    <url>http://localhost:8081/repository/talend-custom-libs-snapshot</url>
                    <layout>default</layout>
                    <releases>
                        <enabled>false</enabled>
                    </releases>
                    <snapshots>
                        <enabled>true</enabled>
                    </snapshots>
                </repository>
            </repositories>
            
            <pluginRepositories>            
                <pluginRepository>
                    <id>central</id>
                    <name>central</name>
                    <url>http://localhost:8081/repository/maven-central/</url>
                    <layout>default</layout>
                </pluginRepository>
                <pluginRepository>
                    <id>thirdparty</id>
                    <name>thirdparty</name>
                    <url>http://localhost:8081/repository/thirdparty</url>
                    <layout>default</layout>
                </pluginRepository>
            </pluginRepositories>
        </profile>
    </profiles>
    
    <activeProfiles>
        <activeProfile>talend-ci</activeProfile>
    </activeProfiles>
    
    </settings>

 

Install CI Builder

  1. Extract the Talend-CI-Builder-V7.0.1.zip archive file in the directory of your choice.

  2. Browse to the installation directory and execute the following command:

    mvn install:install-file -Dfile=ci.builder-7.0.1.jar -DpomFile=ci.builder-7.0.1.pom
  3. Browse to the CI Builder installation directory and execute the following command to deploy the new repository on Nexus:

    mvn deploy:deploy-file -Dfile=ci.builder-7.0.1.jar -DpomFile=ci.builder-7.0.1.pom -DrepositoryId=thirdparty -Durl=http://127.0.0.1:8081/repository/talend-custom-libs-release/ -s <commandlinePath>/configuration/maven_user_settings.xml

    where the -Durl parameter value corresponds to your repository URL on Nexus, and the -s parameter value corresponds to the path to your maven_user_settings.xml file.

  4. Log back in to the Nexus web UI, click Browse, then navigate to the thirdparty repository.

    image.png

     

  5. This Maven plugin is now available for anyone and can be incorporated in your builds.

 

Install Jenkins

  1. From the Jenkins web page, download Jenkins.

  2. Download the appropriate version for your environment.

    image.png

     

  3. Before starting the installation, verify that you have installed a Java JDK, and set your JAVA_HOME and PATH, then follow the Jenkins documentation on Installing Jenkins on Red Hat distributions.

 

Install Jenkins plugins

  1. After installing Jenkins with the default setup, log in and install additional plugins.

    image.png

     

  2. Click Manage Jenkins then select Manage Plugins.

    image.png

     

  3. Click the Available tab.

    image.png

     

  4. Select and install the following plugins: Bitbucket, GitLab, Pipeline, Build Pipeline, Green Balls, Publish Over SSH, SSH, and Workspace Cleanup.

 

Configuring Jenkins

  1. Navigate to the Jenkins main page.

  2. Click Manage Jenkins, then Global Tool Configuration.

    image.png

     

  3. Click JDK > JDK installations, fill in Name and JAVA_HOME, as it is set up on your CI server, then Save.

    image.png

     

  4. Click Git, fill in the Name and Path to Git executable where Git is installed on your CI server, then Save.

    image.png

     

  5. Click Maven installations, fill in the Name and MAVEN_HOME of your CI server, then Save.

    image.png

     

  6. You are all set to start building a Jenkins CI Pipeline. You will specify your Bitbucket credential within your Jenkins Jobs.

 

Create a Jenkins CI Pipeline

image.png

 

Add a Bitbucket webhook

  1. Login to Bitbucket.

  2. Select your repository.

  3. Click Settings.

    image.png

     

  4. From WORKFLOW select Webhooks.

    image.png

     

  5. Click Add webhook, specify a title, and build the URL as follows: http://PUBLIC IP of JENKINS:8080/bitbucket-hook/. Configure the remaining options as shown below, then Save.

    webhook1.png

     

  6. To the right of your webhook, click View requests.

    webhook2.png

     

  7. Go back into Studio and make a small change to your Job, for example, change the name of a component, then push the change to Git, come back to the Bitbucket webhooks request, and make sure that the webhook is working.

    webhook3.png

     

 

Compile

  1. Go to Jenkins main page and select New Item.

    image.png

     

  2. Enter the Job name: 01_BitBucket_Compile, select Maven Project, click OK.

    image.png

     

  3. Scroll down to Source Code Management, select Git, enter your Repository URL, add your credentials, and specify the branch, for example, */master.

    image.png

     

  4. Scroll down to Build Trigger, then select Build when a change is pushed to BitBucket.

    image.png

     

  5. Scroll down to Build Environment, then select Delete workspaces before starts.

    image.png

     

  6. Scroll down to Build, and complete the first section as shown below: add the Root POM of your Talend project, define your Maven Goals and options, and point to your cmdline instance for MAVEN_OPTS.

    image.png

     

  7. Complete the section as shown below: locate your maven_user_settings.xml file.

    image.png

     

  8. Click Save. Your first Jenkins Job is complete.

    image.png

     

  9. To run it manually, click the green arrow with a clock, on the right side of the screen.

    image.png

     

  10. Click the console output to see the results, and you should end up with the status: Finished: SUCCESS.

    image.png

     

  11. As you did earlier, make a small change to your Job from the Studio, push the modification to Git, and make sure that the compile of the Jenkins Job has been triggered.

    buildexecutorstatus.png

     

 

Test

  1. Create a new Maven project, name it 02_Test, scroll down to Build Triggers, configure the project as shown below:

    image.png

     

  2. Set up the Build Environment:

    image.png

     

  3. Configure Build as shown below, then click Save:

    image.png

     

 

Package

  1. Create a new Maven project, name it 03_Package, scroll down to Build Triggers, and configure the project as shown below:

    image.png

     

  2. Set up the Build Environment:

    image.png

     

  3. Configure Build as shown below, then click Save:

    image.png

     

 

Install

  1. Create a new Maven project, name it 04_Install, scroll down to Build Triggers, and configure the project as shown below:

    image.png

     

  2. Set up the Build Environment:

    image.png

     

  3. Configure Build as shown below, then click Save:

    image.png

     

  4. Create a new Maven project, name it 05_Deploy, scroll down to Build Triggers, and configure the project as shown below:

    image.png

     

  5. Set up the Build Environment:

    image.png

     

  6. Configure Build as shown below, then click Save:

    image.png

     

    MAVEN_OPTS:

    -Dproduct.path=/opt/cmdline
    -Dgeneration.type=local
    -DaltDeploymentRepository=snapshots::default::http://YOUR NEXUS PUBLIC IP:8081/repository/ben_snapshots/
  7. Create a new Pipeline view from the Jenkins main page by clicking the plus [+] sign next to the tabs.

    image.png

     

  8. Give it a name, Pipeline View, select Build Pipeline View, then OK.

    image.png

     

  9. Specify the Pipeline Flow, as shown below. Keep the rest of the default settings. Click OK.

    image.png

     

  10. Run the first job manually, or make a small change to your Talend Job, and push the update to Bitbucket. Click the Pipeline view you just created.

    image.png

     

  11. Make sure that your pipeline ran successfully, and check your repository to confirm that your Talend Job has deployed.

    image.png

     

    image.png

     

Version history
Revision #:
25 of 25
Last update:
‎10-02-2018 06:05 AM
Updated by:
 
Comments
Four Stars

When i build pom.xm in poms folder, i'm getting following error

[ERROR] Failed to execute goal org.talend:ci.builder:7.0.1:generate (default) on project code.Master: Fail to execute command: BuildItemSourcesCommand buildItemSources '_Cre8IMdzEeirn5yX-EZo7Q' -iv '0.1': Have exceptions when execute the command: BuildItemSourcesCommand buildItemSources '_Cre8IMdzEeirn5yX-EZo7Q' -iv '0.1': java.lang.RuntimeException: java.lang.NullPointerException
[ERROR] at org.talend.commandline.command.ExtensionCommandSwitch.caseExtensionServerCommand(ExtensionCommandSwitch.java:191)
[ERROR] at org.talend.commandline.client.command.extension.AbstractExtensionCommandSwitch.doSwitch(AbstractExtensionCommandSwitch.java:31)
[ERROR] at org.talend.commandline.command.CommandProcessorSwitch.caseExtensionCommand(CommandProcessorSwitch.java:179)
[ERROR] at org.talend.commandline.client.util.CommandAbstractSwitch.doSwitch(CommandAbstractSwitch.java:70)
[ERROR] at org.talend.commandline.command.CommandConsumer.executeCommand(CommandConsumer.java:57)
[ERROR] at org.talend.commandline.command.CommandConsumer.execute(CommandConsumer.java:35)
[ERROR] at org.talend.commandline.mode.ServerCommandLine$CommmandConsumerRunnable.run(ServerCommandLine.java:139)
[ERROR] at java.lang.Thread.run(Thread.java:748)
[ERROR] Caused by: java.lang.NullPointerException
[ERROR] at org.talend.commandline.command.execute.BuildItemSourcesExecuteCommand.execute(BuildItemSourcesExecuteCommand.java:91)
[ERROR] at org.talend.commandline.command.ExtensionCommandSwitch.caseExtensionServerCommand(ExtensionCommandSwitch.java:189)
[ERROR] ... 7 more
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.talend:ci.builder:7.0.1:generate (default) on project code.Master: Fail to execute command: BuildItemSourcesCommand buildItemSources '_Cre8IMdzEeirn5yX-EZo7Q' -iv '0.1'

 

it doesn't like mvn org.talend:ci.builder:generate

Community Manager

Hi @vivek88vardhan,

You'll get more answers to your question if you post it to one of the Community discussion boards, for example Design and Development.

HTH,

Alyce

 

Employee

Hi @vivek88vardhan can you tell me on which Jenkins job this error is happening?