Azure DevOps pipeline for Continuous Integration with Talend Administration Center

Overview

This article shows you how to create an Azure DevOps pipeline to implement Continuous Integration (CI) to deploy a Talend Job as an artifact into Nexus. The Nexus artifact is then used to create and run a Task in Talend Administrator Center (TAC).

 

Azure DevOps pipeline uses agents to run CI/CD Jobs, and you can host these agents on any Windows, Linux, or macOS machine. Every agent Job on any Azure DevOps pipeline supports adding different kinds of tasks for building, testing, packaging, and deploying as part of CI/CD process implementation.

 

For more information on Azure DevOps agents, see the Microsoft Build and release agents page, and to learn more about building CI/CD pipelines on Azure DevOps, read the Talend blog, Building a CI/CD pipeline with Talend and Azure DevOps.

 

Prerequisites

 

Environment

  • Operating System (OS): Windows 10
  • Embedded Nexus 3.9.0 - Upgraded to 3.15.2
  • Git repository: GitLab
  • Java version: 1.8.0_191
  • Maven version: 3.5.4
  • Azure DevOps portal: https://dev.azure.com/
  • Windows PowerShell for running Azure DevOps agent

Note: For this KB article, everything is installed on localhost for building the Azure DevOps pipeline proof-of-concept (PoC), but for actual implementation, Talend recommends using the Reference Architecture guidelines.

 

The diagram below depicts the environment and flow.

flow_diagram.png

 

Configuring the Azure DevOps pipeline

  1. Access the Azure DevOps portal https://dev.azure.com/; either through Azure portal https://portal.azure.com/ DevOps project or by registering/signing in with an existing account on the DevOps portal.
  2. Create a private project.

    azure_devops_project.png

     

    Your project dashboard should look similar to this:

    azure_devops_project_dashboard.png

     

  3. Open the Pipelines section, then click New pipeline.

    pipeline_1.png

     

  4. This article uses the GitLab repository; thus you need to select Other Git. However, in a real use case, select the option based on the Git repository that has your code.

    pipeline_2.png

     

  5. Select Other Git as the source, then click Add Connection.

    pipeline_3.png

     

  6. Enter your connections and credentials details. Click OK to save the connection.

    pipeline_4.png

     

  7. Once connected to the Git repository, ensure the branch name (the connection from Talend Studio, that you commit the Job to) is correct.

    pipeline_5.png

     

  8. Click Continue to load the list of available Pipeline templates. Select Maven, from the Featured list.

    pipeline_6.png

     

  9. Click Apply to load the Maven template. This is an empty template that you will configure this in the next few steps.

    pipeline_7.png

     

  10. Before configuring the Maven task; as an optional step, upload the Maven settings file as a secure file into the Pipelines > Library section of the Azure DevOps portal. Create the Maven settings file according to the Talend Software Development Life Cycle Best Practices Guide, Storing your custom Maven settings file in Jenkins section. Instead of creating it in Jenkins, copy the settings.xml content into any text editor, then save the file in a path of your choice.

     

    Either upload the file as a secure file into the Pipelines Library section or save the setting.xml in the local system path. Based on the saved path, specify it in the configuration.

    To create a secure file in the library, upload it from the local system, (click Save on the pipeline configuration page, go to the Library section).

    secure_file_1.png

    secure_file_2.png

    secure_file_3.png

     

  11. Before configuring the Maven task, push the Talend Job you created in Step 9 of the Setting up Talend section to the Git repository. For more information, see How to push changes on a local branch to the remote end (Git only) in the Talend Help Center.

     

Setting up an agent

An agent is a host where a build is queued. An agent can be selected from a Microsoft-hosted pool or a self-hosted pool. You can manage a self-hosted pool; agents can be set up on Azure VM, Windows, Linux, or macOS systems. In this case, the agent is set up on a local system; the agent installation path is C:/agent, so the POM file path configured in Step 1 of the Configuring the Maven task to deploy the Job to Nexus Artifact repository section, points to the agent workspace path where the Git project is checked out as part of the build.

  1. Navigate to the My organizations dashboard, then select Organization settings.

    ad_dashboard_1.png

     

  2. Select Pipelines > Agent pools > New agent pool > Download agent, to open the Get agent page.

    ad_dashboard_2.png

    Note: For information on setting up a self-hosted agent, see the Azure DevOps Services documentation, Deploy an agent on Windows page. Optionally you can use any of the Microsoft hosted agents.

     

  3. Select the OS, in this case, Windows, where you want to set up the agent, then download the agent and follow the steps to install the agent (steps vary for different OS).

    ad_agent_1.png

    In this case, the agent is run in Windows PowerShell in interactive mode.

    powershell.png

    You can choose any label for the self-hosted agent; in this article the agent pool is named Default. Notice that the State column displays Online, this indicates that the agent is set up successfully.

    agent_pool.png

     

Configuring the Maven task to deploy the Job to Nexus Artifact repository

  1. Edit the details of the pipeline, that uses the Maven template, you created in Step 8 and Step 9 of the Configuring the Azure DevOps Pipeline section.

    1. Select the agent of your choice – self-hosted if one is configured according to the Setting up an agent section or a Microsoft hosted agent – from the pull-down menu. For this article, Default is selected, which is hosted on Windows.
    2. Enter the path to your pom.xml file in the Maven POM file text box.

    pipeline_8.png

     

  2. Select the Agent Job 1 label, to deploy the setting for the Pipeline. Add a task by clicking the + sign to the right of Agent Job 1. Enter secure file in the Add tasks search box and select the Download Secure File. Click Add, to access the secure Maven settings file you uploaded in Step 10.

    pipeline_11.png

    pipeline_12.png

    Note: If you did not add a file to the library per Step 10, then in the next step pass the absolute path to the Maven settings file to --settings option and ensure that the absolute path is a path on the agent where Maven settings file is available.

     

  3. Select the Maven task and configure the build as shown below:

    pipeline_13.png

    Note: The generic Goal(s) command is:

    --settings  clean deploy -fn -e -pl  -am

     

  4. Notice that Custom Path is selected for the Maven path; this means that an existing Maven installation is required. Using Default does not require a separate installation of Maven.

  5. Select the Save & queue option to queue the build.

    To save the Pipeline configuration (without running a build), select the Save option from the Save & queue drop-down menu.

    When queuing Azure gives you the option to select agent pool and branch again, in case you want to queue the build in a different environment (agent) or use a different branch for the build.

    savebuildpipelineandqueue.PNG

  6. The build runs. All builds are listed under Pipelines > Builds.

    pipeline_14.png

     

  7. Click the build to look at the list of tasks executed. Notice that the Maven task took 1 minute 31 seconds to run. Azure DevOps also automatically executes tasks such as Prepare Job, Initialize job, Checkout, and Finalize Job.

    pipeline_15.png

     

  8. Click any task to view the logs related to that task. The logs for the Maven task are shown below:

    pipeline_16.png

    pipeline_17.png

     

  9. Log in to Nexus, select Browse, navigate to releases repository and expand the path where the artifact is deployed.

    pipeline_18.png

     

  10. In the next section, you will create a task for the Nexus Artifact and schedule the task to run at required intervals. In this case, the task is run manually.

     

Creating a task in TAC from Nexus artifact

  1. Log in to TAC, and ensure the TAC configuration uses the correct Nexus. Navigate to Job Conductor, then create a new Artifact Task using the artifact you deployed in Nexus (Step 9 in the Configuring the Maven task to deploy the Job to Nexus Artifact repository section).

    pipeline_19.png

    pipeline_20.png

     

  2. Run the task.

    pipeline_21.png

     

Conclusion

This article explains how to create an Azure DevOps pipeline as part of the CI process to deploy Talend artifact to Nexus. Using Azure DevOps pipeline is beneficial for use cases that are implemented with Azure services; it easy to build a Talend Job for scenarios that require consuming or publishing data to different services in Microsoft Azure and to use Azure DevOps pipeline to automate the build process.

Version history
Revision #:
28 of 28
Last update:
‎05-20-2019 02:20 PM
Updated by: