This article shows you how to create an Azure DevOps pipeline to implement Continuous Integration (CI) to deploy a Talend Job as an artifact into Nexus. The Nexus artifact is then used to create and run a Task in Talend Administrator Center (TAC).
Azure DevOps pipeline uses agents to run CI/CD Jobs, and you can host these agents on any Windows, Linux, or macOS machine. Every agent Job on any Azure DevOps pipeline supports adding different kinds of tasks for building, testing, packaging, and deploying as part of CI/CD process implementation.
For more information on Azure DevOps agents, see the Microsoft Build and release agents page, and to learn more about building CI/CD pipelines on Azure DevOps, read the Talend blog, Building a CI/CD pipeline with Talend and Azure DevOps.
A current installation of Talend tools, following the guidelines in the Talend Data Fabric Installation Guide and including the following components:
Talend Administration Center
Use the following settings:
Note: For this KB article, everything is installed on localhost for building the Azure DevOps pipeline proof-of-concept (PoC), but for actual implementation, Talend recommends using the Reference Architecture guidelines.
The diagram below depicts the environment and flow.
Create a private project.
Your project dashboard should look similar to this:
This article uses the GitLab repository; thus you need to select Other Git. However, in a real use case, select the option based on the Git repository that has your code.
Select Other Git as the source, then click Add Connection.
Enter your connections and credentials details. Click OK to save the connection.
Once connected to the Git repository, ensure the branch name (the connection from Talend Studio, that you commit the Job to) is correct.
Before configuring the Maven task; as an optional step, upload the Maven settings file as a secure file into the Pipelines > Library section of the Azure DevOps portal. Create the Maven settings file according to the Talend Software Development Life Cycle Best Practices Guide, Storing your custom Maven settings file in Jenkins section. Instead of creating it in Jenkins, copy the settings.xml content into any text editor, then save the file in a path of your choice.
Either upload the file as a secure file into the Pipelines Library section or save the setting.xml in the local system path. Based on the saved path, specify it in the configuration.
To create a secure file in the library, upload it from the local system, (click Save on the pipeline configuration page, go to the Library section).
Before configuring the Maven task, push the Talend Job you created in Step 9 of the Setting up Talend section to the Git repository. For more information, see How to push changes on a local branch to the remote end (Git only) in the Talend Help Center.
An agent is a host where a build is queued. An agent can be selected from a Microsoft-hosted pool or a self-hosted pool. You can manage a self-hosted pool; agents can be set up on Azure VM, Windows, Linux, or macOS systems. In this case, the agent is set up on a local system; the agent installation path is C:/agent, so the POM file path configured in Step 1 of the Configuring the Maven task to deploy the Job to Nexus Artifact repository section, points to the agent workspace path where the Git project is checked out as part of the build.
Navigate to the My organizations dashboard, then select Organization settings.
Select Pipelines > Agent pools > New agent pool > Download agent, to open the Get agent page.
Note: For information on setting up a self-hosted agent, see the Azure DevOps Services documentation, Deploy an agent on Windows page. Optionally you can use any of the Microsoft hosted agents.
Select the OS, in this case, Windows, where you want to set up the agent, then download the agent and follow the steps to install the agent (steps vary for different OS).
In this case, the agent is run in Windows PowerShell in interactive mode.
You can choose any label for the self-hosted agent; in this article the agent pool is named Default. Notice that the State column displays Online, this indicates that the agent is set up successfully.
Select the Agent Job 1 label, to deploy the setting for the Pipeline. Add a task by clicking the + sign to the right of Agent Job 1. Enter secure file in the Add tasks search box and select the Download Secure File. Click Add, to access the secure Maven settings file you uploaded in Step 10.
Note: If you did not add a file to the library per Step 10, then in the next step pass the absolute path to the Maven settings file to --settings option and ensure that the absolute path is a path on the agent where Maven settings file is available.
Select the Maven task and configure the build as shown below:
Note: The generic Goal(s) command is:
--settings clean deploy -fn -e -pl -am
Notice that Custom Path is selected for the Maven path; this means that an existing Maven installation is required. Using Default does not require a separate installation of Maven.
Select the Save & queue option to queue the build.
To save the Pipeline configuration (without running a build), select the Save option from the Save & queue drop-down menu.
When queuing Azure gives you the option to select agent pool and branch again, in case you want to queue the build in a different environment (agent) or use a different branch for the build.
The build runs. All builds are listed under Pipelines > Builds.
Click the build to look at the list of tasks executed. Notice that the Maven task took 1 minute 31 seconds to run. Azure DevOps also automatically executes tasks such as Prepare Job, Initialize job, Checkout, and Finalize Job.
Click any task to view the logs related to that task. The logs for the Maven task are shown below:
In the next section, you will create a task for the Nexus Artifact and schedule the task to run at required intervals. In this case, the task is run manually.
Log in to TAC, and ensure the TAC configuration uses the correct Nexus. Navigate to Job Conductor, then create a new Artifact Task using the artifact you deployed in Nexus (Step 9 in the Configuring the Maven task to deploy the Job to Nexus Artifact repository section).
Run the task.
This article explains how to create an Azure DevOps pipeline as part of the CI process to deploy Talend artifact to Nexus. Using Azure DevOps pipeline is beneficial for use cases that are implemented with Azure services; it easy to build a Talend Job for scenarios that require consuming or publishing data to different services in Microsoft Azure and to use Azure DevOps pipeline to automate the build process.