A Github Action for Talend CI/CD!

As you may already know, you can get the best of Talend by using our continuous integration and deployment capabilities. While I already have covered how simple it is to integrate Talend with CI/CD tool in the cloud and illustrated it with an example on Azure DevOps, we can still go further. Last year GitHub unveiled its brand-new CI/CD service called GitHub Actions with an innovative approach.

 

In this blog post I would like to introduce the first GitHub Action for Talend!

 

What is GitHub Actions?

GitHub, at its core, is a code hosting platform for collaboration and version control. However, GitHub now offers much broader services for developer. Last year GitHub introduced GitHub Actions as their new workflow service. In short it allows you to automate all your workflows with built-in CI/CD.

 

The approach they chose is very innovating and interesting. A GitHub Actions workflow is obviously made of “Actions”. Each action is a piece of code hosted in a GitHub repository. You have two types of actions. The most flexible type of actions is container image definitions. It allows you to use any languages or dependencies you may need in your workflow. The second one is javascript-only actions that are faster to execute but requires javascript knowledge.

 

For me GitHub Actions’ strength lies in the openness and community behind GitHub. This open ecosystem is going to drive innovation and democratize automation for developers. Everyone can build actions by pushing some code to a repository as well as use actions already developed by others.

 

As far as I can tell, it was very easy to create my first action and the number of actions is growing every day. Check out their marketplace.

 

What's in it for Talend?

A lot of customers at Talend use GitHub to store and version control their artifact sources such as data integration and big data jobs or microservices for instance. Then you can publish and deploy your artifacts as Java binaries or Docker containers. Using CI/CD becomes relevant when you want to automate the build, test and deployment processes along the way. As Talend supports Maven, every CI/CD tool supporting Maven themselves can be leveraged with Talend. Therefore, it’s only a matter of time before we see requests from Talend users to take advantage of GitHub Actions. Consequently, we built the first GitHub Action for Talend to publish to Cloud.

 

Remark: You don’t necessarily need to use this packaged action to use Talend CI/CD capabilities. If you desire more flexibility you can still use regular Maven GitHub Actions and start from there.

 

How does it work?

 

The starting point here is obviously GitHub. Once you have your Talend job sources synced with a GitHub repository you can create your first GitHub Actions by clicking on the “Actions” tab.

actions_tab.png

 

This is where you can create your first workflow. Click on “New workflow” which leads you to a home page. Click on “Set up a workflow yourself’:

set_up_workflow.png

 

A workflow is defined by a description file under .github/workflow/ folder by default in your repository. Feel free to rename this file as you see fit. The description file is done in YAML which is a human friendly data serialization language. I personally like it for its clear and concise format.

 

The core of the page you see is where your YAML stands and describes your workflow. On the right you have a menu to insert actions.

 

You can also search for “talend” in the search bar. You’ll be able to select “Talend Publish to Cloud Action”. A brief documentation shows you the different parameters you can use. Let’s start by a simple example:

name: Talend Jobs Publish to Cloud

on: [push]

jobs:
  talend_publish_to_cloud:
    runs-on: ubuntu-latest
    name: An action to publish Talend jobs to Cloud
    steps:
    - name: 'Checkout Github Action'
      uses: actions/checkout@master
      
    - name: 'Publish to Talend Cloud'
      uses: Talend/publish-to-cloud@v0.1
      with:
        args: -e -am -X
        project: CIPROJECT
        updatesite_path: http:///jenkins/userContent/
        service_url: https://tmc.us.cloud.talend.com/inventory/
        cloud_token: ${{ secrets.cloud_token }}
        nexus_url: http:///nexus
        nexus_username: ${{ secrets.nexus_username }}
        nexus_password: ${{ secrets.nexus_password }}
        license_path: license.gpg
        license_passphrase: ${{ secrets.LARGE_SECRET_PASSPHRASE }}

As you can see, this pipeline will be triggered at each commit pushed to the repository. The workflow above entails two steps.

  1. A native GitHub action checks out the repository.
  2. Using GitHub Action for Talend to Publish to Cloud.

Let’s dig in the parameters I used in this workflow. In my GitHub repository my Talend project is called “CIPROJECT”. I have hosted Talend updatesite on a HTTP server publicly available as well as a Nexus instance accessible with a user and password. Eventually, if you plan to publish in Talend Cloud in US region you can let the service_url parameter as it is. If you use a different region, please change it accordingly to your instance. The license needs to be encrypted in your repository for Talend software to run.

 

Please, never publish your license without encryption in your GitHub repository.

 

To encrypt your license, follow the procedure detailed in the marketplace page for Talend Publish to Cloud Action. It’s following GitHub Actions best practices as stated here.

 

Once your pipeline description is ready, commit your YAML file in your repository. It should start a workflow execution as the workflow is triggered at each commit.

 

Go back to the Actions tab and wait for your CI/CD workflow to terminate:

workflow.png

If everything went right, you should get a Build Success at the Talend Publish to Cloud step. If you get an error don’t panic! You may have some issues in your setup and configuration. Don’t hesitate to ask your questions on our community forum.

 

Conclusion

You now have at your disposal a new weapon to get benefits from Talend CI/CD capabilities. Thanks to Talend’s git and Maven support, you can now take full advantage of GitHub services including its latest GitHub Actions service. GitHub Action for Talend Publish to Cloud ease your Maven setup to publish your jobs to Talend Cloud.

Version history
Revision #:
2 of 2
Last update:
‎03-19-2020 06:07 PM
Updated by:
 
Labels (3)
Contributors
Comments
Six Stars

Hi Team,

 

Thanks much for the post!

 

Can you please share guidelines or steps to implement CI/CD for the non cloud platforms?! 

We can see many post related to this but we need a proper step by step process to implement the same in on prim, we are using the Talend Big data platform 7.2.1. 

 

Thanks 

Tamil Mynthan

 

Employee

Hello @tamil19mynthan,

 

Steps to implement CI/CD for non cloud platforms have been heavily documented in our Knowledge Base, look for continuous integration and/or jenkins keywords. There is also an entire section in our documentation explaining how to achieve this on-prem.

 

Thibaut