Release-/Deployment workflow

Five Stars

Release-/Deployment workflow


I have some questions about Release/Deployment handling with Talend. We use either Talend Open Studio or Data Services Platform (so solutions for open source and subscription versions are welcome).

Our workflow:

We have Releases. Each release is developed and then put through different test systems (integration, staging) before it goes in production. (develop->integration->staging->production).
For each feature we increase the version of all the Jobs that need to be changed to implement the feature. So a Job can have no, one or multiple new versions in a release.

With this workflow I encounter some problems:

- Hotfix:
Example: Version 1.1 is in production, but in development we have version 1.3 (not ready for production).

I then have to duplicate the Job in Version 1.1 and implement the hotfix and deploy this new Job in production. Additional I have to implement the hotfix in version 1.3. When Version 1.3 is ready for production i have to switch the Job back and have to remove the duplicated version.

Is there a better way?

- Managing tRunJobs:
We're using tRunJobs to call other Jobs (with fixed versions). In some parts we have 4-5 levels of them. So if i change a job on the last level (with new version) i also have to update all the Jobs that call that Job to update the called version.

Example: JobA calls JobB. JobB calls JobC. JobC calls JobD. If i change JobD to a new version i have to change A,B and C as well to call the new version of JobD.
For Subscription-Version: Is the usage of "Execution Plan" better in this case? So i only have to update the version of the changed job and nothing more? Is it possibly to input values with Execution Plan like we do it with tRunJob?

- Remove features from a release:
For example we have 3 features in a single job for one release:
FeatureA: Version 1.3
FeatureB: Version 1.4
FeatureC: Version 1.5
Sometimes i have to remove FeatureB from the release. The only way is to remove the changes made in 1.4 in 1.5 but that can get messy. Also if the feature should be later included in the next release i have to implement it again.

Help and best practices for these things are very welcome Smiley Happy




Re: Release-/Deployment workflow

In the commercial Talend version you have the ability to back your projects by source control and this also manage the deploy version of your artefacts (e.g. the Maven or Nexus version, typically Major.Minor.Point). The deploy version is different from the Job Design version (M.n).


Thus, to address the issues you outlined we suggest that you:

  • use an SCM, e.g. Git
  • use branches to represent your current development as well as you maintenance branch for production
  • say you have the 1.0.x branch for all things in the past
  • you also have the current development branch (e.g. head - will become 2.0.x branch upon release)
  • at release time you tag the 2.0.x branch with version 2.0.0
  • then at some point you have to do your bug fix on branch 2.0.x to create version 2.0.1

Above always revers to the deploy version.


As for the job design version you can actually leave it at 0.1 (the default) and then don't change it at all. Doing so keeps your job-subjob or job-metadata relationships intact and you won't run into the issues you described with "copying back" and old version.




Thomas Steinborn
VP Product Management

What’s New for Talend Spring ’19

Watch the recorded webinar!

Watch Now


Introduction to Talend Open Studio for Data Integration.


Downloads and Trials

Test drive Talend's enterprise products.


Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.