Migrating a Talend ESB product from 6.x to 7.0.1

Overview

This article explains how to migrate a Talend ESB product from a 6.x source version to a 7.0.1 target version.

 

Prerequisites

 

New and removed features

  • Read the Release Notes for the list of new, removed, and deprecated features, in the Talend Help Center documentation, ESB: new features page.

  • Publisher in TAC is deprecated and not supported with Nexus 3, instead use the Maven based CI approach.

  • Read the Software Development Life Cycle (SDLC) concepts, in the Talend Help Center documentation, Talend Development life cycle page.

 

System requirements

Verify the appropriate hardware and software sizing:

 

Preparation

  1. Download your license key.

  2. Download the software packages from your license email.

 

Key points to upgrade to ESB 7.0.1

From an ESB product perspective, the steps to upgrade to a new version include:

  1. Install new software

  2. Upgrade the TAC database (optional)

  3. Code migration

  4. Configuration

  5. Post migration verification

 

Installation

 

Installing new software

  1. Install the required Talend Services on the target servers using Talend Installer. For more information, see Installing your Talend Data Services Platform using Talend Installer.

  2. The following table provides a high-level overview of the Talend Services available in an ESB product for versions 6.x and 7.0.1.

    Talend Services in 6.x

    Source system example: 6.5

    Target system 7.0.1

    Studio / CommandLine

    Talend ESB

    ✔ (Karaf 4.1.3, Camel 2.17.6, CXF 3.1.14 )

    ✔ (Karaf 4.1.5, Camel 2.20.1, CXF 3.2.4)

    Talend Administration Center (TAC)

    Nexus 2 ✔ (v2.14.3) ✔ (Nexus 3 default *)

    Identity and Access Management (IAM)

    Talend RFC server

    Talend Log server

    ✔(Elasticsearch v2.4.0,

    Logstash v2.4.0,

    Kibana 4.6.1)

    ✔(Elasticsearch v6.1.2,

    Logstash v6.1.2,

    Kibana v6.1.2 + Filebeat v6.1.2 *)

    Talend CI Builder

    ✔ (CI not supported)

    ✔ (CI - Tech preview *)

    Note: * indicates a newly added services

 

Updating Studio

 

Installing additional packages

  1. Install the required third-party libraries.

    Starting with version 6.3.1, cMessageEndpoint Libraries are not shipped by default with Studio.

  2. In graphical mode, install cMessageEndpoint Libraries as a Studio Additional feature in online or offline mode.

    Required_Third_party_Libraries.pngInstall cMessageEnpoint and other 3rd party libraries

     

  3. For installation in non-GUI mode, download the cMessageEndpoint Libraries from your license email and execute the steps described in the Studio/esb/Readme_cMessageEndpoint_Libraries.txt file.

 

Installing patches

Contact Talend Support for the latest patches.

 

Upgrading the TAC database (optional)

You might want to upgrade the TAC database in the source system to the target version for migrating JobConductor, ESB Conductor, and Execution tasks, for more information, see Upgrading the administration database.

 

Migrating code

 

Prerequisites

  1. Ensure that the SVN/GIT repository is properly backed up before starting the Code Migration.
  2. Retrieve the URL of the SVN/GIT projects from the trunk.

 

Configuring TAC

  1. Connect to TAC in the target environment.

  2. Create a project in TAC, for SVN/GIT URL enter the repository URL.

    GIT_project.pngUpgrade existing remote project in TAC

     

  3. Grant authorized Users access to the project.

    GIT_PROJ_AUTH.pngGrant access to authorized users

     

 

Configuring Studio

  1. Launch Studio.

  2. Click Manage Connections.

  3. Specify the TAC URL and connect to the remote project.

    laucnh.pngConnect to TAC

     

  4. When the Data Services, Jobs and Routes are opened in the ESB Studio, the components that need changes are upgraded automatically.

 

Checking for errors in Jobs and Routes

  1. In Studio, open Jobs and Routes.

  2. Check for any compilation errors from the Code tab, or from the Problems view by navigating to Window > Show view > Talend > Problems.

    problems.pngVerify errors in Jobs and Routes

     

  3. Fix any errors and run the Job or Route.

  4. Build the Jobs and publish them to Nexus.

 

Resolving common issues in Jobs or Routes

 

Problem:

  • External libraries for Jobs, Routes, Beans, or Routines were not imported to the target environment

Solution:

  • Verify that the JAR needs an upgrade of version. For example, an upgrade of the database requires a newer version of the driver

  • Add the JAR using a tLibraryLoad or cConfig component as an External module

Problem:

  • Routes, Beans, or Routines have an older version of Camel or CXF dependencies

Solution:

  • After migration the version of new Camel or CXF JARs is 2.20.1 and 3.2.4 respectively

  • Most of the Camel or CXF JARs should be present after Studio updates

  • Update the Routes or Beans to use the new version of the JARs

  • Add JARs from the Inner modules drop-down list in the cConfig or tLibraryLoad component

 

Uploading JARs using the cConfig component

If you are connected to a remote project in Studio and added external libraries using the cConfig component, when saving the Route, the external libraries are uploaded to the Talend Artifact Repository configured in TAC.

2018-09-13_223241.jpgSetting Version in cConfig component

 

The libraries with the -SNAPSHOT suffix in the version number are published to the Snapshot repository. Otherwise the libraries are published to the Release repository.

 

Note: By default the cConfig component dynamically populates the Version field. However, with Talend 7.0.1, it is recommended to use a static version 6.0.0-SNAPSHOT / 6.0.0 if the same JAR file is also required as a dependency in a DI Job, added using tLibraryLoad, or added as a dependency in a Routine or Bean.

 

Configuration

After the installation tasks are completed in the target environment, as a prerequisite the following configuration and verification steps must be made.

 

Verifying and updating the TAC configuration

  1. Verify the configuration parameters in TAC are updated to the target host and ports.

  2. Verify the status is Green for CommandLine, SVN/GIT, Nexus, and other ESB services.

  3. Update the parameters if TAC is not configured with Nexus, SVN/GIT, ESB service endpoints.

 

Updating the ESB configuration

  1. Update the etc/org.ops4j.pax.url.mvn.cfg file with the Nexus Snapshot and Release repository URLs.

  2. If your ESB needs to be secured with SSL, update the configuration in the etc/org.ops4j.pax.web.cfg file with the valid certificates.

  3. If you use any custom JAAS users, add the users in the etc/users.properties file, this is optional.

  4. Configure Event logging to connect to ElasticSearch in the etc/org.talend.eventlogging.server.cfg file, this is optional.

 

Updating the Syncope configuration (optional)

After the installation of IAM in the target system, it may be necessary to migrate the Syncope configuration - Users and Groups from the source system.

Note: Skip this section if you are not using SyncopeLoginModule in Runtime.

 

Syncope 1.2 Syncope 2.0.2 (merged into IAM)

Talend Release 5.x to 6.3

Talend Release 6.4 to 7.0.1

 

Migrating from Syncope 1.x to Syncope 2.x

  • Syncope 2.x contains several new features and enhancements.

  • It is not possible to update an existing Syncope 1.x deployment.

  • Export the entire configuration - Users and Groups, to an XML file and import it in to the target environment.

  • Follow the detailed migration steps in section 4.7. Upgrade from Apache Syncope 2.0 of the Apache Syncope - Reference Guide.

 

Migrating from the Syncope 2.x configuration

  • Export the Syncope configuration from the source system.

  • The configuration in Syncope can be exported to a configuration file, for example: context.xml.

    curl -X GET -u username:password -o content.xml protocol://host:port/syncope/rest/configurations/stream

 

Importing the Syncope configuration in to the target system

Once you obtain the context.xml you can import the configuration in to the target system using syncopeadm CLI.

syncopeadm migrate --conf \path\to\content.xml \dest\path\to\MasterContent.xml

 

Post migration verification

 

Deploying Jobs and Routes

  1. Build the Jobs and them publish to Nexus.

  2. Add a new ESB server in the Servers section of TAC.

  3. From ESBConductor deploy the latest published task to ESB.

  4. Verify that the task state is DEPLOYED.

  5. For any deployment issues check the ESB and TAC logs.

Version history
Revision #:
107 of 107
Last update:
‎02-24-2019 11:15 PM
Updated by:
 
Labels (6)