Best practices for adding custom libraries in Talend Jobs, routines, beans, and routes

Overview

Talend Studio provides a new approach to adding custom and third-party JAR files. When a Studio user adds a custom library into a Job, the library is uploaded to the Nexus shared library repositories. Syncing custom libraries to Nexus enables collaboration between all of the users on the project by enabling sharing and automatic downloading of the libraries to their local m2 repositories. It allows users to run Jobs easily and to add libraries into new Studio Jobs, routines, beans, and routes.

 

This article explains the best practices that Talend suggests you follow when adding custom / third-party libraries into Jobs, routines, routes, and beans.

 

Sources for the project are attached to this article.

 

Prerequisites

  1. Install Talend 7.0, 7.1, or 7.2.

  2. Configure TAC and Nexus settings for shared libraries (follow the instructions in the Custom libraries process KB article).

  3. Create a Project in TAC (this article uses, LibraryManagement).

  4. Create two users in TAC (this article uses, userA@talend.com and userB@talend.com).

  5. Authorize users to have Read/Write permissions in TAC.

     

Process

The following diagram depicts how the upload, sync, and sharing happens:

How_it_works.jpg

 

Creating a custom JAR file

Create a simple Maven project as a JAR module.

  1. In Talend Studio, click the Open Perspective icon, then select Java

    java_perspective.jpg

     

  2. Create a Java Maven project called LibraryManagement.

  3. Create a Maven module called Greet.

  4. Create a Java file called Greet.java that returns a Welcome message.

    Note: in this step, you create a custom JAR file to use in Talend Jobs / Routes.

    java_project.jpg

     

Verifying settings for User Libraries

  1. Launch TAC.

  2. Navigate to Configuration > User Libraries.

  3. Verify that your Nexus repository details are configured correctly.

    tac_config.jpg

     

Configuring Studio as userA

This section shows you (logged in as userA) how to create a Job, routine, bean, and a route, and add a custom library as a dependency.

 

Add a library in a standard DI Job

  1. Create a standard Data Integration (DI) Job called Job_LibraryManagement.

  2. In the Designer, add a tLibraryLoad, component. Click the [...] button to the right of the Library field.

  3. In the popup window, select the Artifact repository(local m2/nexus) radio button, then select the Install a new module radio button. Click the [...] button on the right, then add the new Greet-0.0.1-SNAPSHOT.jar module.

    Note: There is a warning message, "Please detect if the module already installed with maven URI".

    Di_job_dependency_add.jpg

     

  4. Click the Detect the module install status button.

  5. Because the module Greet-0.0.1-SNAPSHOT.jar doesn't exist in the Nexus repository, you can upload this version, and the warning message disappears automatically.

    install_new_module.jpg

    Note:

    • Observe the Default MVN URI field, the new module Greet-0.0.1-SNAPSHOT.jar is uploaded to the 6.0.0-SNAPSHOT folder of the nexus repository talend-custom-libs-snapshot.

    • Similarly, if you want to upload the release version of the Greet-0.0.1.jar module to the talend-custom-libs-release repository, create a release version of the Greet-0.0.1.jar module, select the Custom MVN URI check box, then, in the Custom MVN URI field add the following:

      mvn:org.talend.libraries/Greet-0.0.1/6.0.0/jar
  6. Click OK to upload the JAR to your Nexus repository.

    snapshot_repo_jar_added.jpg

     

  7. In the Designer, add a tPrejob component, then link it to the tLibraryLoad component using the OnComponentOk trigger.

    step7.jpg

     

  8. Add a tJava component, initialize the Greet class, then, in the Code window add a system.out to display the Greet message by adding the following:

    org.talend.demo.customlibrary.Greet job_greetings = new org.talend.demo.customlibrary.Greet();
    System.out.println(job_greetings.greet());

    tjava_settings.jpg

     

  9. Run the Job, then view the Job execution details.

    Job_execution.jpg

     

  10. The Job executes the code from the Greet-0.0.1-SNAPSHOT.jar library and logs the Welcome to Talend. You are running version 1 of this library message.

     

Add a library in a routine

  1. Create a routine called Routine_Greet.

  2. Right-click Routine_Greet, then select Edit Routine Libraries.

  3. In the Import External Library window, select New.

    Add_dependency.jpg

     

  4. Another pop-up window appears, select Greet-0.0.1-SNAPSHOT.jar on the Platform drop-down list. Click OK.

    Note: the Greet-0.0.1-SNAPSHOT.jar library already appears in the Platform drop-down list because it was added to the project in the Add a library in Standard DI Job section of this article.

    Platform_libraries.jpg

     

  5. The library is added to the Routine, notice that the Required check box is enabled by default. Click Finish.

    Mark_as_Required.jpg

    Note:

    • Enable the Required check box, if the Greet library is not already added to the Job design using a tLibraryLoad component or with any other Talend components. You can verify the external libraries used by the Talend components from the Studio Module view.

    • If the check box is disabled, it means the Greet library is already added to the Job and is used only as a compile-time dependency in the Routine.

  6. Modify the Routine_Greet definition, update the helloExample method, then instantiate the Greet class.

    Routine_greet.jpg

    Routine_Greet is now set to handle external dependencies.

  7. Create a Job called Routine_LibraryManagement, right-click the Job, then select Setup routine dependencies.

    routine_job.jpg

     

  8. Add a User routine as a dependency to the Routine_LibraryManagement Job, by following the steps, numbered in sequence, below:

    AddRoutineDependencies_2.jpg

     

  9. Open the Routine_LibraryManagement Job, and add a tFixedFlowInput and tLogRow component.

  10. Double-click the tFixedFlowInput, component to open its Basic setting view, edit the schema and in the Value column, add the string column message, Routine_Greet.helloExample(). The Job is now configured to use a routine that uses a custom library.

  11. Run the Job and view the Job execution details.

    Job_execution.jpg

    The routine executes the code in the Greet-0.0.1-SNAPSHOT.jar library and logs the Welcome to Talend. You are running version 1 of this library message.

     

Increment the version of the custom library

This section shows you how to create a new version of the Greet-0.0.2-SNAPSHOT.jar file to add to ESB Jobs.

  1. In the Studio, go to the Java perspective.

  2. Open the LibraryManagement project. In the pom.xml file, increment the version of the custom library to 0.0.2-SNAPSHOT.

  3. Run Maven install and generate the Greet-0.0.2-SNAPSHOT.jar file.

    version-2-osgi.jpg

     

Add a library in OSGi routes

  1. Create a new route called Route_LibraryManagment, then add a cConfig component.

  2. Double-click the cConfig component to open its Component view. In the Dependencies table, click the [...] button to the right of "newLine".

  3. In the Select Module window, select the External modules radio button.

    External_module_1.jpg

     

  4. Add the new version of the external Greet-0.0.2-SNAPSHOT.jar library, notice that by default, the version field is always set to 0.0.1-SNAPSHOT.

    default-0.0.1.jpg

     

  5. As a best practice, when uploading SNAPSHOT artifacts, change the version to 6.0.0-SNAPSHOT.

    Note: when uploading a RELEASE artifact to the talend-custom-libs-release repository, set the Version field to 6.0.0.

    version-6.0.0.jpg

     

  6. Notice that the JAR is added to the cConfig component.

    External_module_2.jpg

     

  7. Verify that the Greet-0.0.2-SNAPSHOT.jar is uploaded to the Nexus repository.

    Nexus_artifact.jpg

    Note: Observe that the Greet library uploaded to Nexus under the 6.0.0-SNAPSHOT folder using the tLibraryLoad and cConfig components, respectively.

     

  8. Open the Route_LibraryManagment route, click the Dependencies tab, then select the Bundle ClassPath check box.

    dependencies.jpg

     

  9. In the Route_LibraryManagment route, click the Designer tab, then add a cTimer and cProcessor component.

  10. Open the cProcessor component, and add the following:

    Import

    import org.talend.demo.customlibrary.Greet;

    Code

    Greet greetObj = new Greet();
    String message = greetObj.greet();
    System.out.println(message);

    Route_design.jpg

     

  11. Run the route and view the execution details.

    Route_exec.jpg

    The route executes the code in the Greet-0.0.2-SNAPSHOT.jar library and logs the Welcome to Talend. You are running version 2 of this library message.

     

Add a library in beans

  1. Create a new route called Route_BeanLibraryManagment.

  2. Create a new bean called Bean_LibraryManagement, right-click the bean, then click Edit Bean Libraries.

  3. In the Import External Library window, click the New button.

  4. A popup window opens, select the external Greet-0.0.2-SNAPSHOT.jar library from the Platform drop-down menu, then click OK.

    Bean_editBeanlibraries.jpg

     

  5. The library is added to the bean, select the Required button to include the library in the route build. Click Finish.

    Bean_added_required.jpg

     

  6. Open the Route_BeanLibraryManagment route, then add the cBeanRegister, cTimer, cSetBody, and cLogRow components.

  7. Double-click the cBeanRegister component to open its Basic settings view, set Id to "greetBean", then, in the Code window, add the following:

    beans.Bean_LibraryManagement greeting_from_bean = new beans.Bean_LibraryManagement();
    beanInstance = greeting_from_bean;

    bean_Component.jpg

     

  8. Double-click the cSetBody component to open its Basic settings view, select Simple in the Language pull-down list, then in the Expression field, add the following:

    "${bean:greetBean.helloExample}"

    body_settings.jpg

     

  9. Connect the cTimer, cSetBody, and cLogRow components.

  10. Run the route and view the execution details.

    Bean_execution.jpg

    The route executes the code both in the bean and Greet-0.0.2-SNAPSHOT.jar library and logs the Welcome to Talend. You are running version 2 of this library message.

     

Using beans with multiple versions of custom libraries

 

Feature not supported

When you build a route, all of the beans are built and added to the route as a dependency. Because all the beans share the same POM file adding multiple versions of the same external library doesn't guarantee the desired results in both Studio and OSGi environments.

 

The following example demonstrates the impact of using several versions of the same library in beans.

  1. In the Repository, duplicate the route Route_BeanLibraryManagment, then create two new routes Route_A, Route_B.

  2. In the Repository, duplicate the bean Bean_LibraryManagement, then create two new beans Bean_A, Bean_B.

    duplicate.jpg

     

  3. Right-click Bean_A, click Edit Bean dependencies, then modify the bean to use the Greet-0.0.1-SNAPSHOT.jar library. Click Finish.

    Bean_A_dependency_change.jpg

     

  4. Right-click Bean_B, click Edit Bean dependencies, then verify that the bean is using the Greet-0.0.2-SNAPSHOT.jar library. Click Finish.

    Bean_B_dependency_change.jpg

     

  5. Open Route_A, click the cBeanRegister component to open its Basic settings view, then replace the contents in the Code window with the following:

    beans.Bean_A greeting_from_bean = new beans.Bean_A();
    beanInstance = greeting_from_bean;

    bean_register_Bean_A.jpg

     

  6. Open Route_B, click the cBeanRegister component to open its Basic settings view, then replace the contents in the Code window with the following:

    beans.Bean_B greeting_from_bean = new beans.Bean_B();
    beanInstance = greeting_from_bean;

    bean_register_Bean_B.jpg

     

  7. Run Route_A and view the execution details, observe a message is logged from the class in the Greet-0.0.1-SNAPSHOT.jar library.

    exec_bean_a.jpg

     

  8. Run Route_B and view the execution details, observe again the same message is logged from the class in the Greet-0.0.1-SNAPSHOT.jar library.

    exec_bean_b.jpg

     

  9. This explains why using multiple versions of custom libraries is not supported with beans.

     

Running Jobs as userB

In this section, you need to log in to Talend Studio as userB and open the Jobs that were created by userA.

 

Download the external libraries to local m2 repository

  1. Open the Standard Job Job_LibraryManagement.

  2. Notice there is an error on the tLibraryLoad component, click the component to open its Basic settings view.

    standard_job_jar_download.jpg

     

  3. Click the Install button to automatically download the library from Nexus repository.

  4. Run the Job without any modifications.

     

Conclusion

This article covered the best practices for adding a custom library into Jobs, routines, routes, and beans, how the libraries are synced to the Talend Artifact Repository, and how the libraries are shared by other project users.

Version history
Revision #:
26 of 26
Last update:
‎10-28-2019 08:52 AM
Updated by:
 
Contributors