Talend Integration Cloud (TIC) is a fully-managed cloud option. It provides the great data integration, data quality, and Big Data features from Talend in a cloud environment that is managed, monitored, maintained, and secured by Talend. This article explains the different architecture models supported by Talend Integration Cloud.
SaaS & Cloud Platforms
This is the first kind of architecture, where Talend Studio is installed locally behind a firewall. Once the design activities are completed, the Jobs are published to the cloud. All the data routing between Studio and the cloud is performed over HTTPS and is very secure.
In this architecture, the integration flows and templates are run in the cloud. The Runtime engines are also in the cloud.
This is the second type of architecture, which is very similar to the SaaS and cloud architecture. However, in this architecture, the runtime engine is behind the firewall - hence it is called a “remote engine”. So here Talend Studio, the remote engine, the apps, and the databases are behind the firewall.
In this architecture, the integration flows and templates are run in this remote engine, and when it comes to the runtime engines that actually perform the execution of these integration workflows, those are in the cloud. Here again the remote engine sends the status and the logs using HTTPS, and also leverages the on-prem apps and databases.
Virtual Private Cloud (VPC)
This is the third type of architecture, similar to the previous architecture with respect to Studio and cloud installation. The different is that the remote engine, the runtime engines, the on-prem apps, and the databases all lie within a virtual private cloud.
Here you could run the integration flows and templates in the remote engine in a VPC. The runtime engines would also be in the VPC.
Virtual Private Cloud - with Log Redirect
This architecture is an extension of VPC where the logs are redirected to a file and/or log services. The remote engine flows back only the status or approved log data to the Integration Cloud Service; the rest of the logs are redirected to a file on the remote engine or other log service within the virtual private cloud.
In this architecture, you could run the integration flows and templates in the remote engine in the VPC, leveraging both public and private sources.
Note: In all of these architectures, an additional capability is that you could leverage is the remote run/debug capability from Studio directly to the remote engine. The remote engine contains a JobServer, so if you need to do a remote run/debug from Studio to the remote engine, you need to configure the remote JobServer details in the Studio preferences to point to the remote engine.
For more information on configuring this capability, see the Remote run configuration (Talend > Run/Debug) page of the Talend Big Data Platform Studio User Guide.
... View more
hi @penmetsasravya , Could you please tell which version and component you are using. tSoap component does not have such feature. Please refer the link given below for the details on the component. https://help.talend.com/reader/v9wFTrfVYq8FM3TePctOTg/oIBkzFroAQwpoqPY4916RQ please elaborate on your requirement. Thanks, RekhaSree
... View more
hi, The component tFileInputMail does work for files without attachment. The link to the KB article also shows an example which extracts all types of attachments. There might be some other issue with the attachment. could you please share your job screen shot and the attachment you are trying to extract. Thanks, RekhaSree
... View more
Hi Saurabh, to use connections, create the custom component with snowflake type. I have done as follows and i am able to connect to snowflake. Studio : TIC connection : Tic Flow : the connection is displayed in the flow. Thanks, RekhaSree
... View more
Hi Saurabh, TIC uses very strict naming conventions for all the parameters. The syntax for native connections must start with 'Connection_<application name>_<parameter_name>'. You might want to use this only if you are using the native connectors provided by TIC. As snowflake connection are not yet part of the native connectors, you need to use user defined parameters. the syntax for user defined parameters is 'Parameter_<parameter name>. See the link given below for details. https://help.talend.com/reader/Mav4Vp0qD5Scvag1kN90pA/mlNjwb5d5bGkY9emOuyixg Regards, RekhaSree
... View more
This article explains how to set up Continuous Integration, Continuous Testing, and Continuous Deployment using Talend CI-Builder. It covers the required preparation in your environment before installation, the configuration and file system folder structures suggested by Talend, and the steps to automate the build process using Jenkins and Maven.
Note: You can use any CI tool that supports Maven. This article shows how to use Jenkins.
Note: The Preparing your environment and Installation and Configuration sections of this article are appropriate for Talend 6.4 users. For Talend 6.4 CI Process, Build, Testing, and Deployment information, see Talend Software Development Life Cycle Best Practices Guide.
Preparing your environment
The following Talend environment must be installed. For details, see these pages on the Talend Help Center.
Talend Administration Center
Set up with a dedicated command line and port (minimum version 6.1.1). For more information, see Installing and configuring Talend Administration Center.
See Installing your Talend Studio.
The Nexus repository is configured, the server is accessible from the CI server, and the interaction between Nexus, TAC, and Studio is fully working. Any HTTP or HTTPS proxies that are present in the environment must be accounted for, and you should ensure that Talend environment communications are not going over a proxy unless it is necessary. See Installing and configuring the Nexus artifact repository under Installing and Configuring Talend server modules.
Subversion or Gitlab
SVN or GIT must be integrated with TAC and Studio, and must be tested and fully functional.
Projects, reference projects, and project authorizations in TAC are set up, with users having appropriate access/authorization.
Jobs containing test cases for Continuous Testing. To familiarize yourself with the test cases, review these videos:
Continuous Integration Part 1 – Creating Test Cases in Studio
Continuous Integration Part 2 – Automating the Build Process
Continuous Integration Part 3 – Deploying Your Code
Minimum System Requirements
Talend Administration Center + Talend Activity Monitoring Console Web application
Linux, Windows, MacOS
Minimum Temporary Desk Space Requirements
Windows or UNIX
If the TAC server has internet access, and the software update configuration is correct in TAC, you can automatically get the patch from Talend on the software update page. Once you choose to download the patch, it should be made available for all Studios and Command Line applications.
For more information, see Configuring the Software Update repository in Talend Administration Center.
If the TAC server has no internet access, follow the instructions in Uploading a patch in the local Nexus to manually download the patch and upload it to your local Nexus.
File System Folder Structure
Having a dedicated CI server is recommended. To run CI in the server, it will need dedicated storage space, so create the folder structure as shown below:
commandline_workspace: allows Jenkins, CommandLine, and CI Builder to utilize and share information. If you have CommandLine currently installed, you can use the existing CommandLine folder and copy it over instead of creating a new commandline_workspace.
JENKINS_HOME: stores the Jenkins configuration (system and projects). Jenkins_Home is also the name of the environment variable.
maven_repo: this folder is optional. Create it only if you need Maven to use the CommandLine Maven artifact repository.
target: stores a copy of the Java code, compiled artifacts, and test results.
To prevent file locking and permissions issues, all components should be run using the same user, for example talenduser on UNIX or Local System Account on Windows. If you choose to change the default users, be very precise with file system permissions for the folders, including their children.
Installation and Configuration
This process covers setup on Windows Server 2008, but most of the steps are identical in a UNIX-based machine. This assumes that the server is properly configured with a Java 8 JDK, the environment connectivity is working correctly, and your JAVA_HOME environment variable is set.
Using the Talend installer, install CommandLine and its Server Service to wherever you will later install Jenkins and Maven. Alternatively, you can use an existing CommandLine install, or turn on Studio's CommandLine, instead of installing a new copy.
Note: The installer may attempt to start the Windows service for CommandLine on completion.
Start CommandLine and install the patch as described under Patch Installation above.
Stop the CommandLine service.
Browse to your CommandLineHome/TalendServices/conf directory:
Edit the wrapper.conf file and find the following section:
# genConfig: further Properties generated by genConfig
Alter the parameter wrapper.app.parameter.7 to point to your commandline_workspace folder:
When CommandLine is used with CI, it will not connect to TAC and does not find the location of the Nexus libraries repository (which by default is talend-custom-libs). Connecting to TAC is required to compile and build the Talend jobs, so to integrate Nexus, you must set the location of the Nexus libraries repository for CommandLine and add the other parameters listed below to the Talend_Root/CommandLineHome/studio/configuration/config.ini file.
Navigate to and open the config.ini file:
Add the following parameters to the file:
Note: If your user and password are the defaults shown above you don't need to include them in the file, though if you changed the defaults, specify the appropriate values.
Start the CommandLine service.
Important: Do not use this CommandLine for anything other than Continuous Integration. It is recommended that the configuration folder of CommandLine remains in its default location.
Continuous Integration CommandLine Server
This server is dedicated for CI use only, and the CI CommandLine cannot be used for normal CommandLine operations such as Audit, TAC publisher, TAC Job Conductor generation, Studio remote generation, and scripting.
Important: Verify that Studio is not being used during installation.
The CI Server will host the following components:
Jenkins: Any CI tool that can use Maven should be compatible
CI Builder plugin
Optional: This server can be a logical place to install SVN, GIT, and/or Nexus.
Example diagram from v6 Reference architecture 1.4
Maven – Windows Installation
Download Maven from Downloading Apache Maven site (binary zip archive).
Extract the downloaded zip file.
Option 1 - Set the location of the Maven .m2 repository
The default location of Maven is ~/.m2 (user home, .m2 directory). With this setup, Maven will use CommandLine’s .m2 repository, as it is the simplest way of ensuring that Maven has the correct Talend libraries.
Browse to MAVEN_ROOT_INSTALL_DIR/conf/settings.xml.
Add the following property:
Where the path specified is the path to the CommandLine .m2 repository:
Navigate to the <servers> section of the same file and add an entry for the Nexus credentials:
You can obfuscate the password if required. Review the instructions on the Maven site: Password Encryption.
Option 2 - Move the path of the local repository
To move the path of the local repository, you need to set the path in settings.xml.
Set up a <server> set of credentials for Nexus as described in Option 1. It is possible to have multiple profiles here if you need to use different users for accessing the libraries repository and the deployment repository:
Locate the <profiles> section and add your Talend libraries repository as a new profile:
The <id> tag should have the same <id> as the <server> section, with the correct credentials of your repository.
Add an <activeprofile> to the <activeprofiles> section:
The file should look like this:
Now Maven will be able to access the Talend Nexus for Talend libraries.
Continuous Integration Builder
Download Talend CI Builder as described in your Talend license email.
Unzip the downloaded file into your root CI folder. This creates ci.builder-version.jar and .pom.
Open a command prompt and browse to the directory where you installed Maven /bin.
Run the following command:
mvn install:install-file -Dfile=CI-Builder_install_dir\ci.builder-6.3.1.jar -DpomFile=CI-Builder_install_dir\ci.builder-6.3.1.pom
This will install CI Builder into Maven, and the supplied pom file will list the dependencies of CI Builder. It will then download these dependencies as part of the install process.
Depending on your setup, the machine will need internet access to download these libraries, or you may have configured a repository for Maven (using Option 2) that contains these libraries.
Create a new file in your CI Root folder called pom.xml, and copy the following contents into it.
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
A Project Object Model or POM is the fundamental unit of work in Maven. It is an XML file that contains information about the project and configuration details used by Maven to build the project. It contains default values for most projects. Some of the configuration that can be specified in the POM are the project dependencies, the plugins or goals that can be executed, the build profiles, and so on. Other information such as the project version, description, developers, mailing lists and such can also be specified. This pom file defines the goals that the first stage of the CI process will attempt to achieve when executed from Jenkins.
To segregate the CI Jobs, create a repository called releases_ci. The Nexus admin user was created to access this repository for deployment actions.
The recommended best practice is to set up suitable users and groups.
As an option for the artifacts to be generated by CI Builder and pushed to a Nexus repository, it is recommended to set up a unique repository for this purpose as shown below:
Jenkins - Windows Installation
Download the Jenkins Windows installer from the Jenkins site: https://jenkins.io.
Follow the installation wizard prompts:
Verify the Jenkins port does not conflict with any Talend ports.
Example is installed to E:\Jenkins.
Another option is to deploy it on Tomcat as a war file in the webapps directory.
Stop the Windows service if it started automatically.
By default, the Jenkins directory is located in ~/.jenkins under the user that is running the Jenkins home directory, and where all the configuration files are stored. If running it as a Windows service, it is located under the Windows system folders in Users/user/.jenkins.
To run the service as a different user, that same user would also need to run CommandLine.
You can set a different path for JENKINS_HOME by editing the file JENKINS_INSTALL_ROOT/Jenkins.xml.
Note: If this was deployed as a war file, you will not have a Jenkins.xml file.
Start the Jenkins service and check that it can connect to the web UI (for example, http://localhost:8380).
For more information about administering Jenkins, see the Administering Jenkins site.
Before continuing, you should have CommandLine, Jenkins and the CI Builder plugin installed into Maven to finish configuring Jenkins.
Navigate to Jenkins System settings
Click Manage Jenkins > Manage Plugins.
Install the Maven Integration Plugin, and either the Subversion Plugin or the Gitlab Plugin, and verify the plugins are enabled.
Click Manage Jenkins > Configure System.
The home directory should reflect how you configured it previously (while installing Jenkins).
In the Global Tool Configuration, under JDK installations, add the path to the Java 8 JDK installation on your CI Server.
Under Maven installations, add the path to the Maven installation on your CI server.
Save and close System Config.
The Process of Continuous Integration, Continuous Testing, and Continuous Deployment begins once a job is committed to the central repository.
The build process uses Jenkins and Maven to take the completed Job and generate Java source code. It also compiles the Job into a Java executable and then publishes it to Nexus. Once the Job is published to Nexus, it can be deployed into any environment in a self-contained executable file. The following sections explain the steps to automate this process using Jenkins Jobs. There are three Jenkins Jobs, as shown below, to automate the build process:
Building is the process of generating Java source code and Java executable binaries.
Create a new Job by clicking Jenkins > New Item and adding a new Maven project named BuildSources.
Configure the Source Code Management tab.
If using Git, provide the Git repository and other details. Configure the repository URL to point to a specific Tag, Branch, and Trunk.
If using SVN, configure the repository URL to point to a specific Tag, Branch, and Trunk on SVN.
Set the local module directory to the same as the project name (multiple branches of the same projects might require unique names).
This will allow Jenkins to perform an SVN checkout to commandline_workspace that replicates the file system structure that would be created if Studio or CommandLine performed a check out of the project.
Note: You will be prompted to enter credentials to communicate with SVN.
Examples of the folder structure comparing Studio to CI:
On the Build Triggers tab, specify the trigger action according to the project's needs. In this example, the build will trigger as soon as a change is pushed to Gitlab.
On the Build tab, set the build options:
Root POM: the pom you created earlier (see the CI Builder section).
Goals and options: in this scenario, you will use the options of Maven and CI Plugin for generate. It is possible to add a -X flag here to get debug information.
MAVEN_OPTS: these parameters are passed to CI Builder to command it to call CommandLine.
Workspace: the same workspace configured in the CommandLine section.
User: not a TAC user; the CI builder does not communicate with TAC. This is a ‘substitute’ user that allows CommandLine to log onto a local project.
Target directory: the target directory, where the results of this stage will be copied.
In addition to these parameters, it is possible to filter what CommandLine builds. Behind the scenes, CI builder is calling the CommandLine command buildProjectSources, which can take filter parameters:
The filter option is:
Example: Force CI Builder to build only specific jobs from a project, such as jobs in a subfolder called Integration:
Example: Build a specific job called testjob:
For more info on filters, use the CommandLine -help command.
Set the project to use your custom workspace (this assumes the commandline_workspace on the Jenkins machine will be used only for CI Builder).
Add a post-build step for this project to build the RunTests project, which doesn't exist yet. You will create this project in the Testing section below.
Note: There is no automatic check on this field, so if you enter a project that doesn't exist yet, be sure to create it with the same name and capitalization as what you enter here.
Save and close.
Testing the Configuration
To test your configuration, you will need to build the project to generate the Java source code from the Talend metadata. You can view the console log to monitor progress.
Click Jenkins > New Item, then add a new Maven project called RunTests.
Configure it as follows:
Note: The xDeploy project doesn't exist yet, but as before, you can add it as a post-build step now and create it later (in the Deploy section).
Run the project to test the configuration.
View your test results on the Test Result page.
The final step in Continuous Integration is deploying the compiled code to the repository.
For Jenkins to deploy artifacts to the releases_ci Nexus repository, you need to create another project. In this example, it is named xDeploy so it is alphabetized as the third project in the list (see the Process section). Only the Build fields need to be configured:
Root POM: use the same target pom.xml you generated earlier (see the CI Builder section).
Goals and options: the action is a built-in Maven goal that deploys to an artifact repository.
MAVEN_OPTS: set this parameter as the deployment repository.
Username and Password: The entries after talendNexus in the screenshot below refer to the set of credentials for Nexus defined in the Maven settings.xml file (see the Maven - Option 1 section).
Note: The version number of the artifact that is published to Nexus will be the Talend version number (that is, the one next to each Job in the Studio). Currently there is no option to control artifact version numbers independently of this version number.
This article introduced Continuous Integration, Continuous Testing, and Continuous Deployment using Talend CI-Builder, and explained the steps required for you to successfully configure and deploy them.
... View more