As part of our project we have developed many etl solutions through Talend, they are extensively tested across platforms which are contained to local system - meaning they were tested by manually executing through the desktop software both the free version and subscription version. Now we are targeting to push/publish the Jobs gradually to Talend Cloud. How the Talend Cloud (management console etc...) works I have a fair bit of knowledge on that but what keeps me on thinking are the file based jobs.
Scenario: We have few jobs which are file based - access files from FTP or push files to FTP or write data to excel/delimited files. Now we couldn't read/write data directly from files that are in FTP so we need to have local copy first. Until now as we had developed/tested on desktop software this was all good but now what I am not sure is how do we handle the situation when we publish these jobs to Talend Cloud.
More specifically how do we handle/change the file paths that were until now pointing to some random C/D/E drive but now needs to be remapped to Cloud paths - how are the Talend Cloud directories defined?
Development Platform: Talend Open Studio for Data Integration Version: 6.5.1
Cloud Platform: Talend Integration Cloud Hybrid Edition Version: 6.5.1
You need to setup Remote Engine. This allows you to run data integration which uses on-premise applications and databases.
Read more about it here.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
An integration platform-as-a-serviceto help enterprises collect, govern, transform, and share data from any data sources
Move from On-Premises to the Cloud by following the advice of experts
Learn how to deploy Talend Jobs as Docker images to Amazon, Azure and Google Cloud registries