As part of our project we have developed many etl solutions through Talend, they are extensively tested across platforms which are contained to local system - meaning they were tested by manually executing through the desktop software both the free version and subscription version. Now we are targeting to push/publish the Jobs gradually to Talend Cloud. How the Talend Cloud (management console etc...) works I have a fair bit of knowledge on that but what keeps me on thinking are the file based jobs.
Scenario: We have few jobs which are file based - access files from FTP or push files to FTP or write data to excel/delimited files. Now we couldn't read/write data directly from files that are in FTP so we need to have local copy first. Until now as we had developed/tested on desktop software this was all good but now what I am not sure is how do we handle the situation when we publish these jobs to Talend Cloud.
More specifically how do we handle/change the file paths that were until now pointing to some random C/D/E drive but now needs to be remapped to Cloud paths - how are the Talend Cloud directories defined?
Development Platform: Talend Open Studio for Data Integration Version: 6.5.1
Cloud Platform: Talend Integration Cloud Hybrid Edition Version: 6.5.1
You need to setup Remote Engine. This allows you to run data integration which uses on-premise applications and databases.
Read more about it here.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Learn about modern data engineering in the Cloud
Learn how to deploy Talend Jobs as Docker images to Amazon, Azure and Google Cloud registries
Pick up some tips and tricks with Context Variables