I have been tasked to see how we can efficiently use Talend to copy large files (approximately 2TB - we don't have control over this file size) from S3 to Amazon RDS (Postgres). We are using the Talend Data Management Platform.
Up until now, we have been dealing with smaller datasets (a few hundred MBs at most) . We download these files from S3 to our job server and then load it into a Postgres RDS instance.
However, with a substantial increase in the file size, we are wondering if Talend has any components to stream data from S3 to Postgres RDS without having to download the file to the job server.
This would be similar to how we can use the "COPY" command to load data from S3 to Amazon Reshift.
We would appreciate any suggestion or references to working examples.
Thank you and happy holidays.
So far, talend don't support for transferring data by air.
Watch the recorded webinar!
Accelerate your data lake projects with an agile approach
Create systems and workflow to manage clean data ingestion and data transformation.
Introduction to Talend Open Studio for Data Integration.