Six Stars

Copying multi terabytes of data from Amazon S3 to Amazon RDS (Postgres)

Hello, 
I have been tasked to see how we can efficiently use Talend to copy large files (approximately 2TB - we don't have control over this file size) from S3 to Amazon RDS (Postgres). We are using the Talend Data Management Platform.

Up until now, we have been dealing with smaller datasets (a few hundred MBs at most) . We download these files from S3 to our job server and then load it into a Postgres RDS instance. 

However, with a substantial increase in the file size, we are wondering if Talend has any components to stream data from S3 to Postgres RDS without having to download the file to the job server. 

This would be similar to how we can use the "COPY" command to load data from S3 to Amazon Reshift. 

We would appreciate any suggestion or references to working examples.

 

Thank you and happy holidays.

1 REPLY
Moderator

Re: Copying multi terabytes of data from Amazon S3 to Amazon RDS (Postgres)

Hello,

So far, talend don't support for transferring data by air.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.