Issue with loading huge file to Amazon S3

Eight Stars

Issue with loading huge file to Amazon S3

Hello, 

 

 

I am loading data from teradata to delimited file to S3 AWS . It failed with below error. Kindly help me out how to load big files 

 

tS3Put_1 - Your proposed upload exceeds the maximum allowed size (Service: Amazon S3; Status Code: 400; Error Code: EntityTooLarge; Request ID: D9E800BC064FE9B3)

Fifteen Stars TRF
Fifteen Stars

Re: Issue with loading huge file to Amazon S3

A single put operation is limited to 5 GB.
If you need to load large objects up to 5 TB, you need to use multipart upload API as explained in documentation (https://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html).
This option is accessible from tS3Put Advanced settings using the Part Size field.
You have to set the part size according to your file size and the following contraints (issued from documentation):
- maximum number of parts per upload 10,000
- part size 5 MB to 5 GB, last part can be < 5 MB
Hope this helps.

TRF
Eight Stars

Re: Issue with loading huge file to Amazon S3

Talend version is 6.2 , so tS3Put doesnt have feature for multipart upload 

Fifteen Stars TRF
Fifteen Stars

Re: Issue with loading huge file to Amazon S3

You're right, it seems you have to upgrade to 6.3.1 at least to have this functionality.

TRF

15TH OCTOBER, COUNTY HALL, LONDON

Join us at the Community Lounge.

Register Now

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now