Issue with loading huge file to Amazon S3

Eight Stars

Issue with loading huge file to Amazon S3

Hello, 

 

 

I am loading data from teradata to delimited file to S3 AWS . It failed with below error. Kindly help me out how to load big files 

 

tS3Put_1 - Your proposed upload exceeds the maximum allowed size (Service: Amazon S3; Status Code: 400; Error Code: EntityTooLarge; Request ID: D9E800BC064FE9B3)

Sixteen Stars TRF
Sixteen Stars

Re: Issue with loading huge file to Amazon S3

A single put operation is limited to 5 GB.
If you need to load large objects up to 5 TB, you need to use multipart upload API as explained in documentation (https://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html).
This option is accessible from tS3Put Advanced settings using the Part Size field.
You have to set the part size according to your file size and the following contraints (issued from documentation):
- maximum number of parts per upload 10,000
- part size 5 MB to 5 GB, last part can be < 5 MB
Hope this helps.

TRF
Eight Stars

Re: Issue with loading huge file to Amazon S3

Talend version is 6.2 , so tS3Put doesnt have feature for multipart upload 

Sixteen Stars TRF
Sixteen Stars

Re: Issue with loading huge file to Amazon S3

You're right, it seems you have to upgrade to 6.3.1 at least to have this functionality.

TRF
One Star

Re: Issue with loading huge file to Amazon S3

Hello,

 

We are attempting to use the s3Put component to load a large file. We have Talend 6.3 and have tried a large variety of part sizes in an attempt to get it to work; however, we keep failing with the following error: 

 

The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchUpload; Request ID: [our request id is here])

 

Can anyone help on this error?

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog