Six Stars

Issue with loading huge file to Amazon S3

Hello, 

 

 

I am loading data from teradata to delimited file to S3 AWS . It failed with below error. Kindly help me out how to load big files 

 

tS3Put_1 - Your proposed upload exceeds the maximum allowed size (Service: Amazon S3; Status Code: 400; Error Code: EntityTooLarge; Request ID: D9E800BC064FE9B3)

3 REPLIES
Thirteen Stars TRF
Thirteen Stars

Re: Issue with loading huge file to Amazon S3

A single put operation is limited to 5 GB.
If you need to load large objects up to 5 TB, you need to use multipart upload API as explained in documentation (https://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html).
This option is accessible from tS3Put Advanced settings using the Part Size field.
You have to set the part size according to your file size and the following contraints (issued from documentation):
- maximum number of parts per upload 10,000
- part size 5 MB to 5 GB, last part can be < 5 MB
Hope this helps.

TRF
Six Stars

Re: Issue with loading huge file to Amazon S3

Talend version is 6.2 , so tS3Put doesnt have feature for multipart upload 

Thirteen Stars TRF
Thirteen Stars

Re: Issue with loading huge file to Amazon S3

You're right, it seems you have to upgrade to 6.3.1 at least to have this functionality.

TRF