s3 – how to define endpoint to be of form <bucket_name>.s3-<optional_zone>.amazonaws.com

Six Stars

s3 – how to define endpoint to be of form <bucket_name>.s3-<optional_zone>.amazonaws.com

Hey,

 

I'm having trouble copying data to s3 in the ap-southeast-2 (Sydney) zone, using the tS2Put component. We've never had any issues when copying to s3-eu-west-1 (Ireland).

 

when using the tS3Put component I'm getting the error

 

The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

 

Now, via a browser this can be solved if you change the endpoint from https://s3.amazonaws.com/ume-redshift-jonas-aus (where you can see the above error displayed, followed by the endpoint format aws wants) to http://ume-redshift-jonas-aus.s3-ap-southeast-2.amazonaws.com/ (where you get the expected access denied error instead).

 

Is there anyway to force the put S3 component, or is there an alternative to the s3 put component, in order to access the s3 bucket in the form <bucket_name>.s3.amazonaws.com?

 

Thanks!

 

Six Stars

Re: s3 – how to define endpoint to be of form <bucket_name>.s3-<optional_zone>.amazonaws.com

In case anyone else has issues with this error, I used https://gist.github.com/chrismdp/6c6b6c825b07f680e710 to help me out. Create a bash script and run it from a tSystem component with some params.

 

script:

#!/bin/bash

S3KEY=$1
S3SECRET=$2
S3BUCKET=$3
S3STORAGETYPE="STANDARD"
AWSREGION=$4
source_filepath=$5
object_name=$6

bucket="${S3BUCKET}"
date=$(date +"%a, %d %b %Y %T %z")
acl="x-amz-acl:private"
content_type="application/octet-stream"
storage_type="x-amz-storage-class:${S3STORAGETYPE}"
string="PUT\n\n$content_type\n$date\n$acl\n$storage_type\n/$bucket$object_name"
signature=$(echo -en "${string}" | openssl sha1 -hmac "${S3SECRET}" -binary | base64)

curl -s -X PUT -T "$source_filepath" \
  -H "Host: $bucket.${AWSREGION}.amazonaws.com" \
  -H "Date: $date" \
  -H "Content-Type: $content_type" \
  -H "$storage_type" \
  -H "$acl" \
  -H "Authorization: AWS ${S3KEY}:$signature" \
  "https://$bucket.${AWSREGION}.amazonaws.com$object_name"

and command in tSystem like:

"./s3_post_aus.sh " + context.s3_access_key + " " + context.s3_secret_key + " " + context.s3_bucket + " s3-ap-southeast-2 " + globalMap.get("file_to_copy") + " /" + globalMap.get("s3_file_name")

cheers

Six Stars

Re: s3 – how to define endpoint to be of form <bucket_name>.s3-<optional_zone>.amazonaws.com

I would like to raise this as a bug/feature suggestion though, thoughts? For the endpoint used to change from

 

s3.awsamazon.com/<bucket_name>/<filepath>

 

to

 

<bucket_name>.s3.awsamazon.com/<filepath>

 

which seems to be aws' preferred format now?

Moderator

Re: s3 – how to define endpoint to be of form <bucket_name>.s3-<optional_zone>.amazonaws.com

Hello,

Have you already raised a jira issue on talend bug tracker?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.