tS3List Connectivity Issues

One Star tt1
One Star

tS3List Connectivity Issues

All,

I am attempting to read from Amazon S3, and am receiving the following error

connecting to socket on port 3997
connected
The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
disconnected

I can use the same details with Cloudberry and get access.

Any thoughts appreciated.
Moderator

Re: tS3List Connectivity Issues

Hi,
Your component setting screenshot is preferred.
In addition that, Amazon S3 component is available since 5.4.0M3 https://jira.talendforge.org/browse/TDI-22143.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star tt1
One Star

Re: tS3List Connectivity Issues

Thankyou for replying.

Yes I am using 5.4 and the S3 components.

Job screenshot attached.
Employee

Re: tS3List Connectivity Issues

Can we have a screenshot of the component: tS3Connection? This will help us a lot.
One Star tt1
One Star

Re: tS3List Connectivity Issues

Thanks for looking.

Screenshot attached, it does not show much as I have obviously had to remove the keys.

Regards.
Employee

Re: tS3List Connectivity Issues

Hi All,
I can replicate this error message, to me its not an issue. You have this behaviour when you specify in the bucket name field the bucketName+Sub Folder where you have the file to process.
To avoid this error you just need to use the key prefix field to specify the file to process, or just put the sub folder name where you have your files to prrocess.
To me it is more design stuff than bug, there is lot of way to go over this error message. See the screenshots of the configuration that i use in my tS3List component: List all the file in folder or filter on one file

Thanks,
One Star

Re: tS3List Connectivity Issues

Has anyone resolved this? It appears that Amazon S3 limits the "GET" operation of a bucket to 1,000. It's unclear (at least in the link below) if the "max-keys" parameter can be set to an amount greater than 1,000. Another option would be to set a "marker" which would start the next search at the 1,001 object in alphabetical order.