Six Stars

Issue with aws s3 components(ts3get)

Hi ,

   I am trying to download all files in my s3 bucket in one go.

  I am able to list all files using ts3list but not able to download all files .

  ts3get is only downloading the last file.

  Please guide me through this or this function is not applicable in talend.

Please go the below link , I am getting same issue.

https://www.talendforge.org/forum/viewtopic.php?id=45397

 

6 REPLIES
Ten Stars

Re: Issue with aws s3 components(ts3get)

If you followed the job design in the link you provided, the cause of the issue is the trigger link between tS3List and tS3Get. To get all the files found by tS3List you need to connect it to tS3Get with an Iterate link.
Seven Stars

Re: Issue with aws s3 components(ts3get)

did you understand the solution provide by cterenzi?

Please post an update we can help you if need further help
Thanks,
Sid
Mark as solution if this resolved your issue
Six Stars

Re: Issue with aws s3 components(ts3get)

I have written  a python code to download all files from my s3 bucket and running my code from tsystem component.

Six Stars

Re: Issue with aws s3 components(ts3get)

Python script to extract s3 files seem to be a great idea! share with us how you did it or personal message me if possible, Iterating the output was all you need if you are trying to download s3 files using Talend components,don't hesitate to get back to me on this, if you are still looking for any help on this as I have already developed a similar job in our project. Thanks.

~ ````Please don't forget to give "kudos" when a reply is helpful and click "Accept the solution" when you think you're good with it. Thanks! ~
Four Stars

Re: Issue with aws s3 components(ts3get)

Hi Vineet,

 

Please could you provide an example of how you overcame this issue as I am current facing the same issue. 

I am trying to retrieve all files from a bucket with different names (key prefix), can this be done irrespective of the file name ? 

 

Regards,

Naledi

Four Stars

Re: Issue with aws s3 components(ts3get)

Hi Rohitthapliyal, 

 

Hi Vineet,

 

Please could you provide an example of how you overcame this issue with the python code, as I am currently facing the same issue. 

I am trying to retrieve all files from a bucket with different names (key prefix), can this be done irrespective of the file name ? 

 

Regards,

Naledi