Not able to Download all files from AWS S3 bucket to Local

Six Stars

Not able to Download all files from AWS S3 bucket to Local

Hello,

 I want to download all the files from AWS S3 bucket to my local directory.

I am using Talend flow as:

tS3connection->tS3List->(Iterate)->tS3Get->tS3Close.

The issue is only last file from S3 bucket is getting downloaded. All files are not downloaded.

As per the community discussions, this should work, but it is not working for me.

tS3List,tS3Get settings details are attached.

 

Can anyone help me out with the issue.

 

Thank you


Accepted Solutions
Nine Stars

Re: Not able to Download all files from AWS S3 bucket to Local

The issue is not with the key but with the job design. After your iteration you have "On Component ok" to close the S3 connection as a result once your first iteration is done it would call the tS3Close and the connection to S3 would be closed which would give you the error - "Connection pool shut down". Instead close your connections using tPostJob and likewise open all your connections using tPrejob.
JobDesign-2.JPG


All Replies
Employee

Re: Not able to Download all files from AWS S3 bucket to Local

Hi,

 

   Could you please refer a similar issue below? I believe your current key value is allowing to pick only one file.

 

https://community.talend.com/t5/Design-and-Development/Unable-to-put-multiple-files-in-Amazon-S3/td-...

 

   Could you please chnage the values to key filed and try to print the current file value in a tjava and see the results?

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

 

   

Highlighted
Six Stars

Re: Not able to Download all files from AWS S3 bucket to Local

Hi Nikhil,

 Thank you for your inputs.

I tried changing the key now.

1st file got downloaded, but got error after that.

Error is:


Exception in component tS3Get_1 (Job_try_out)
java.lang.IllegalStateException: Connection pool shut down

 

Any idea why this error is coming after downloading one file from AWS S3?

Nine Stars

Re: Not able to Download all files from AWS S3 bucket to Local

The issue is not with the key but with the job design. After your iteration you have "On Component ok" to close the S3 connection as a result once your first iteration is done it would call the tS3Close and the connection to S3 would be closed which would give you the error - "Connection pool shut down". Instead close your connections using tPostJob and likewise open all your connections using tPrejob.
JobDesign-2.JPG

Six Stars

Re: Not able to Download all files from AWS S3 bucket to Local

Hi Abhishek,

 Thanks for your help. I did not recognize that.

I am able to download all files from S3 now.

 

thank you

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Modern Data Engineering in the Cloud

Learn about modern data engineering in the Cloud

Watch Now

How to deploy Talend Jobs as Docker images to Amazon, Azure and Google Cloud reg...

Learn how to deploy Talend Jobs as Docker images to Amazon, Azure and Google Cloud registries

Blog

Best Practices for Using Context Variables with Talend – Part 3

Read about some useful Context Variable ideas

Blog