Copy multiple files with pattern from S3 bucket to local unix server

One Star

Copy multiple files with pattern from S3 bucket to local unix server

Hi All,

 

Need suggestion for below:

 

I need to copy multiple files from S3 bucket to local unix server. I am using TS3List to iterate each file and passing tS3list current key to tS3Get. My files are in subfolders under S3 bucket ("inbound_bucket/DW/Vendor"). If i am giving this whole bucket in S3list bucket, am getting error as "tS3List_1 - Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler".

 

I tried below 2 options:

1. If am putting my files in top bucket itself -"inbound_bucket", then am able to copy into local unix server.

2. If am putting top bucket only - "inbound_bucket" in bucket and putting subfolders along with file pattern ("DW/Vendor/filepattern"), then files are getting copy in local along with S3 subfolders. This is what i get in local:

Lets say i have given unix local path in S3Get is /user/data. Then under this one new subfolder-/DW is getting created then another subfolder-/Vendor, then multiple files are getting copy-filepattern1.txt/filepattern2.txt.

 

Please suggest

 

Six Stars

Re: Copy multiple files with pattern from S3 bucket to local unix server

Could you please show your job design as well as component settings?

One Star

Re: Copy multiple files with pattern from S3 bucket to local unix server

Am sorry but i can't show job design.

What am mentioning in bucket and key prefix i have mentioned in query itself

15TH OCTOBER, COUNTY HALL, LONDON

Join us at the Community Lounge.

Register Now

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Modern Data Engineering in the Cloud

Learn about modern data engineering in the Cloud

Watch Now

How to deploy Talend Jobs as Docker images to Amazon, Azure and Google Cloud reg...

Learn how to deploy Talend Jobs as Docker images to Amazon, Azure and Google Cloud registries

Blog

Talend API Services Publish to Talend Cloud

Learn how to publish your API Services to Talend Cloud

Watch Now