Migrating multiple S3 files from one Bucket to Another Bucket

Highlighted
Eight Stars

Migrating multiple S3 files from one Bucket to Another Bucket

Under the  source bucket, there are multiple folders and in each folder, there are multiple files

 

As the example below 

/folder/export=2019-06-01/
/folder/export=2019-06-02/
/folder/export=2019-06-03/
/folder/export=2019-06-04/

 

I want to move each file from each folder from the source bucket to the Target Bucket as below, Basically, files should be under the same date as in source bucket  

 

/abc/2019-06-01/
/abc/2019-06-02/
/abc/2019-06-03/
/abc/2019-06-04/

Accepted Solutions
Employee

Re: Migrating multiple S3 files from one Bucket to Another Bucket

Hi,

 

    Here you go!

image.png

image.png

 

row1.data.replaceAll("\\D+","") 

You can do the same stuff in other java components in Talend too.

 

For your earlier query, you can pick the records inside folder.

 

Hope I have answered your query. Please spare a second to mark the topic as resolved :-)

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

View solution in original post


All Replies
Employee

Re: Migrating multiple S3 files from one Bucket to Another Bucket

Hi,

 

    The high level steps will be as below.

 

a) Use S3List to iterate the objects for a folder

b) Parse the date part of the folder from the source 

b)  Add the parsed date part in the target bucket key while using tS3Copy to copy the object from source to target

 

    In this way, you can move the components from source to target.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

Eight Stars

Re: Migrating multiple S3 files from one Bucket to Another Bucket

Should i get the key prefix part as below in the tS3List object ?  So that it goes through each subfolder ?

 

/folder/

 

Eight Stars

Re: Migrating multiple S3 files from one Bucket to Another Bucket

Also, need help in Java how to extract date part 

 

abc/ex_date=20190610/
Employee

Re: Migrating multiple S3 files from one Bucket to Another Bucket

Hi,

 

    Here you go!

image.png

image.png

 

row1.data.replaceAll("\\D+","") 

You can do the same stuff in other java components in Talend too.

 

For your earlier query, you can pick the records inside folder.

 

Hope I have answered your query. Please spare a second to mark the topic as resolved :-)

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

View solution in original post

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog