Archiving multiple files in S3

Five Stars

Archiving multiple files in S3

Hi,

I am trying to archive the files present in S3 to a new folder appended with current date.

I need to create a new folder in the same bucket and move all the files starting with XXX to that folder.

Could anyone please suggest an approach to achieve this.

 

Thanks in advance!

 

Alekhya.

Employee

Re: Archiving multiple files in S3

Hi,

 

    If a folder is not present, it will be automatically created during Put event using tS3Put command.

image.png

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

Five Stars

Re: Archiving multiple files in S3

Thanks for the reply.

But, my requirement is to move the files(already present in S3 in a Source folder) to a new folder(to be created from Talend whose name should be appended with sysdate).

Employee

Re: Archiving multiple files in S3

Hi,

 

    There are no components right now which can move the data directly within S3. You will have to use Get and Put combination.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

Five Stars

Re: Archiving multiple files in S3

Thank You Nikhil.

But, can I move multiple files(placed in a folder) at once?

 

Thank You,

Alekhya.

Employee

Re: Archiving multiple files in S3

Hi,

 

    You will have to do it in iterative fashion. First you will have to list the components and pick the files which you need to move (Or have the list handy by any other method). Then iteratively, you can move the files one by one using Get and put commands.

 

    Ideally you should target the final resting place in S3 when you are moving the files from source itself since each transfer will increase your cost in AWS. So I would consider rewriting the first loading program itself to arrange the files in systematic way so that later file movements are not required.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

Employee

Re: Archiving multiple files in S3

@alekhyak 

 

Hi,

 

Hope I have answered your queries related to this post. If you do not have any outstanding query, could you please mark the topic as resolved? :-)

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog