Writing the data to S3 sub-folders

One Star

Writing the data to S3 sub-folders

Hi All,
We are currently working on a requirement which would write files to the S3 bucket.
Under the main bucket there are many sub-folders and some of them need to be created on the fly.
For example we need to create folders on a daily basis for each hour under the main bucket.
Has anyone worked on similar requirement before?If yes,could you suggest a good approach?
One option which I can think of is to replicate the folder structure in the location from where I need to upload to S3,then put this filepath as a dynamic key in the ts3put bucket.but there is no option to create folders if it doesn't exist.not sure this works similar to the aws s3:cp command with recursive option.Currently I am waiting to get write access on the bucket,before I can actually try this out
Regards,
Sunil
Seventeen Stars

Re: Writing the data to S3 sub-folders

hi,
did you already have a look at component around s3 from Talend help ?
there is a list of some video/tutos that can help you I guess ...
Working with Talend's Amazon Simple Storage Service %28S3%29 components - Talend v5.4 features
https://help.talend.com/search/all?query=tS3Put&content-lang=en
and so on
hope it helps
regards
laurent
One Star

Re: Writing the data to S3 sub-folders

Hi Laurent,
Thanks.
I already checked the Talend videos . I have been able to get the files recursively.However I am looking at an option where the copy can be done recursively to the S3 bucket.
Regards,
Sunil

Calling Talend Open Studio Users

The first 100 community members completing the Open Studio survey win a $10 gift voucher.

Start the survey

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Why Companies Move to the Cloud: 7 Success Stories

Learn how and why companies are moving to the Cloud

Read Now