One Star

Writing the data to S3 sub-folders

Hi All,
We are currently working on a requirement which would write files to the S3 bucket.
Under the main bucket there are many sub-folders and some of them need to be created on the fly.
For example we need to create folders on a daily basis for each hour under the main bucket.
Has anyone worked on similar requirement before?If yes,could you suggest a good approach?
One option which I can think of is to replicate the folder structure in the location from where I need to upload to S3,then put this filepath as a dynamic key in the ts3put bucket.but there is no option to create folders if it doesn't exist.not sure this works similar to the aws s3:cp command with recursive option.Currently I am waiting to get write access on the bucket,before I can actually try this out
Regards,
Sunil
2 REPLIES
Seventeen Stars

Re: Writing the data to S3 sub-folders

hi,
did you already have a look at component around s3 from Talend help ?
there is a list of some video/tutos that can help you I guess ...
Working with Talend's Amazon Simple Storage Service %28S3%29 components - Talend v5.4 features
https://help.talend.com/search/all?query=tS3Put&content-lang=en
and so on
hope it helps
regards
laurent
One Star

Re: Writing the data to S3 sub-folders

Hi Laurent,
Thanks.
I already checked the Talend videos . I have been able to get the files recursively.However I am looking at an option where the copy can be done recursively to the S3 bucket.
Regards,
Sunil