Writing the data to S3 sub-folders

One Star

Writing the data to S3 sub-folders

Hi All,
We are currently working on a requirement which would write files to the S3 bucket.
Under the main bucket there are many sub-folders and some of them need to be created on the fly.
For example we need to create folders on a daily basis for each hour under the main bucket.
Has anyone worked on similar requirement before?If yes,could you suggest a good approach?
One option which I can think of is to replicate the folder structure in the location from where I need to upload to S3,then put this filepath as a dynamic key in the ts3put bucket.but there is no option to create folders if it doesn't exist.not sure this works similar to the aws s3:cp command with recursive option.Currently I am waiting to get write access on the bucket,before I can actually try this out
Regards,
Sunil
Highlighted
Seventeen Stars

Re: Writing the data to S3 sub-folders

hi,
did you already have a look at component around s3 from Talend help ?
there is a list of some video/tutos that can help you I guess ...
Working with Talend's Amazon Simple Storage Service %28S3%29 components - Talend v5.4 features
https://help.talend.com/search/all?query=tS3Put&content-lang=en
and so on
hope it helps
regards
laurent
Highlighted
One Star

Re: Writing the data to S3 sub-folders

Hi Laurent,
Thanks.
I already checked the Talend videos . I have been able to get the files recursively.However I am looking at an option where the copy can be done recursively to the S3 bucket.
Regards,
Sunil

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog