s3List recursive

One Star

s3List recursive

Hello,
I am using TOS DI 6.3.
Is there a way to do use the Talend tS3List component to do a recursive list of objects in an Amazon S3 bucket?
I am able to get a list of object under a defined prefix in the component, but if objects are added into additional deeper levels, those new objects are not being detected.
For example:
bucket/folder1/folder2/filename.text 
-> this is fine when listing to the folder2 prefix.
bucket/folder1/folder2/random_folder/newfile.txt 
-> the random folder and files therein are not being detected by ts3List.
I would prefer not to have to develop using the s3 CLI --recursive option to get a real-time list.
As I never know what the "random_folder" may be, I'd like to emulate the --recursive function in Talend to detect new files in s3.
I do not see any --recursive option for tS3List in Talend.  Any suggestions?
Thank you.

Highlighted
Moderator

Re: s3List recursive

Hi,
There is no recursive function in tS3List component.
Have you tried to check out "List all buckets objects" option and enter the prefix of files to be listed? In this way, tS3list will list all the files on the S3 server.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: s3List recursive

Hi Sabrina,
Thank you for the quick reply.  I'll make use of the "List all buckets objects" and change the job design to apply the filters I was using from context since I don't want to poll all prefixes in the bucket.  I'll be able to achieve the desired result with just a few extra steps.
Thank you.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog