I am using TOS DI 6.3.
Is there a way to do use the Talend tS3List component to do a recursive list of objects in an Amazon S3 bucket?
I am able to get a list of object under a defined prefix in the component, but if objects are added into additional deeper levels, those new objects are not being detected.
-> this is fine when listing to the folder2 prefix.
-> the random folder and files therein are not being detected by ts3List.
I would prefer not to have to develop using the s3 CLI --recursive option to get a real-time list.
As I never know what the "random_folder" may be, I'd like to emulate the --recursive function in Talend to detect new files in s3.
I do not see any --recursive option for tS3List in Talend. Any suggestions?
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Learn how to do cool things with Context Variables
Find out how to migrate from one database to another using the Dynamic schema
Pick up some tips and tricks with Context Variables