Employee

tS3List : not able to iterate on sub directories (buckets)

Hi,

I compared tFileList and tS3List , they are not same.

 

My requirement:

a/b/1.txt , 2.txt , 3.txt

 

I want to iterate on dir a (and subdirectories) and catch all 1 , 2 , 3 files.

Now with tS3List and tS3Get I have to mention a/b/ , how can I manage just with a/ but stil can catch hold of all files ?

 

Basicaly I am looking to iterate on subdirectoeris or sub buckets in S3 ?

 

Any idea ?

  • Talend Integration Cloud
Tags (1)
6 REPLIES
Employee

Re: tS3List : not able to iterate on sub directories (buckets)

S3 is not a drive.  It is an object object with key/value pair.  So basically the notion of directory is not the same.  

And the tS3List component only leverages AWS API.  So whatever the API provides you can do. You will need to iterate multiple times I think.

Employee

Re: tS3List : not able to iterate on sub directories (buckets)

This question was a combination scenario involving tFTPFileList and tS3List.

Now my main problem arising with is "tFTPFileList" not giving any option to iterate on sub directories.

Any idea how can we iterate on all files under subdirectories and go one after other file control ?

 

Later I can handle tS3 if this one was done. 

Employee

Re: tS3List : not able to iterate on sub directories (buckets)

as of now I have tried something like this but looking for better solution if any:

https://community.talend.com/t5/Design-and-Development/How-to-browse-subfolders-from-FTP-directory/m...

Ten Stars

Re: tS3List : not able to iterate on sub directories (buckets)

it working solution

 

so the question - what is Your problem? what push You to search for other solution?

-----------
Employee

Re: tS3List : not able to iterate on sub directories (buckets)

If I have mutiple directories , which client is expecting in future. I want to restruct the number of tfileftplist components , using multiple tfileftplist does not seem to be intelligent solution. just a workaround.

Ten Stars

Re: tS3List : not able to iterate on sub directories (buckets)

IF - it very wide :-)

 

I prefer start from the problem - what I want to do?

 

and then choose working solution

You can use:

aws cli - http://docs.aws.amazon.com/cli/latest/reference/s3/index.html#single-local-file-and-s3-object-operat...

with recursive key You can go thru subfolders

 

For example - when I need restore most resent backup from S3

 

I use aws cli for list, take only 1 most resent file and download it to server, all by tSystem

 

 

-----------