Five Stars

tS3List and filename

Hello,

I'm using the tS3List component to list files in a bucket according to a prefix key. At each iteration, it gives me :

  • tS3List_1_CURRENT_BUCKET : the current bucket
  • tS3List_1_CURRENT_KEY : the global current key path

When using the tS3Get component, it asked me for a the destination filepath, which I want to be dynamic according to the iteration. Why the tS3List component doesn't give us directly the 3 following paramaters ?

  • current_file
  • current_file_path
  • current_directory

 

  • Data Integration
3 REPLIES
Employee

Re: tS3List and filename

This is because S3 is not a normal drive with directory and file path.  It is a key/value pair.  The value being any byte stream object. As per the example from AWS documentation, an object in S3 has a key, not a filename, filepath or directory.

 

S3List.png

Five Stars

Re: tS3List and filename

I am interested in filename from tS3List component, but there is no global variable available. I have used StringHandling to do it, but not very happy with it.

 

The files are stored in S3 bucket as abc/abc/abc/abc/filename.txt, as I know this I have used StringHandling.Right, StringHandling.LEN and SQLike.mid functions.

 

 StringHandling.RIGHT( ((String)globalMap.get("tS3List_1_CURRENT_KEY")),(StringHandling.LEN(((String)globalMap.get("tS3List_1_CURRENT_KEY"))) - StringHandling.LEN((String)SQLike.mid_index(((String)globalMap.get("tS3List_1_CURRENT_KEY")),"/",4)))-1)

 

Is there a way to parse tS3List_1_CURRENT_KEY upto first "/" from right? That should give me filename little more efficiently and I do not have to hack with values such as '4'

Five Stars

Re: tS3List and filename

I use the following code to get the filename : 

((String)globalMap.get("tS3List_1_CURRENT_KEY")).substring( ((String)globalMap.get("tS3List_1_CURRENT_KEY")).lastIndexOf("/") + 1 )