tS3List and filename

Five Stars

tS3List and filename

Hello,

I'm using the tS3List component to list files in a bucket according to a prefix key. At each iteration, it gives me :

  • tS3List_1_CURRENT_BUCKET : the current bucket
  • tS3List_1_CURRENT_KEY : the global current key path

When using the tS3Get component, it asked me for a the destination filepath, which I want to be dynamic according to the iteration. Why the tS3List component doesn't give us directly the 3 following paramaters ?

  • current_file
  • current_file_path
  • current_directory

 

Employee

Re: tS3List and filename

This is because S3 is not a normal drive with directory and file path.  It is a key/value pair.  The value being any byte stream object. As per the example from AWS documentation, an object in S3 has a key, not a filename, filepath or directory.

 

S3List.png

Five Stars

Re: tS3List and filename

I am interested in filename from tS3List component, but there is no global variable available. I have used StringHandling to do it, but not very happy with it.

 

The files are stored in S3 bucket as abc/abc/abc/abc/filename.txt, as I know this I have used StringHandling.Right, StringHandling.LEN and SQLike.mid functions.

 

 StringHandling.RIGHT( ((String)globalMap.get("tS3List_1_CURRENT_KEY")),(StringHandling.LEN(((String)globalMap.get("tS3List_1_CURRENT_KEY"))) - StringHandling.LEN((String)SQLike.mid_index(((String)globalMap.get("tS3List_1_CURRENT_KEY")),"/",4)))-1)

 

Is there a way to parse tS3List_1_CURRENT_KEY upto first "/" from right? That should give me filename little more efficiently and I do not have to hack with values such as '4'

Five Stars

Re: tS3List and filename

I use the following code to get the filename : 

((String)globalMap.get("tS3List_1_CURRENT_KEY")).substring( ((String)globalMap.get("tS3List_1_CURRENT_KEY")).lastIndexOf("/") + 1 )

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Downloads and Trials

Test drive Talend's enterprise products.

Downloads