downloading all files from S3

Eight Stars

downloading all files from S3

I tried using tS3List-->tS3Get components 

 

in tS3list I gave the bucket name and the key prefix. Under this folder, there are multiple files 

Key prefix as below 

/folder/export=2019/

 in tS3Get component for Key i gave as below 

 

((String)globalMap.get("tS3List_1_CURRENT_KEY"))

 File: as below 

/folder/work/

The problem here is if I include as File as below, it creates the folder inside from the current_key 

 

"/folder/work/" + ((String)globalMap.get("tS3List_1_CURRENT_KEY"))

it creates as below

/folder/work/folder/export=2019/a.json

my expectation is as below 

/folder/work/a.json

Accepted Solutions
Nine Stars

Re: downloading all files from S3

You could do a string replace to remove the key prefix text:

 

String keyPrefix = "/folder/export=2019/";
"/folder/work/" + ((String)globalMap.get("tS3List_1_CURRENT_KEY")).replaceAll(keyPrefix,"");

 

--
Please give Kudos and mark topics as solved where appropriate.

View solution in original post


All Replies
Nine Stars

Re: downloading all files from S3

You could do a string replace to remove the key prefix text:

 

String keyPrefix = "/folder/export=2019/";
"/folder/work/" + ((String)globalMap.get("tS3List_1_CURRENT_KEY")).replaceAll(keyPrefix,"");

 

--
Please give Kudos and mark topics as solved where appropriate.

View solution in original post

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog