Unable to put multiple files in Amazon S3

Six Stars

Unable to put multiple files in Amazon S3

I'm trying to put multiple files from a folder into a Bucket in Amazon S3. 

 

My process involves creating new files daily from various stock exchanges and putting these files in a local folder. The new files are timestamped by current date:

"/Users/zalibra/Desktop/Exchanges" + TalendDate.getDate("CCYYMMDD") + "-ExchangeA.json" and "/Users/zalibra/Desktop/Exchanges" + TalendDate.getDate("CCYYMMDD") + "-ExchangeB.json"

 

Once the files are generated, I want to load the current day's files directly from the folder into S3. I've configured the process as per the picture. 

 

tFileInputJSON --> tLogRow --> tFileOutputJSON (2 exchanges, 2 files per day which get saved in Desktop/Exchanges folder)

tS3Connection --> tFileList --> tS3Put

 

I've provided the file structure in tFileList and now I'm trying to figure out how to configure tS3Put to come directly to my Desktop/Exchanges folder and take only today's files and put in S3. 

 

This is what I've put in tS3Put but I'm getting errors:

Bucket: "<root folder in S3>/<sub folder>"

Key: "/Users/zalibra/Desktop/Exchanges"

File or Stream: "" + TalendDate.getDate("CCYYMMDD") + "" <-- this is where I'm getting the error. 

 

Any help would be appreciated. 1.jpg2.jpg3.jpg


Accepted Solutions
Community Manager

Re: Unable to put multiple files in Amazon S3

This document should explain the AWS interpretation of "key" https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html

The difference between "file" and "key" within the tS3Put component is that "file" is the name of the local file to be added to the S3 bucket, and "key" is it's unique key within S3. They are for different environments, your local and your S3 environment.


All Replies
Community Manager

Re: Unable to put multiple files in Amazon S3

I've just skimmed through this quickly, so I might not have caught all your problems, but I suspect it is this code that is causing your problem....

 

"" + TalendDate.getDate("CCYYMMDD") + "" 

I believe you need to use something more like this....

((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))

This is a globalMap variable that passes the name and path of the file picked up by the tFileList component to the tS3Put component

Six Stars

Re: Unable to put multiple files in Amazon S3

It worked, thank you, but I'm not sure I follow the logic. Isn't the "Key" supposed to be the folder where I want to put the files? What is the difference between the fields "Key" and "File"?

 

Below is what I did in tS3Put (got it from another thread to modify the Key):

Key: ((String)globalMap.get("tFileList_1_CURRENT_FILE"))
File: ((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))

Community Manager

Re: Unable to put multiple files in Amazon S3

This document should explain the AWS interpretation of "key" https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html

The difference between "file" and "key" within the tS3Put component is that "file" is the name of the local file to be added to the S3 bucket, and "key" is it's unique key within S3. They are for different environments, your local and your S3 environment.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Downloads and Trials

Test drive Talend's enterprise products.

Downloads