How to configure tS3Put to load a list of files to S3?

How to configure tS3Put to load a list of files to S3?

Hi! I was able to upload 1 specific file into Amazon S3 using tS3Put, but now i want to upload the content of a file directory in my local machine into Amazon S3, I want to know how to configure the task with file names that can vary in every execution.
Moderator

Re: How to configure tS3Put to load a list of files to S3?

Hi,
Have you tried to use tfilelist to fetch your files in a folder and then put them into S3 server by using ts3Put component?
The workflow should be:tfilelist(retrieves a set of files)-->iterate-->tS3put
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

Re: How to configure tS3Put to load a list of files to S3?

Thanks Sabrina! this worked, i configured tS3Put as follows:
Bucket: context.strBucketNm
Key: ((String)globalMap.get("tFileList_1_CURRENT_FILE"))
File: context.strSourceFolder + "/" +((String)globalMap.get("tFileList_1_CURRENT_FILE"))
And when running i get the following message:
connecting to socket on port 3386
connected
Connection pool shut down
Connection pool shut down

disconnected
Although all files are uploaded, why could i be receiving this Connection pool shut down?
Thanks!!
Moderator

Re: How to configure tS3Put to load a list of files to S3?

Hi,
Have you tried to check out option "Die on error" on tS3Put component to see if there is any more error information printed on console?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars

Re: How to configure tS3Put to load a list of files to S3?

I'm trying to put multiple files from a folder into a Bucket in Amazon S3. 

 

My process involves creating new files daily from various stock exchanges and putting these files in a local folder. The new files are timestamped by current date:

"/Users/zalibra/Desktop/Exchanges" + TalendDate.getDate("CCYYMMDD") + "-ExchangeA.json" and "/Users/zalibra/Desktop/Exchanges" + TalendDate.getDate("CCYYMMDD") + "-ExchangeB.json"

 

Once the files are generated, I want to load the current day's files directly from the folder into S3. I've configured the process as per the picture. 

 

tFileInputJSON --> tLogRow --> tFileOutputJSON (2 exchanges, 2 files per day which get saved in Desktop/Exchanges folder)

tS3Connection --> tFileList --> tS3Put

 

I've provided the file structure in tFileList and now I'm trying to figure out how to configure tS3Put to come directly to my Desktop/Exchanges folder and take only today's files and put in S3. 

 

This is what I've put in tS3Put but I'm getting errors:

Bucket: "<root folder in S3>/<sub folder>"

Key: "/Users/zalibra/Desktop/Exchanges"

File or Stream: "" + TalendDate.getDate("CCYYMMDD") + "" <-- this is where I'm getting the error. 

 

1.jpg2.jpg3.jpg

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.

Download