I'm trying to put multiple files from a folder into a Bucket in Amazon S3.
My process involves creating new files daily from various stock exchanges and putting these files in a local folder. The new files are timestamped by current date:
"/Users/zalibra/Desktop/Exchanges" + TalendDate.getDate("CCYYMMDD") + "-ExchangeA.json" and "/Users/zalibra/Desktop/Exchanges" + TalendDate.getDate("CCYYMMDD") + "-ExchangeB.json"
Once the files are generated, I want to load the current day's files directly from the folder into S3. I've configured the process as per the picture.
tFileInputJSON --> tLogRow --> tFileOutputJSON (2 exchanges, 2 files per day which get saved in Desktop/Exchanges folder)
tS3Connection --> tFileList --> tS3Put
I've provided the file structure in tFileList and now I'm trying to figure out how to configure tS3Put to come directly to my Desktop/Exchanges folder and take only today's files and put in S3.
This is what I've put in tS3Put but I'm getting errors:
Bucket: "<root folder in S3>/<sub folder>"
File or Stream: "" + TalendDate.getDate("CCYYMMDD") + "" <-- this is where I'm getting the error.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Part 2 of a series on Context Variables
Learn how to do cool things with Context Variables
Find out how to migrate from one database to another using the Dynamic schema