I'm trying to put multiple files from a folder into a Bucket in Amazon S3.
My process involves creating new files daily from various stock exchanges and putting these files in a local folder. The new files are timestamped by current date:
"/Users/zalibra/Desktop/Exchanges" + TalendDate.getDate("CCYYMMDD") + "-ExchangeA.json" and "/Users/zalibra/Desktop/Exchanges" + TalendDate.getDate("CCYYMMDD") + "-ExchangeB.json"
Once the files are generated, I want to load the current day's files directly from the folder into S3. I've configured the process as per the picture.
tFileInputJSON --> tLogRow --> tFileOutputJSON (2 exchanges, 2 files per day which get saved in Desktop/Exchanges folder)
tS3Connection --> tFileList --> tS3Put
I've provided the file structure in tFileList and now I'm trying to figure out how to configure tS3Put to come directly to my Desktop/Exchanges folder and take only today's files and put in S3.
This is what I've put in tS3Put but I'm getting errors:
Bucket: "<root folder in S3>/<sub folder>"
File or Stream: "" + TalendDate.getDate("CCYYMMDD") + "" <-- this is where I'm getting the error.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Pick up some tips and tricks with Context Variables
Learn how media organizations have achieved success with Data Integration
Create systems and workflow to manage clean data ingestion and data transformation.