We have data coming into S3 ,And based on File creation , we want to track the s3 event notifications and trigger a talend job.For this we are pang the below design:
S3 bucket -> Event Notification --> SNS -- SQS Queue --> for each event trigger the Talend job which reads data from S3.
i am just stuck in the last part.how shd i read the each message and trigger the talend child job? Should i read the meesage create a file and keep a file watcher and then kick off.or is there anyway where i can loop the job through the each mesage..
require a Wise solution for this as the performance peeps in when i loop thru each message.
You do not need a file watcher. The SQS Message Notification should be enough if you provide the path to the S3 file in the message. The job can read the SQS Message and get the path of the file and then process it.
Thanks for the information.One technical Doubt i have is..SQS Queue will have the event notifications,How should i design my Job to have a conditional trigger?
I have one job developed which downlods from S3 ,but hw shd i design the job to integrae it with SQS ..initial step.
I wil configure my TSQSINput to read message ,but hw shd i loop my code based on one message(event),take the filename from event and run S3get..
I am just not able to visualise this. and stuck here.can anyone help pls?
Join us at the Community Lounge.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Learn how to make your data more available, reduce costs and cut your build time
Read about OTTO's experiences with Big Data and Personalized Experiences
Take a look at this video about Talend Integration with Databricks