event based trigger in talend cloud

Highlighted
Eight Stars

event based trigger in talend cloud

Hi,

 

I have been using the talend subscription version( talend cloud). I now have a requirement where my job has to be triggered as soon as a file is available in amazon s3. So my question is does talend cloud have support for event based triggers? If not how do you suggest implementing this?

 

thanks in advance

Employee

Re: event based trigger in talend cloud

@nivedhitha 

 

Please refer the link given in the below KB article.

 

https://community.talend.com/t5/Design-and-Development/To-trigger-talend-Job-based-on-S3-bucket-even...

 

The link is for TAC but with some modifications, you should be able to do it in TMC also. 

 

Below article shows the similar requirement for Azure flavour.

 

https://community.talend.com/t5/Architecture-Best-Practices-and/Using-Azure-Functions-to-trigger-Tal...

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

Eight Stars

Re: event based trigger in talend cloud

@nikhilthampi ,

 

I have set up s3 to send notifications to SQS and my Talend Job is also able to receive messages from the queue , parse the JSON and get the name of the file uploaded in s3.

Im looking for help/suggestion on how to trigger this job(which will eventually be deployed on talend cloud) as soon as a file becomes available on s3. Is it possible only by involving lambda in between?

 

thanks in advance

Employee

Re: event based trigger in talend cloud

Hi,

 

    Lambda is the best triggering mechanism in your case. End of day you are trying to build an event based triggering mechanism and Lambda is a service whose primary purpose is to handle this situation.

 

     Unfortunately I will not be able to check the details of the solution right now. I would suggest to get the help of Talend professional services in this case since its a one off setup operation.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

 

Calling Talend Open Studio Users

The first 100 community members completing the Open Studio survey win a $10 gift voucher.

Start the survey

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Self-service Talend Migration: Moving from On-Premises to the Cloud

Move from On-Premises to the Cloud by following the advice of experts

Read Now

Modern Data Engineering in the Cloud

Learn about modern data engineering in the Cloud

Watch Now

How to deploy Talend Jobs as Docker images to Amazon, Azure and Google Cloud reg...

Learn how to deploy Talend Jobs as Docker images to Amazon, Azure and Google Cloud registries

Blog