Trigger Job via Cloudwatch/Lambda

Four Stars

Trigger Job via Cloudwatch/Lambda


at the moment I'm searching for a solution to trigger a job via Coudwatch/Lambda.The situation is that we have to shift data from s3 to our hadoop cluster. Instead of running a job which checks for new Data every 5 Minutes we'd really like to implement trigger the Job-Runs via Cloudwatch.Until now i only saw the file and cron based triggers.Anyone built a setup in which jobruns were started externally?Best regards

Eight Stars

Re: Trigger Job via Cloudwatch/Lambda

Hi there,


You have a number of options, including:


  • Using Talend ESB - Create an end-point which your remove servers can trigger via a HTTP request, which then runs your job. ESB includes all of the components from DI, so you could import your existing job as a starting point. You'll have a lot more to consider compared to just triggering a shell script via cron, but a basic implementation isn't too much more work.
  • Drop a web server like Apache or Nginx on the server running your Talend job, and then create a page which will execute your job on this server, which can again be triggered by a HTTP request by the remote server. PHP is often a good choice for this, but there are plenty of other options.


Not sure if it's relevant in your case, but both of these approaches would allow you to return a status and/or data to the server which remotely triggered the job.


You'll need to consider the security implications of doing this, and even if it's only going to be accessible within your own network, you'll almost certainly want to limit access to this new web server by IP etc.







Introduction to Talend Open Studio for Data Integration.

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.