Trigger Job via Cloudwatch/Lambda

Four Stars

Trigger Job via Cloudwatch/Lambda

Hi,

at the moment I'm searching for a solution to trigger a job via Coudwatch/Lambda.The situation is that we have to shift data from s3 to our hadoop cluster. Instead of running a job which checks for new Data every 5 Minutes we'd really like to implement trigger the Job-Runs via Cloudwatch.Until now i only saw the file and cron based triggers.Anyone built a setup in which jobruns were started externally?Best regards

Eight Stars

Re: Trigger Job via Cloudwatch/Lambda

Hi there,

 

You have a number of options, including:

 

  • Using Talend ESB - Create an end-point which your remove servers can trigger via a HTTP request, which then runs your job. ESB includes all of the components from DI, so you could import your existing job as a starting point. You'll have a lot more to consider compared to just triggering a shell script via cron, but a basic implementation isn't too much more work.
  • Drop a web server like Apache or Nginx on the server running your Talend job, and then create a page which will execute your job on this server, which can again be triggered by a HTTP request by the remote server. PHP is often a good choice for this, but there are plenty of other options.

 

Not sure if it's relevant in your case, but both of these approaches would allow you to return a status and/or data to the server which remotely triggered the job.

 

You'll need to consider the security implications of doing this, and even if it's only going to be accessible within your own network, you'll almost certainly want to limit access to this new web server by IP etc.

 

Regards,

 

 

Chris

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

6 Ways to Start Utilizing Machine Learning with Amazon We Services and Talend

Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend

Blog