at the moment I'm searching for a solution to trigger a job via Coudwatch/Lambda.The situation is that we have to shift data from s3 to our hadoop cluster. Instead of running a job which checks for new Data every 5 Minutes we'd really like to implement trigger the Job-Runs via Cloudwatch.Until now i only saw the file and cron based triggers.Anyone built a setup in which jobruns were started externally?Best regards
You have a number of options, including:
Not sure if it's relevant in your case, but both of these approaches would allow you to return a status and/or data to the server which remotely triggered the job.
You'll need to consider the security implications of doing this, and even if it's only going to be accessible within your own network, you'll almost certainly want to limit access to this new web server by IP etc.
Introduction to Talend Open Studio for Data Integration.
Practical steps to developing your data integration strategy.
Create systems and workflow to manage clean data ingestion and data transformation.