at the moment I'm searching for a solution to trigger a job via Coudwatch/Lambda.The situation is that we have to shift data from s3 to our hadoop cluster. Instead of running a job which checks for new Data every 5 Minutes we'd really like to implement trigger the Job-Runs via Cloudwatch.Until now i only saw the file and cron based triggers.Anyone built a setup in which jobruns were started externally?Best regards
Using Talend ESB - Create an end-point which your remove servers can trigger via a HTTP request, which then runs your job. ESB includes all of the components from DI, so you could import your existing job as a starting point. You'll have a lot more to consider compared to just triggering a shell script via cron, but a basic implementation isn't too much more work.
Drop a web server like Apache or Nginx on the server running your Talend job, and then create a page which will execute your job on this server, which can again be triggered by a HTTP request by the remote server. PHP is often a good choice for this, but there are plenty of other options.
Not sure if it's relevant in your case, but both of these approaches would allow you to return a status and/or data to the server which remotely triggered the job.
You'll need to consider the security implications of doing this, and even if it's only going to be accessible within your own network, you'll almost certainly want to limit access to this new web server by IP etc.
