Is there a way to get the execution plan that executed the job from the job itself?
I'm trying to create a template for every job that's going to be developed in my team and would like to handle errors through email notification or logging into a table containing the execution plan that executed the job
Some advice, on how I/we do things (open source version, no license): - Make sure your configuration information is outside of Talend, yes you could use Talend's context vars which I also recommend doing but for very specific Talend related stuff. So I have a json config file where I add almost every kind of parameter(s) source systems, privacy gdpr, which (main) jobs to or not process, etc...
The config is accessible via nodejs and a restful api which my jobs use in their components. - Logging / Error handling, take a closer look at ELK (Elastic Logstash Kibana) to monitor and implement log4j. I ingest directly into Elastic and create json from basic logging components.
- Realtime logging isn't really available, suggest trailing on log-files for realtime monitoring. But you have to implement your custom strategy to effectively log within your jobs (tError tWarning).
- Provenance... parent child jobs and dependencies... in the logging components a parent / child and root is used which you could use. Write this logging also to a database and create a view on this data. - Emailing logs, don't implement it in Talend... crontab and email log, if an error simply check exit code > 0 and provide log.
Keep your jobs clean and separate concerns... our IT monitors, set indicator and triggers / send emails to data-teams. Focus on your data and quality... let IT solve your monitoring...
Thanks for the reply! That's what I'm thinking, just having a framework that will load the config either from a file or a table and then will custom log to file/table and notify by email (if we want to).
But that framework won't do any real work, it will just call the job that actually does the work after setting up the environment for it.
My only concern is how do I know which execution plan / flow call from the Talend Cloud call that job, I'm thinking in doing it through context variables and just set that up in the flow but then that flow would only have to be executed in one execution plan, if that flow is re-used in several execution plans then that defeats the purpose as I still wouldn't know which execution plan executed the job.
