Dynamically generating Job Script and executing it?
I?m wondering if I could get some help on an approach to a problem I have.
I need to build the backend java server for a new UI at our company. The user will be able to places boxes on the screen and connect them together to describe a processing flow. Yes, this sounds like Talend itself (and many other ETL tools) and there are similarities. The UI will be simpler and with fewer options ? with domain specific abilities.
After taking a several day Talend training course I had an idea for a solution which I?m hesitant to try without seeing if it is possible (or reasonable) first.
- My java server accepts a request to run one of my ?Jobs?
- I convert that description into a Job Script, submit it to TAC.
- Assuming no errors when compiling/building it, I request a Distance Run
- I monitor the run (some how) to make sure it doesn?t fail
- The Job Script has been properly configured to write status info of the run via a tStatCatcher, so I can get that and make it available to the UI.
- If the user asks for the run to be killed, I have to somehow well TAC to kill it.
I?m guessing this would work, although I don?t know how to do it. But my memory of Distant Execution was that startup was VERY SLOW for the first Distant Run request. In my case, every Job Script would be a new one, so it would be so slow that this design would, I think, be a non-starter.
Anyone have any thoughts.
An alternative, which I don?t know if it could be done would be:
- My java server accepts a request to run one of my ?Jobs?
- I convert that description into a Job Script, submit it to TAC, assuming no errors when compiling/building it and somehow get the jar from TAC back to my system.
- Or I locally (without TAC) convert the Job Script into a Jar via a command line tool?
- Load that jar dynamically, spin up a new thread, and create and run the java class that Talend built for me.
- If the job fails, I?ll know? and I can use a tLogCatcher (maybe?) to get why.
- The Job Script has been properly configured to write status info of the run via a tStatCatcher, so I can get that and make it available to the UI.
- If the user asks for the run to be killed, I have to kill the thread (and hope that properly shuts things down?)
Not doing the Distance run would be faster, but converting it to a Jar sounds tricky, and stopping a running job also sounds like it could be a problem. (What if the Talend Job forked a thread? How does that thread get stopped?)
I have an alternate solution, but I wonder about one of the above solutions.
Anyone have any thoughts about if they could even be done? And if so, are they advisable?
Thanks!
There is some overlap, ya.
I have a feeling that it can't be done. The UI does something similar, but that doesn't mean that the API is available.
Maybe my question was too big and I should have broken it down into questions like:
- From my own java program, can I build a file with JobScript and turn it into a job? Can I turn it into a Job in the enterprise server (TAC)
- From my own java program, can I then compile that Job that I built from JobScript?
- From my own java program, can I remotely request that Job be run as a Distance Run?
- From my own java program, can I remotely request that the Distnace Run be killed?
The UI does these things, but that doesn't mean I can from my own program. If there was a command line interface, I could wrap that (I don't *want* to, but I could.) I don't know if there is documentation on the CLI.
There is some overlap, ya. I have a feeling that it can't be done. The UI does something similar, but that doesn't mean that the API is available. Maybe my question was too big and I should have broken it down into questions like: - From my own java program, can I build a file with JobScript and turn it into a job? Can I turn it into a Job in the enterprise server (TAC) - From my own java program, can I then compile that Job that I built from JobScript? - From my own java program, can I remotely request that Job be run as a Distance Run? - From my own java program, can I remotely request that the Distnace Run be killed? The UI does these things, but that doesn't mean I can from my own program. If there was a command line interface, I could wrap that (I don't *want* to, but I could.) I don't know if there is documentation on the CLI.
I am looking to do something similar. I noticed this is an old thread. Have any features been added since then to support this requirement? Thanks in advance!
