I?m wondering if I could get some help on an approach to a problem I have.
I need to build the backend java server for a new UI at our company. The user will be able to places boxes on the screen and connect them together to describe a processing flow. Yes, this sounds like Talend itself (and many other ETL tools) and there are similarities. The UI will be simpler and with fewer options ? with domain specific abilities.
After taking a several day Talend training course I had an idea for a solution which I?m hesitant to try without seeing if it is possible (or reasonable) first.
- My java server accepts a request to run one of my ?Jobs?
- I convert that description into a Job Script, submit it to TAC.
- Assuming no errors when compiling/building it, I request a Distance Run
- I monitor the run (some how) to make sure it doesn?t fail
- The Job Script has been properly configured to write status info of the run via a tStatCatcher, so I can get that and make it available to the UI.
- If the user asks for the run to be killed, I have to somehow well TAC to kill it.
I?m guessing this would work, although I don?t know how to do it. But my memory of Distant Execution was that startup was VERY SLOW for the first Distant Run request. In my case, every Job Script would be a new one, so it would be so slow that this design would, I think, be a non-starter.
Anyone have any thoughts.
An alternative, which I don?t know if it could be done would be:
- My java server accepts a request to run one of my ?Jobs?
- I convert that description into a Job Script, submit it to TAC, assuming no errors when compiling/building it and somehow get the jar from TAC back to my system.
- Or I locally (without TAC) convert the Job Script into a Jar via a command line tool?
- Load that jar dynamically, spin up a new thread, and create and run the java class that Talend built for me.
- If the job fails, I?ll know? and I can use a tLogCatcher (maybe?) to get why.
- The Job Script has been properly configured to write status info of the run via a tStatCatcher, so I can get that and make it available to the UI.
- If the user asks for the run to be killed, I have to kill the thread (and hope that properly shuts things down?)
Not doing the Distance run would be faster, but converting it to a Jar sounds tricky, and stopping a running job also sounds like it could be a problem. (What if the Talend Job forked a thread? How does that thread get stopped?)
I have an alternate solution, but I wonder about one of the above solutions.
Anyone have any thoughts about if they could even be done? And if so, are they advisable?
Thanks!
