topic Re: Sending job instances to a queue in Talend Studio

Sending job instances to a queue

Anonymous — Fri, 25 Mar 2016 12:11:45 GMT

Hi,
We are using 5.6 Enterprise edition. In TAC, I am able to run the job only once. I cannot start another instance of the same job until the first one completes. I get an error message: "Job is still processing". How do I kick off multiple instances of the same job so it goes into a queue?
To describe a use case, we have an external Jenkins application that calls a Talend job. There can be multiple instances each time with different set of context variables.

Thank you

Re: Sending job instances to a queue

Anonymous — Sat, 26 Mar 2016 20:37:54 GMT

I fear the TAC is not designed to do this.
I usually use a different approach. The job itself detects the work to be done and starts for the work (e.g. files) so called worker jobs which proceed with a piece of work. From the outside of the TAC I have to start the task simply when I know there is something to do. The parallelisation is solved inside the job.
One good example is file processing. The main job actually contains a tFileList and triggers via parallelized Iterate trigger tRunJob component which refers to the worker job and this worker job accepts per context variable the file path and does everything for this one file.
If you want to handover the work items from your external system to your task (job) you need to persist the data reachable from the job and the job itself decide how many parallel processes will be involved to get the work done.

Re: Sending job instances to a queue

Anonymous — Sun, 27 Mar 2016 23:52:46 GMT

Thank you. I will try something like this.