topic Multiple Job instance in Talend Studio

Multiple Job instance

Anonymous — Mon, 24 Jul 2017 10:51:45 GMT

Hi,
I am creating some products through API.
For this I am sending json in the Post request. I can send 40 requests in one second and one request is taking around 4 second to create a product.

I need to run 60 jobs in parallel,
For this I have created a Master job which will iterate 60 times and call the subjob1 with run as independent Subjob.

Subjob1 will iterate for 100 products with run as independent Subjob.

But the problem is, Master job is not running as independent Subjob.

I am using Open Studio for Talend.

Re: Multiple Job instance

Anonymous — Mon, 24 Jul 2017 11:10:38 GMT

I think you need to be a bit clearer about what you want. When you say "subjob" do you mean "child job"? A subjob is a complete end to end section (usually represented as a series of components in a shaded box area within a job). A child job is simply a Talend job being run by another (parent) job.

You said "But the problem is, Master job is not running as independent Subjob.". The Master job will be running your child jobs, unless you are wanting some sort of recursive process.

What have you got at the moment? Can you post a screenshot?

Re: Multiple Job instance

Anonymous — Mon, 24 Jul 2017 11:30:33 GMT

Hi,
Thanks for your reply.
I am calling a child job. I have attached the screenshot to the post.
Can you suggest any other logic through which i can run 60 parallel instances.

Re: Multiple Job instance

Anonymous — Mon, 24 Jul 2017 12:15:32 GMT

You are iterating over your recordset to fire your child jobs. As such, you will only get 1 at a time. The Open Source edition of Talend does not come with multitasking (it's one of the few things they make you pay for). You can manually enable multitasking (of a kind) in Open Studio by triggering compiled jobs to be run using a script and having those jobs controlled by a shared database table (ensuring the same data is not processed by more than one job at a time). There are other ways of doing this kind of thing. You could write a small multi-threaded Java application to do this as well. Unfortunately unless you purchase the Enterprise Edition, you can't really do this with the Open Source Edition.

Re: Multiple Job instance

Anonymous — Mon, 24 Jul 2017 12:34:45 GMT

Thanks for your reply
I will try to write program for multi threading in java.