topic Re: [resolved] Parallel processing of ActiveMQ data in Talend Studio

[resolved] Parallel processing of ActiveMQ data

Aukema — Sat, 16 Nov 2024 10:50:49 GMT

I am trying to do some parallel processing of ActiveMQ messages. I created the following job:

I would expect the job to process 20 messages in parallel, but it keeps processing 1 message at a time.
What should I change to make parallel processing work?

Re: [resolved] Parallel processing of ActiveMQ data

Anonymous — Thu, 28 Jan 2016 13:15:32 GMT

Hi
Do you check the 'receive number of messages' box on tMomInput component? And change the job to:
tMomInput--main--tFlowToIterate--iterate--tJava--oncomponent--tRun
Check the 'Enable parallel execution' box on iterate link.

Re: [resolved] Parallel processing of ActiveMQ data

Aukema — Mon, 22 Feb 2016 10:26:12 GMT

When setting 'receive number of messages' to a number instead of 'keep listening' only changes the number of messages after which the jobs stops and doesn't say anything about number of parallel messages to proces.
I have created the job as you mentioned, but there is always just 1 message coming from the queue.

Re: [resolved] Parallel processing of ActiveMQ data

Anonymous — Mon, 22 Feb 2016 11:18:08 GMT

You have an 'iterate' link from the setContext, so you're executing it one by one. In this setup you set the context (static/global) and execute the subjob using the 'iterate' link, therefore one by one.
You could use tFlowIterate component and there's a checkbox 'execute parallel'. However - you'd overwrite the job context values and you will get into thread race condition. So you will have to pass data another way. (still there's a valid question - how?) You have to enable calling the subjobs in parallel too.
You can still use the mediation routes (in fact they are Camel routes) where you can control the parallelism using the ActiveMQ connection pool settings.
Gabriel

Re: [resolved] Parallel processing of ActiveMQ data

Aukema — Mon, 22 Feb 2016 12:46:46 GMT

Thanks, I already had some fear for thread race condition. This will probably also appear with hashes used.