load distribution with SEDA

[Anonymous]

Hi,
I have a scenario which I believe can be solved in many different ways. I am currently focusing on SEDA components for that but not really getting what I want.
The scenario consists of distributing load among nodes based on nodes' availability. Nodes (for the sake of simplicity of this example they are 2) will pick up messages from a queue and process them. If nothing in the queue then the nodes will simply sleep. If both nodes are sleeping when a message arrives, then only one node should be a able to pick up the message. If both nodes are busy then the messages will simply queue up. Processing of a message by a node can take an arbitrary amount of time (depending on the individual content of a message). It is possible a node may potentially pick up and process 2 (or more) light messages from the queue, in a row, if the other node is busy on a fat message meanwhile.
I believe that can be done with JMS without much of a hassle but I would prefer to do it without JMS get involved and I am trying SEDA for that. However I am not able to do it with cSEDA (or with cVM as in the attachment).
I am trying different combinations of configuration to cSEDA or cVM components but either it does not work at all or it works sequentially, i.e. processing of the fat message blocks the other thread as well.
Please tell me if the above scenario is possible in SEDA style or if I am barking at the wrong tree. The only example in Talend manual I could have found is connecting cSEDAs (or cVMs) one-to-one.
Cheers.

[Anonymous]

Hi Levent,
Do you have two separate routes defined, each to listen on the JMS queue, or just one with a message router like before? With this configuration, a single message will be processed at each consumer. So the broker won't dispatch another message to your consumer until the first is finished processing.
If you're using just one route, I suggest trying the concurrentConsumers option (&concurrentConsumers=3). If you're looking at the ActiveMQ web admin or JMX MBeans, you should see the number of consumers on the queue match what you configure. It sounds like right now it's just 1 consumer.
Best Regards,
Ben

[Anonymous]

Hi Ben,
I am using two routes. (Please see the image.)
The ultimate intention is distributing the load across servers sitting on different pieces of hardware. ConcurrentConsumers option existed in seda as well. The mechanism distributes the load as I liked but in both the problem with concurrent consumers approach is that further down the route I cannot tell which consumer I am in, and therefore I cannot dispatch the job to the hardware I wanted.
Hope I am making sense.
Best regards,
Levent

[Anonymous]

Hi Levent,
That makes sense.
Do you have an ESB Runtime set up as well? I'm curious to see if the behavior is the same there if you deploy the route.
Best Regards,
Ben

[Anonymous]

You could also set up two separate route 'jobs' in Studio. Meaning you'd go right click and select 'New Route'. Then set up your second consumer there and run both of them at the same time.
The reason for my last 2 suggestions is that I think thread concurrency is restricted for routes running in Studio, but I'm not positive. Splitting into two should take care of that.