Message Production Slows on ActiveMQ Failover Within a Route

[davidferguson]

Hi there,

I've set up two servers with instances of ActiveMQ in a Master/Slave configuration with an Oracle database as the message store and lock mechanism.

To test the failover I've created the following Route in Talend ESB Studio:

cMQConnectionFactory contains (servers masked, but example Broker URI is below the picture):

Broker URI example:

"failover:(tcp://PrimaryHost:61616?connectionTimeout=3000,tcp://SecondaryHost:61616?connectionTimeout=3000,tcp://localhost:61616)?randomize=false&maxReconnectAttempts=0&useExponentialBackOff=false"

cTimer contains (it's a performance test):

cSetBody contains:

cJMS contains:

So the failover appears to work ok - in that when I kill the Master instance of ActiveMQ on PrimaryHost, the messages will be sent to SecondaryHost. But what I have noticed in the above Route is that on the failure of the Master instance, the messages are not produced as quickly as they were when the Master was active.

When the Master instance was active, messages were capable of being produced as fast as the Route could send them. However, on failover condition, the message production slows to about one message per second or slower. I had expected a slight backlog of message production on failing over, but I also expected that message production would increase to the original speed after a time - this doesn't appear to be the case.

So, my question: does anyone have an idea about what's causing this? Is the producer constantly trying to first send the message to the original Master instance each time, despite configuring maxReconnectAttempts=0? Is there any way to change this behaviour?

Thanks!

David.

Edit:

I've read https://activemq.apache.org/producer-flow-control.html and ensured that the JVM memoryLimit isn't being exceeded by enabling JMX monitoring of the Heap and had a look at systemUsage configuration regarding tempUsage and storeUsage which are also nowhere near being at their limits.

I've also tried compiling the Route and deploying it within the Runtime, but the behaviour is the same. I'm at a loss as to what's causing this behaviour.

[Anonymous]

Hi,

your expectation is right so we need to see why it not happens in your tests:

A few things to check

The connectionTimeout parameter is an ActiveMQ parameter but not an camel-jms parameter so you might change your URL to

failovertcp://PrimaryHost:61616,tcp://SecondaryHost:61616)?initialReconnectDelay=20&maxReconnectDelay=500&randomize=false&maxReconnectAttempts=2

I would also recommend to enable the 'Use PooledConnectionFactory'

Can you check with these two changes if you see differences in the behavior?

Thanks,
Dietmar

[davidferguson]

Hallo Dietmar,

Thanks for the help!

I've set up the cMQConnectionFactory component as suggested:

failover:(tcp://primaryHost:61616,tcp://secondaryHost:61616,tcp://localhost:61616)?initialReconnectDelay=20&maxReconnectDelay=500&randomize=false&maxReconnectAttempts=2

The "Use PooledConnectionFatory" checkbox was also checked.

The result of this test is pretty much the same - the production of messages doesn't regain its original speed on failover. However, the sending of messages on failover using this Broker URI configuration is much slower than my initial entry (~10-15 seconds per message, compared to ~5 seconds in my initial test). I thought it could have had something to do with the maxReconnectAttempts=2, so I also tested:

failover:(tcp://primaryHost:61616,tcp://secondaryHost:61616,tcp://localhost:61616)?initialReconnectDelay=20&maxReconnectDelay=500&randomize=false&maxReconnectAttempts=0

but this configuration is also much slower.

Just to cover all the bases I also tested this (note the use of connectionTimeout):

failover:(tcp://primaryHost:61616?connectionTimeout=3000,tcp://secondaryHost:61616?connectionTimeout=3000,tcp://localhost:61616)?initialReconnectDelay=20&maxReconnectDelay=500&randomize=false&maxReconnectAttempts=0

I know you said that connectionTimeout is an ActiveMQ parameter and isn't used in Camel - but it sure does seem like it has an effect in Talend Studio as the failover messages using this Broker URI configuration on a failover condition slow to a speed of ~5 seconds per message (quicker than the previous Broker URI configuration).

Can you also produce this behaviour? Hopefully the findings above can help you to pinpoint the issue. If you need me to check other configurations, just let me know!

Thanks,

David.