ActiveMQ broker crashes complaining about max connections

[bthompson_nd]

We have configured a Master-Slave pair of brokers and the master recently failed after about an hour of this message showing up in the logs:

2017-06-20 11:00:20,794 | ERROR | Could not accept connection  : org.apache.activemq.transport.tcp.ExceededMaximumConnectionsException: Exceeded the maximum number of allowed client connections. See the 'maximumConnections' property on the TCP transport configuration URI in the ActiveMQ configuration file (e.g., activemq.xml) | org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server Thread Handler: tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600

I am not sure why this should be, since we do not have nearly enough producers and consumers running in this environment to reach 1000 connections. We discovered the issue when Amazon's CloudWatch warned us that the host was reaching its RAM limit. A few minutes later the ActiveMQ process died.

This is on an EC2 instance with 1GB of RAM and the ActiveMQ JVM (the only thing really running on the host) shouldn't go above 512MB. The memory options are -Xms512M -Xmx512M. After the failover, I've been watching the second host and, as expected, the number of active connections reported over JMX never goes over 10. So if my maximum is set to 1000, how can it run out? If a rogue process did open 990 connections, would I run out of RAM first?

Relevant snippets from the activemq.xml:

          <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage percentOfJvmHeap="70" />
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="5 gb"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="512 mb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>

<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>

I was hoping the problem was my ActiveMQ configuration. But searching around on Google, I find people running into this error because their client applications hand-coded in Java are doing something wrong. Since my clients are Talend jobs, I am not sure how to proceed troubleshooting.

[Anonymous]

Hello,

We have redirected your issue to talend experts and then come back to you as soon as we can.

Best regards

Sabrina

[Benoit1]

Hello,

I have the same problem on ActiveMQ (Talend ESB 6.4.1).
Is there any feedback ?

Thanks.

[Anonymous]

Hello,

Are you able to see this error in ActiveMq console? We have fixed this JMS connection leak problem on v 6.4.1.

Are you using talend subscription solution ESB 6.4.1?

Best regards

Sabrina

[Anonymous]

I am having the same error on Talend Data Services Platform version v7.1.1

In my case, the Active MQ is receiving the messages from SAP via the Talend SAP RFC Server and I see that there are several 100's 'Active' connections on the Active MQ Console.

2019-03-04 13:08:43,453 | ERROR | Could not accept connection  : {} | org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server Thread Handler: tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600

org.apache.activemq.transport.tcp.ExceededMaximumConnectionsException: Exceeded the maximum number of allowed client connections. See the 'maximumConnections' property on the TCP transport configuration URI in the ActiveMQ configuration file (e.g., activemq.xml)

I understand that the issue was fixed in version v6.4.1 so I would expect this remains intact in v7.1.1 as well.

Please can you confirm what could be the issue and if there is any configuration on the client (Talend SAP RFC Server) or Active MQ Server can be done to resolve this issue?