ActiveMQ broker crashes complaining about max connections

Four Stars

ActiveMQ broker crashes complaining about max connections

We have configured a Master-Slave pair of brokers and the master recently failed after about an hour of this message showing up in the logs:

 

2017-06-20 11:00:20,794 | ERROR | Could not accept connection  : org.apache.activemq.transport.tcp.ExceededMaximumConnectionsException: Exceeded the maximum number of allowed client connections. See the 'maximumConnections' property on the TCP transport configuration URI in the ActiveMQ configuration file (e.g., activemq.xml) | org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server Thread Handler: tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600

 

I am not sure why this should be, since we do not have nearly enough producers and consumers running in this environment to reach 1000 connections. We discovered the issue when Amazon's CloudWatch warned us that the host was reaching its RAM limit. A few minutes later the ActiveMQ process died.

 

This is on an EC2 instance with 1GB of RAM and the ActiveMQ JVM (the only thing really running on the host) shouldn't go above 512MB. The memory options are -Xms512M -Xmx512M. After the failover, I've been watching the second host and, as expected, the number of active connections reported over JMX never goes over 10. So if my maximum is set to 1000, how can it run out? If a rogue process did open 990 connections, would I run out of RAM first?

Relevant snippets from the activemq.xml:

 

          <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage percentOfJvmHeap="70" />
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="5 gb"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="512 mb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>

 

<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>

I was hoping the problem was my ActiveMQ configuration. But searching around on Google, I find people running into this error because their client applications hand-coded in Java are doing something wrong. Since my clients are Talend jobs, I am not sure how to proceed troubleshooting.

Moderator

Re: ActiveMQ broker crashes complaining about max connections

Hello,

We have redirected your issue to talend experts and then come back to you as soon as we can.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: ActiveMQ broker crashes complaining about max connections

Hello,

I have the same problem on ActiveMQ (Talend ESB 6.4.1).
Is there any feedback ?

Thanks.
Moderator

Re: ActiveMQ broker crashes complaining about max connections

Hello,

Are you able to see this error in ActiveMq console? We have fixed this JMS connection leak problem on v 6.4.1.

Are you using talend subscription solution ESB 6.4.1?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Highlighted
Employee

Re: ActiveMQ broker crashes complaining about max connections

I am having the same error on Talend Data Services Platform version v7.1.1

 

In my case, the Active MQ is receiving the messages from SAP via the Talend SAP RFC Server and I see that there are several 100's 'Active' connections on the Active MQ Console.

 

2019-03-04 13:08:43,453 | ERROR | Could not accept connection  : {} | org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server Thread Handler: tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600

org.apache.activemq.transport.tcp.ExceededMaximumConnectionsException: Exceeded the maximum number of allowed client connections. See the 'maximumConnections' property on the TCP transport configuration URI in the ActiveMQ configuration file (e.g., activemq.xml)

 

I understand that the issue was fixed in version v6.4.1 so I would expect this remains intact in v7.1.1 as well.

 

Please can you confirm what could be the issue and if there is any configuration on the client (Talend SAP RFC Server) or Active MQ Server can be done to resolve this issue?

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

An API-First Approach to Modernizing Applications

Learn how to use an API-First Approach to Modernize your Applications

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

Talend API Designer – Technical Overview

Take a look at this technical overview video of Talend API Designer

Watch Now