We have configured a Master-Slave pair of brokers and the master recently failed after about an hour of this message showing up in the logs:
2017-06-20 11:00:20,794 | ERROR | Could not accept connection : org.apache.activemq.transport.tcp.ExceededMaximumConnectionsException: Exceeded the maximum number of allowed client connections. See the 'maximumConnections' property on the TCP transport configuration URI in the ActiveMQ configuration file (e.g., activemq.xml) | org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server Thread Handler: tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
I am not sure why this should be, since we do not have nearly enough producers and consumers running in this environment to reach 1000 connections. We discovered the issue when Amazon's CloudWatch warned us that the host was reaching its RAM limit. A few minutes later the ActiveMQ process died.
This is on an EC2 instance with 1GB of RAM and the ActiveMQ JVM (the only thing really running on the host) shouldn't go above 512MB. The memory options are -Xms512M -Xmx512M. After the failover, I've been watching the second host and, as expected, the number of active connections reported over JMX never goes over 10. So if my maximum is set to 1000, how can it run out? If a rogue process did open 990 connections, would I run out of RAM first?
Relevant snippets from the activemq.xml:
<systemUsage> <systemUsage> <memoryUsage> <memoryUsage percentOfJvmHeap="70" /> </memoryUsage> <storeUsage> <storeUsage limit="5 gb"/> </storeUsage> <tempUsage> <tempUsage limit="512 mb"/> </tempUsage> </systemUsage> </systemUsage>
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
I was hoping the problem was my ActiveMQ configuration. But searching around on Google, I find people running into this error because their client applications hand-coded in Java are doing something wrong. Since my clients are Talend jobs, I am not sure how to proceed troubleshooting.
We have redirected your issue to talend experts and then come back to you as soon as we can.
Are you able to see this error in ActiveMq console? We have fixed this JMS connection leak problem on v 6.4.1.
Are you using talend subscription solution ESB 6.4.1?
Introduction to Talend Open Studio for Data Integration.
Practical steps to developing your data integration strategy.
Create systems and workflow to manage clean data ingestion and data transformation.