One Star

JMXClient - Connection refused

I have a set of clustered jobservers running inside Karaf
These jobservers worked fine for some time but recently two of my jobserver nodes are showing as down in the TAC.
I've checked that the jobserver bundles are Active, as well as the Talend ESB registry bundles. 
I do not see any errors emitted by the jobserver bundles on the affected nodes. 
I've restarted these bundles and Karaf itself, but neither seem to resolve the issue. 

When I check the catalina.out file I see the following errors for the jobservers that are down:
2016-08-12 17:03:05 WARN  JMXClient - Connection refused 'service:jmx:jmxmp://node1:8888'
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at java.net.Socket.connect(Socket.java:528)
        at java.net.Socket.<init>(Socket.java:425)
        at java.net.Socket.<init>(Socket.java:208)
        at com.sun.jmx.remote.socket.SocketConnection.connect(SocketConnection.java:94)
        at com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl.connect(ClientSynchroMessageConnectionImpl.java:70)
        at javax.management.remote.generic.GenericConnector.connect(GenericConnector.java:177)
        at javax.management.remote.jmxmp.JMXMPConnector.connect(JMXMPConnector.java:119)
        at org.talend.monitoring.jmx.client.JMXClient.connectWithTimeout(Unknown Source)
        at org.talend.monitoring.jmx.client.JMXClient.connect(Unknown Source)
        at org.talend.monitoring.jmx.client.ServerInfoCentralizer$1.run(Unknown Source)
2016-08-12 17:03:05 DEBUG JMXClient - Got no JMXConnector for service:jmx:jmxmp://node1:8888
2016-08-12 17:03:05 DEBUG JMXClient - nb JMXClient: 231
2016-08-12 17:03:05 INFO  JMXClient - Connecting a JMX connector (231) for URL: service:jmx:jmxmp://node3:8888
2016-08-12 17:03:05 WARN  JMXClient - Connection refused 'service:jmx:jmxmp://node3:8888'
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at java.net.Socket.connect(Socket.java:528)
        at java.net.Socket.<init>(Socket.java:425)
        at java.net.Socket.<init>(Socket.java:208)
        at com.sun.jmx.remote.socket.SocketConnection.connect(SocketConnection.java:94)
        at com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl.connect(ClientSynchroMessageConnectionImpl.java:70)
        at javax.management.remote.generic.GenericConnector.connect(GenericConnector.java:177)
        at javax.management.remote.jmxmp.JMXMPConnector.connect(JMXMPConnector.java:119)
        at org.talend.monitoring.jmx.client.JMXClient.connectWithTimeout(Unknown Source)
        at org.talend.monitoring.jmx.client.JMXClient.connect(Unknown Source)
        at org.talend.monitoring.jmx.client.ServerInfoCentralizer$1.run(Unknown Source)

I've run out of ideas on how to debug this issue. Once it occurs I cannot seem to resolve it without reinstalling Karaf. 
2 REPLIES
One Star

Re: JMXClient - Connection refused

If your Karaf's are in a cloud environment, you may need to define their RMI hostnames. Amazon/Azure do some silly things with network interfaces and that might be causing connection failures.
add this to your karaf startup params to define it:
-Djava.rmi.server.hostname=<public/NAT address>

some more tips on how to debug connection failures:
ttps://blogs.oracle.com/jmxetc/entry/troubleshooting_connection_problems_in_jconsole

One Star

Re: JMXClient - Connection refused

I am in Azure -- good guess! I added the -Djava.rmi.server.hostname param but no love Smiley Sad
I'm still getting the same error:

2016-08-12 20:57:32 DEBUG JMXClient - nb JMXClient: 548
2016-08-12 20:57:32 DEBUG JMXClient - nb JMXClient: 549
2016-08-12 20:57:32 INFO  JMXClient - Connecting a JMX connector (548) for URL: service:jmx:jmxmp://node1:8888
2016-08-12 20:57:32 INFO  JMXClient - Connecting a JMX connector (549) for URL: service:jmx:jmxmp://node3:8888
2016-08-12 20:57:32 WARN  JMXClient - Connection refused 'service:jmx:jmxmp://node1:8888'
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at java.net.Socket.connect(Socket.java:528)
        at java.net.Socket.<init>(Socket.java:425)
        at java.net.Socket.<init>(Socket.java:208)
        at com.sun.jmx.remote.socket.SocketConnection.connect(SocketConnection.java:94)
        at com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl.connect(ClientSynchroMessageConnectionImpl.java:70)
        at javax.management.remote.generic.GenericConnector.connect(GenericConnector.java:177)
        at javax.management.remote.jmxmp.JMXMPConnector.connect(JMXMPConnector.java:119)
        at org.talend.monitoring.jmx.client.JMXClient.connectWithTimeout(Unknown Source)
        at org.talend.monitoring.jmx.client.JMXClient.connect(Unknown Source)
        at org.talend.monitoring.jmx.client.ServerInfoCentralizer$1.run(Unknown Source)
2016-08-12 20:57:32 DEBUG JMXClient - Got no JMXConnector for service:jmx:jmxmp://node1:8888
2016-08-12 20:57:32 WARN  JMXClient - Connection refused 'service:jmx:jmxmp://node3:8888'
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at java.net.Socket.connect(Socket.java:528)
        at java.net.Socket.<init>(Socket.java:425)
        at java.net.Socket.<init>(Socket.java:208)
        at com.sun.jmx.remote.socket.SocketConnection.connect(SocketConnection.java:94)
        at com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl.connect(ClientSynchroMessageConnectionImpl.java:70)
        at javax.management.remote.generic.GenericConnector.connect(GenericConnector.java:177)
        at javax.management.remote.jmxmp.JMXMPConnector.connect(JMXMPConnector.java:119)
        at org.talend.monitoring.jmx.client.JMXClient.connectWithTimeout(Unknown Source)
        at org.talend.monitoring.jmx.client.JMXClient.connect(Unknown Source)
        at org.talend.monitoring.jmx.client.ServerInfoCentralizer$1.run(Unknown Source)
2016-08-12 20:57:32 DEBUG JMXClient - Got no JMXConnector for service:jmx:jmxmp://node3:8888
2016-08-12 20:57:32 DEBUG RatedServer - Rate= 0.0 normalized = 100.0
2016-08-12 20:57:32 WARN  JMXClient - Attempted to close a null JMX connector: service:jmx:jmxmp://node1:8888
2016-08-12 20:57:32 WARN  JMXClient - Attempted to close a null JMX connector: service:jmx:jmxmp://node3:8888
2016-08-12 20:57:32 WARN  FileSenderClient - Server unreachable 127.0.1.1:8001, message: Connection refused
2016-08-12 20:57:32 WARN  FileSenderClient - Server unreachable 172.16.21.203:8001, message: Connection refused
2016-08-12 20:57:32 ERROR TalendRuntimeServerStateChecker - init talendRuntime Error!Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException
2016-08-12 20:57:32 ERROR TalendRuntimeServerStateChecker - Connection refused
2016-08-12 20:57:32 WARN  FileSenderClient - Server unreachable 172.16.21.203:8001, message: Connection refused
2016-08-12 20:57:32 ERROR TalendRuntimeServerStateChecker - init talendRuntime Error!no such object in table
2016-08-12 20:57:32 WARN  FileSenderClient - Server unreachable 127.0.1.1:8001, message: Connection refused
2016-08-12 20:57:32 DEBUG JobServerStateChecker - Duration of checkAllJobServer operation:352 ms