JMXClient - Connection refused

One Star

JMXClient - Connection refused

I have a set of clustered jobservers running inside Karaf
These jobservers worked fine for some time but recently two of my jobserver nodes are showing as down in the TAC.
I've checked that the jobserver bundles are Active, as well as the Talend ESB registry bundles. 
I do not see any errors emitted by the jobserver bundles on the affected nodes. 
I've restarted these bundles and Karaf itself, but neither seem to resolve the issue. 

When I check the catalina.out file I see the following errors for the jobservers that are down:
2016-08-12 17:03:05 WARN  JMXClient - Connection refused 'service:jmx:jmxmp://node1:8888'
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at java.net.Socket.connect(Socket.java:528)
        at java.net.Socket.<init>(Socket.java:425)
        at java.net.Socket.<init>(Socket.java:208)
        at com.sun.jmx.remote.socket.SocketConnection.connect(SocketConnection.java:94)
        at com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl.connect(ClientSynchroMessageConnectionImpl.java:70)
        at javax.management.remote.generic.GenericConnector.connect(GenericConnector.java:177)
        at javax.management.remote.jmxmp.JMXMPConnector.connect(JMXMPConnector.java:119)
        at org.talend.monitoring.jmx.client.JMXClient.connectWithTimeout(Unknown Source)
        at org.talend.monitoring.jmx.client.JMXClient.connect(Unknown Source)
        at org.talend.monitoring.jmx.client.ServerInfoCentralizer$1.run(Unknown Source)
2016-08-12 17:03:05 DEBUG JMXClient - Got no JMXConnector for service:jmx:jmxmp://node1:8888
2016-08-12 17:03:05 DEBUG JMXClient - nb JMXClient: 231
2016-08-12 17:03:05 INFO  JMXClient - Connecting a JMX connector (231) for URL: service:jmx:jmxmp://node3:8888
2016-08-12 17:03:05 WARN  JMXClient - Connection refused 'service:jmx:jmxmp://node3:8888'
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at java.net.Socket.connect(Socket.java:528)
        at java.net.Socket.<init>(Socket.java:425)
        at java.net.Socket.<init>(Socket.java:208)
        at com.sun.jmx.remote.socket.SocketConnection.connect(SocketConnection.java:94)
        at com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl.connect(ClientSynchroMessageConnectionImpl.java:70)
        at javax.management.remote.generic.GenericConnector.connect(GenericConnector.java:177)
        at javax.management.remote.jmxmp.JMXMPConnector.connect(JMXMPConnector.java:119)
        at org.talend.monitoring.jmx.client.JMXClient.connectWithTimeout(Unknown Source)
        at org.talend.monitoring.jmx.client.JMXClient.connect(Unknown Source)
        at org.talend.monitoring.jmx.client.ServerInfoCentralizer$1.run(Unknown Source)

I've run out of ideas on how to debug this issue. Once it occurs I cannot seem to resolve it without reinstalling Karaf. 
One Star

Re: JMXClient - Connection refused

If your Karaf's are in a cloud environment, you may need to define their RMI hostnames. Amazon/Azure do some silly things with network interfaces and that might be causing connection failures.
add this to your karaf startup params to define it:
-Djava.rmi.server.hostname=<public/NAT address>

some more tips on how to debug connection failures:
ttps://blogs.oracle.com/jmxetc/entry/troubleshooting_connection_problems_in_jconsole

One Star

Re: JMXClient - Connection refused

I am in Azure -- good guess! I added the -Djava.rmi.server.hostname param but no love Smiley Sad
I'm still getting the same error:

2016-08-12 20:57:32 DEBUG JMXClient - nb JMXClient: 548
2016-08-12 20:57:32 DEBUG JMXClient - nb JMXClient: 549
2016-08-12 20:57:32 INFO  JMXClient - Connecting a JMX connector (548) for URL: service:jmx:jmxmp://node1:8888
2016-08-12 20:57:32 INFO  JMXClient - Connecting a JMX connector (549) for URL: service:jmx:jmxmp://node3:8888
2016-08-12 20:57:32 WARN  JMXClient - Connection refused 'service:jmx:jmxmp://node1:8888'
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at java.net.Socket.connect(Socket.java:528)
        at java.net.Socket.<init>(Socket.java:425)
        at java.net.Socket.<init>(Socket.java:208)
        at com.sun.jmx.remote.socket.SocketConnection.connect(SocketConnection.java:94)
        at com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl.connect(ClientSynchroMessageConnectionImpl.java:70)
        at javax.management.remote.generic.GenericConnector.connect(GenericConnector.java:177)
        at javax.management.remote.jmxmp.JMXMPConnector.connect(JMXMPConnector.java:119)
        at org.talend.monitoring.jmx.client.JMXClient.connectWithTimeout(Unknown Source)
        at org.talend.monitoring.jmx.client.JMXClient.connect(Unknown Source)
        at org.talend.monitoring.jmx.client.ServerInfoCentralizer$1.run(Unknown Source)
2016-08-12 20:57:32 DEBUG JMXClient - Got no JMXConnector for service:jmx:jmxmp://node1:8888
2016-08-12 20:57:32 WARN  JMXClient - Connection refused 'service:jmx:jmxmp://node3:8888'
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at java.net.Socket.connect(Socket.java:528)
        at java.net.Socket.<init>(Socket.java:425)
        at java.net.Socket.<init>(Socket.java:208)
        at com.sun.jmx.remote.socket.SocketConnection.connect(SocketConnection.java:94)
        at com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl.connect(ClientSynchroMessageConnectionImpl.java:70)
        at javax.management.remote.generic.GenericConnector.connect(GenericConnector.java:177)
        at javax.management.remote.jmxmp.JMXMPConnector.connect(JMXMPConnector.java:119)
        at org.talend.monitoring.jmx.client.JMXClient.connectWithTimeout(Unknown Source)
        at org.talend.monitoring.jmx.client.JMXClient.connect(Unknown Source)
        at org.talend.monitoring.jmx.client.ServerInfoCentralizer$1.run(Unknown Source)
2016-08-12 20:57:32 DEBUG JMXClient - Got no JMXConnector for service:jmx:jmxmp://node3:8888
2016-08-12 20:57:32 DEBUG RatedServer - Rate= 0.0 normalized = 100.0
2016-08-12 20:57:32 WARN  JMXClient - Attempted to close a null JMX connector: service:jmx:jmxmp://node1:8888
2016-08-12 20:57:32 WARN  JMXClient - Attempted to close a null JMX connector: service:jmx:jmxmp://node3:8888
2016-08-12 20:57:32 WARN  FileSenderClient - Server unreachable 127.0.1.1:8001, message: Connection refused
2016-08-12 20:57:32 WARN  FileSenderClient - Server unreachable 172.16.21.203:8001, message: Connection refused
2016-08-12 20:57:32 ERROR TalendRuntimeServerStateChecker - init talendRuntime Error!Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException
2016-08-12 20:57:32 ERROR TalendRuntimeServerStateChecker - Connection refused
2016-08-12 20:57:32 WARN  FileSenderClient - Server unreachable 172.16.21.203:8001, message: Connection refused
2016-08-12 20:57:32 ERROR TalendRuntimeServerStateChecker - init talendRuntime Error!no such object in table
2016-08-12 20:57:32 WARN  FileSenderClient - Server unreachable 127.0.1.1:8001, message: Connection refused
2016-08-12 20:57:32 DEBUG JobServerStateChecker - Duration of checkAllJobServer operation:352 ms

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Agile Data lakes & Analytics

Accelerate your data lake projects with an agile approach

Watch