Do not input private or sensitive data. View Qlik Privacy & Cookie Policy.
Skip to main content

cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Announcements
Qlik Connect 2026 Agenda Now Available: Explore Sessions
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
_AnonymousUser
Specialist III
Specialist III
‎2016-08-12 06:15 PM

JMXClient - Connection refused

I have a set of clustered jobservers running inside Karaf
These jobservers worked fine for some time but recently two of my jobserver nodes are showing as down in the TAC.
I've checked that the jobserver bundles are Active, as well as the Talend ESB registry bundles. 
I do not see any errors emitted by the jobserver bundles on the affected nodes. 
I've restarted these bundles and Karaf itself, but neither seem to resolve the issue. 

When I check the catalina.out file I see the following errors for the jobservers that are down: 
2016-08-12 17:03:05 WARN  JMXClient - Connection refused 'service:jmx:jmxmp://node1:8888'
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at java.net.Socket.connect(Socket.java:528)
        at java.net.Socket.<init>(Socket.java:425)
        at java.net.Socket.<init>(Socket.java:208)
        at com.sun.jmx.remote.socket.SocketConnection.connect(SocketConnection.java:94)
        at com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl.connect(ClientSynchroMessageConnectionImpl.java:70)
        at javax.management.remote.generic.GenericConnector.connect(GenericConnector.java:177)
        at javax.management.remote.jmxmp.JMXMPConnector.connect(JMXMPConnector.java:119)
        at org.talend.monitoring.jmx.client.JMXClient.connectWithTimeout(Unknown Source)
        at org.talend.monitoring.jmx.client.JMXClient.connect(Unknown Source)
        at org.talend.monitoring.jmx.client.ServerInfoCentralizer$1.run(Unknown Source)
2016-08-12 17:03:05 DEBUG JMXClient - Got no JMXConnector for service:jmx:jmxmp://node1:8888
2016-08-12 17:03:05 DEBUG JMXClient - nb JMXClient: 231
2016-08-12 17:03:05 INFO  JMXClient - Connecting a JMX connector (231) for URL: service:jmx:jmxmp://node3:8888
2016-08-12 17:03:05 WARN  JMXClient - Connection refused 'service:jmx:jmxmp://node3:8888'
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at java.net.Socket.connect(Socket.java:528)
        at java.net.Socket.<init>(Socket.java:425)
        at java.net.Socket.<init>(Socket.java:208)
        at com.sun.jmx.remote.socket.SocketConnection.connect(SocketConnection.java:94)
        at com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl.connect(ClientSynchroMessageConnectionImpl.java:70)
        at javax.management.remote.generic.GenericConnector.connect(GenericConnector.java:177)
        at javax.management.remote.jmxmp.JMXMPConnector.connect(JMXMPConnector.java:119)
        at org.talend.monitoring.jmx.client.JMXClient.connectWithTimeout(Unknown Source)
        at org.talend.monitoring.jmx.client.JMXClient.connect(Unknown Source)
        at org.talend.monitoring.jmx.client.ServerInfoCentralizer$1.run(Unknown Source)

I've run out of ideas on how to debug this issue. Once it occurs I cannot seem to resolve it without reinstalling Karaf. 
Labels (3)
Labels
322 Views
0 Likes
2 Replies
_AnonymousUser
Specialist III
Specialist III
‎2016-08-12 08:39 PM
Author

If your Karaf's are in a cloud environment, you may need to define their RMI hostnames. Amazon/Azure do some silly things with network interfaces and that might be causing connection failures.
add this to your karaf startup params to define it:
-Djava.rmi.server.hostname=<public/NAT address>

some more tips on how to debug connection failures:
ttps://blogs.oracle.com/jmxetc/entry/troubleshooting_connection_problems_in_jconsole
322 Views
0 Likes
_AnonymousUser
Specialist III
Specialist III
‎2016-08-12 10:03 PM
Author

I am in Azure -- good guess! I added the -Djava.rmi.server.hostname param but no love 0683p000009MPcz.png
I'm still getting the same error: 

2016-08-12 20:57:32 DEBUG JMXClient - nb JMXClient: 548
2016-08-12 20:57:32 DEBUG JMXClient - nb JMXClient: 549
2016-08-12 20:57:32 INFO  JMXClient - Connecting a JMX connector (548) for URL: service:jmx:jmxmp://node1:8888
2016-08-12 20:57:32 INFO  JMXClient - Connecting a JMX connector (549) for URL: service:jmx:jmxmp://node3:8888
2016-08-12 20:57:32 WARN  JMXClient - Connection refused 'service:jmx:jmxmp://node1:8888'
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at java.net.Socket.connect(Socket.java:528)
        at java.net.Socket.<init>(Socket.java:425)
        at java.net.Socket.<init>(Socket.java:208)
        at com.sun.jmx.remote.socket.SocketConnection.connect(SocketConnection.java:94)
        at com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl.connect(ClientSynchroMessageConnectionImpl.java:70)
        at javax.management.remote.generic.GenericConnector.connect(GenericConnector.java:177)
        at javax.management.remote.jmxmp.JMXMPConnector.connect(JMXMPConnector.java:119)
        at org.talend.monitoring.jmx.client.JMXClient.connectWithTimeout(Unknown Source)
        at org.talend.monitoring.jmx.client.JMXClient.connect(Unknown Source)
        at org.talend.monitoring.jmx.client.ServerInfoCentralizer$1.run(Unknown Source)
2016-08-12 20:57:32 DEBUG JMXClient - Got no JMXConnector for service:jmx:jmxmp://node1:8888
2016-08-12 20:57:32 WARN  JMXClient - Connection refused 'service:jmx:jmxmp://node3:8888'
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at java.net.Socket.connect(Socket.java:528)
        at java.net.Socket.<init>(Socket.java:425)
        at java.net.Socket.<init>(Socket.java:208)
        at com.sun.jmx.remote.socket.SocketConnection.connect(SocketConnection.java:94)
        at com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl.connect(ClientSynchroMessageConnectionImpl.java:70)
        at javax.management.remote.generic.GenericConnector.connect(GenericConnector.java:177)
        at javax.management.remote.jmxmp.JMXMPConnector.connect(JMXMPConnector.java:119)
        at org.talend.monitoring.jmx.client.JMXClient.connectWithTimeout(Unknown Source)
        at org.talend.monitoring.jmx.client.JMXClient.connect(Unknown Source)
        at org.talend.monitoring.jmx.client.ServerInfoCentralizer$1.run(Unknown Source)
2016-08-12 20:57:32 DEBUG JMXClient - Got no JMXConnector for service:jmx:jmxmp://node3:8888
2016-08-12 20:57:32 DEBUG RatedServer - Rate= 0.0 normalized = 100.0
2016-08-12 20:57:32 WARN  JMXClient - Attempted to close a null JMX connector: service:jmx:jmxmp://node1:8888
2016-08-12 20:57:32 WARN  JMXClient - Attempted to close a null JMX connector: service:jmx:jmxmp://node3:8888
2016-08-12 20:57:32 WARN  FileSenderClient - Server unreachable 127.0.1.1:8001, message: Connection refused
2016-08-12 20:57:32 WARN  FileSenderClient - Server unreachable 172.16.21.203:8001, message: Connection refused
2016-08-12 20:57:32 ERROR TalendRuntimeServerStateChecker - init talendRuntime Error!Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException 
2016-08-12 20:57:32 ERROR TalendRuntimeServerStateChecker - Connection refused
2016-08-12 20:57:32 WARN  FileSenderClient - Server unreachable 172.16.21.203:8001, message: Connection refused
2016-08-12 20:57:32 ERROR TalendRuntimeServerStateChecker - init talendRuntime Error!no such object in table
2016-08-12 20:57:32 WARN  FileSenderClient - Server unreachable 127.0.1.1:8001, message: Connection refused
2016-08-12 20:57:32 DEBUG JobServerStateChecker - Duration of checkAllJobServer operation:352 ms
322 Views
0 Likes