-
Bug
-
Resolution: Unresolved
-
Major
-
None
Hi there,
I've set up a Docker swarm with three worker-nodes and connected it via the docker-swarm-plugin to my Jenkins Buildserver.
After a few days (or sometimes just a few hours) the created services which are executing the build process are exiting with an UnresolvedAddressException error. This will have the created Docker-Swarm Nodes in Jenkins to be offline as a result.
The error will occur every time until I restart Jenkins and my Docker-Swarm Nodes.
The only working workaround I've found to solve this issue is to define the Jenkins tunnel via IP-Address and not via hostname. You can see it in the configuration.
To see more about the error please have a look into the JNLP client logs:
curl --connect-timeout 20 --max-time 60 -o agent.jar https://jenkins.dummyhost.com/jenkins/jnlpJars/agent.jar,
% Total % Received % Xferd Average Speed Time Time Time Current,
Dload Upload Total Spent Left Speed,
0 0 0 0 0 0 0 0 -::- -::- -::- 0
0 0 0 0 0 0 0 0 -::- -::- -::- 0
100 856k 100 856k 0 0 2022k 0 -::- -::- -::- 2023k,
+ export '_JAVA_OPTIONS=-Dhttp.proxyHost=10.0.0.5 -Dhttp.proxyPort=8888 -Dhttps.proxyHost=10.0.0.5 -Dhttps.proxyPort=8888 -Dhttp.nonProxyHosts=localhost|127.0.0.1|10.0.12.|10.0.13.|10.0.14.|.dummyhost.com|10.1.17.*',
+ _JAVA_OPTIONS='-Dhttp.proxyHost=10.0.0.5 -Dhttp.proxyPort=8888 -Dhttps.proxyHost=10.0.0.5 -Dhttps.proxyPort=8888 -Dhttp.nonProxyHosts=localhost|127.0.0.1|10.0.12.|10.0.13.|10.0.14.|.dummyhost.com|10.1.17.*',
+ java -jar agent.jar -jnlpUrl https://jenkins.dummyhost.com/jenkins/computer/agt-ashen_bringen_3-22/slave-agent.jnlp -secret 7cc72d8ac89d9189011a4cbaf60984eeaafeb17dc70c9b4aa891e864390a71f5 -workDir /home/jenkins,
Picked up _JAVA_OPTIONS: -Dhttp.proxyHost=10.0.0.5 -Dhttp.proxyPort=8888 -Dhttps.proxyHost=10.0.0.5 -Dhttps.proxyPort=8888 -Dhttp.nonProxyHosts=localhost|127.0.0.1|10.0.12.|10.0.13.|10.0.14.|.dummyhost.com|10.1.17.*,
Jan 15, 2020 6:06:08 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir,
INFO: Using /home/jenkins/remoting as a remoting work directory,
Jan 15, 2020 6:06:08 AM org.jenkinsci.remoting.engine.WorkDirManager setupLogging,
INFO: Both error and output logs will be printed to /home/jenkins/remoting,
Jan 15, 2020 6:06:09 AM hudson.remoting.jnlp.Main createEngine,
INFO: Setting up agent: agt-ashen_bringen_3-22,
Jan 15, 2020 6:06:09 AM hudson.remoting.jnlp.Main$CuiListener <init>,
INFO: Jenkins agent is running in headless mode.,
Jan 15, 2020 6:06:09 AM hudson.remoting.Engine startEngine,
INFO: Using Remoting version: 3.36,
Jan 15, 2020 6:06:09 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir,
INFO: Using /home/jenkins/remoting as a remoting work directory,
Jan 15, 2020 6:06:09 AM hudson.remoting.jnlp.Main$CuiListener status,
INFO: Locating server among https://jenkins.dummyhost.com/jenkins/,
Jan 15, 2020 6:06:09 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve,
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping],
Jan 15, 2020 6:06:09 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve,
INFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check,
Jan 15, 2020 6:06:09 AM hudson.remoting.jnlp.Main$CuiListener status,
INFO: Agent discovery successful,
Agent address: server-hostname.dummyhost.com,
Agent port: 34119,
Identity: 71:01:9b:12:d7:3a:23:20:95:15:60:53:1d:3a:a4:5a,
Jan 15, 2020 6:06:09 AM hudson.remoting.jnlp.Main$CuiListener status,
INFO: Handshaking,
Jan 15, 2020 6:06:09 AM hudson.remoting.jnlp.Main$CuiListener status,
INFO: Connecting to server-hostname.dummyhost.com:34119,
Jan 15, 2020 6:06:09 AM hudson.remoting.jnlp.Main$CuiListener error,
SEVERE: null,
java.nio.channels.UnresolvedAddressException,
at java.base/sun.nio.ch.Net.checkAddress(Net.java:130),
at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:675),
at java.base/java.nio.channels.SocketChannel.open(SocketChannel.java:194),
at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:204),
at hudson.remoting.Engine.connectTcp(Engine.java:678),
at hudson.remoting.Engine.innerRun(Engine.java:556),
at hudson.remoting.Engine.run(Engine.java:488),
Please note, that I've censored or manipulated the IP-Addresses and hostnames in the log and the screenshots to avoid passing out internal data of my company.
[JENKINS-61353] UnresolvedAddressException in JNLP client running in Docker container
Julian Breitung
added a comment - Hi, I don't think that we've an infrastructure failue here, because if it would be so, our whole company would collapse.
The IP which is not resolved is the IP of the Jenkins server which hosts the docker-swarm-plugin and connects to the swarm agents. There is no problem with resolving the IP addresses of the docker swarm agents. The provided log output is out of the running docker-container on the docker-swarm which provides the JNLP client.
Julian Breitung
added a comment - Hi, I don't think that we've an infrastructure failue here, because if it would be so, our whole company would collapse.
The IP which is not resolved is the IP of the Jenkins server which hosts the docker-swarm-plugin and connects to the swarm agents. There is no problem with resolving the IP addresses of the docker swarm agents. The provided log output is out of the running docker-container on the docker-swarm which provides the JNLP client. I should have been more clear. The DNS resolver that I suspect is failing is inside the docker swarm, not in your corporate infrastructure. I'm not a Docker swarm expert by any stretch, but I suspect there is a DNS resolver inside it which provides DNS resolution as docker images are executed on various nodes in the swarm.
I suspect that there is an infrastructure failure on your site which is causing the DNS resolver to no longer return the IP address of the docker swarm agents. I doubt it is anything that the docker swarm plugin can resolve. If there is a bug, it is likely in the docker swarm code rather than in the swarm plugin.