-
Bug
-
Resolution: Unresolved
-
Critical
-
None
-
Server:
Jenkins 2.263.4 installed on Debian Buster using http://pkg.jenkins.io/debian-stable and apt
java -version
openjdk version "11.0.9.1" 2020-11-04
OpenJDK Runtime Environment (build 11.0.9.1+1-post-Debian-1deb10u2)
OpenJDK 64-Bit Server VM (build 11.0.9.1+1-post-Debian-1deb10u2, mixed mode, sharing)
Agent:
Windows 10.0.18362 Build 18362
java -version
openjdk version "11.0.10" 2021-01-19 LTS
OpenJDK Runtime Environment Zulu11.45+28-SA (build 11.0.10+9-LTS)
OpenJDK 64-Bit Server VM Zulu11.45+28-SA (build 11.0.10+9-LTS, mixed mode)
Server: Jenkins 2.263.4 installed on Debian Buster using http://pkg.jenkins.io/debian-stable and apt java -version openjdk version "11.0.9.1" 2020-11-04 OpenJDK Runtime Environment (build 11.0.9.1+1-post-Debian-1deb10u2) OpenJDK 64-Bit Server VM (build 11.0.9.1+1-post-Debian-1deb10u2, mixed mode, sharing) Agent: Windows 10.0.18362 Build 18362 java -version openjdk version "11.0.10" 2021-01-19 LTS OpenJDK Runtime Environment Zulu11.45+28-SA (build 11.0.10+9-LTS) OpenJDK 64-Bit Server VM Zulu11.45+28-SA (build 11.0.10+9-LTS, mixed mode)
I installed the jenkins package from http://pkg.jenkins.io/debian-stable on a fresh installed Debian Buster machine using apt. After the initial configuration with the recommended plugins I added a new Node and tried to connect this agent.
Initially all looks fine, and the output of the agent.jar process tells
INFO: Connected
But a view minutes later the agent.jar process ends with the exception as shown below:
INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Remote closed connection without specifying reason
{{ at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)}}
{{ at hudson.remoting.Engine.innerRun(Engine.java:744)}}
{{ at hudson.remoting.Engine.run(Engine.java:519)}}
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Remote closed connection without specifying reason
{{ at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.onRecvClosed(ConnectionHeadersFilterLayer.java:440)}}
{{ at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)}}
{{ at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:288)}}
{{ at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:172)}}
{{ at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)}}
{{ at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)}}
{{ at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:48)}}
{{ at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:247)}}
{{ at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)}}
{{ at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)}}
{{ at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:118)}}
{{ at java.base/java.lang.Thread.run(Unknown Source)}}
{{ Suppressed: java.nio.channels.ClosedChannelException}}
{{ ... 7 more}}
Mõrz 02, 2021 2:00:31 NACHM. hudson.remoting.jnlp.Main$CuiListener error
SEVERE: The server rejected the connection: None of the protocols were accepted
java.lang.Exception: The server rejected the connection: None of the protocols were accepted
{{ at hudson.remoting.Engine.onConnectionRejected(Engine.java:829)}}
{{ at hudson.remoting.Engine.innerRun(Engine.java:769)}}
{{ at hudson.remoting.Engine.run(Engine.java:519)}}
More agent.jar log output can be found in the attached Agent.log
On the Jenkins server side I can see the following exception in the log:
2021-03-02 13:00:31.137+0000 [id=322] WARNING o.j.r.p.i.SSLEngineFilterLayer#onRecv: [JNLP4-connect connection from jenkins.windows.node/10.119.64.14:60826]
java.lang.NullPointerException
{{ at jenkins.slaves.DefaultJnlpSlaveReceiver.afterProperties(DefaultJnlpSlaveReceiver.java:127)}}
{{ at org.jenkinsci.remoting.engine.JnlpConnectionState$2.invoke(JnlpConnectionState.java:394)}}
{{ at org.jenkinsci.remoting.engine.JnlpConnectionState.fire(JnlpConnectionState.java:337)}}
{{ at org.jenkinsci.remoting.engine.JnlpConnectionState.fireAfterProperties(JnlpConnectionState.java:391)}}
{{ at org.jenkinsci.remoting.engine.JnlpProtocol4Handler$Handler.onReceiveHeaders(JnlpProtocol4Handler.java:323)}}
{{ at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.onRecv(ConnectionHeadersFilterLayer.java:196)}}
{{ at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:668)}}
{{ at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processRead(SSLEngineFilterLayer.java:369)}}
{{ at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecv(SSLEngineFilterLayer.java:117)}}
{{ at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:668)}}
{{ at org.jenkinsci.remoting.protocol.NetworkLayer.onRead(NetworkLayer.java:136)}}
{{ at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:160)}}
{{ at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795)}}
{{ at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)}}
{{ at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)}}
{{ at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)}}
{{ at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)}}
{{ at java.base/java.lang.Thread.run(Thread.java:834)}}
2021-03-02 13:00:31.137+0000 [id=322] SEVERE o.j.r.p.impl.NIONetworkLayer#ready: [JNLP4-connect connection from jenkins.windows.node/10.119.64.14:60826] Uncaught NullPointerException
java.lang.NullPointerException
{{ at jenkins.slaves.DefaultJnlpSlaveReceiver.afterProperties(DefaultJnlpSlaveReceiver.java:127)}}
{{ at org.jenkinsci.remoting.engine.JnlpConnectionState$2.invoke(JnlpConnectionState.java:394)}}
{{ at org.jenkinsci.remoting.engine.JnlpConnectionState.fire(JnlpConnectionState.java:337)}}
{{ at org.jenkinsci.remoting.engine.JnlpConnectionState.fireAfterProperties(JnlpConnectionState.java:391)}}
{{ at org.jenkinsci.remoting.engine.JnlpProtocol4Handler$Handler.onReceiveHeaders(JnlpProtocol4Handler.java:323)}}
{{ at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.onRecv(ConnectionHeadersFilterLayer.java:196)}}
{{ at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:668)}}
{{ at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processRead(SSLEngineFilterLayer.java:369)}}
{{ at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecv(SSLEngineFilterLayer.java:117)}}
{{ at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:668)}}
{{ at org.jenkinsci.remoting.protocol.NetworkLayer.onRead(NetworkLayer.java:136)}}
{{ at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:160)}}
{{ at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795)}}
{{ at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)}}
{{ at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)}}
{{ at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)}}
{{ at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)}}
{{ at java.base/java.lang.Thread.run(Thread.java:834)}}
More form the Servers log output can be found in Jenkins.log and Node.log.
From the timing of the events I assume the following behavior:
- Agent connects to server without issues
- Server and Agent communicate a bit and then Server get a Nullpointer exception and closes the TCP channel
- Agent tries to send data over the closed channel and sees that he is already disconnected
- Agent tries to reconnect, but even if there was the Nullpointer exception from 2), the server still thinks the Agent is connected and refuses the reconnect
- Agent crashes, since it can not connect the server
So the root cause in my opinion is that there is this Nullpointer exception on Jenkins Server Side, a few seconds after an Agent connects.
I can reproduce this behavior with different Agent PCs, and even reinstalled the Jenkins Package on my Debian machine twice.
The Windows PC was connected to a different Jenkins Server (also on Debian Buster) before. There it operated without any issues. To switch to the new server, I deleted the agent.jar, the workspace and the java cache in %temp% before downloading the new agent.jar from the new server. I also updated the jre on the Windows agent during troubleshooting (from jre8 to jre11) to match the main version of the java jre.
- duplicates
-
JENKINS-64510 [remoting] Issues with TCP agent reconnects
-
- Resolved
-
The NullPointerException is probably a red herring. At least it was when I investigated it recently though it's possible there is some other sequence where it causes a real problem. This section of code is trying to deal with reconnects for Inbound TCP Agents. It hasn't worked correctly for some unknown, probably long, period of time. That's not as surprising or serious as it sounds because there are other reasons why the reconnects have limited value.
I recently worked on a PR to avoid this NullPointerException and to get reconnects working, at least as well as they once did. See
JENKINS-64510and https://github.com/jenkinsci/jenkins/pull/5138 . This PR has been merged and should be in the upcoming weekly release.The real problem is probably something different, but it's not clear from the report.
Which agent version are you using?
I'm going to mark this as a duplicate of
JENKINS-64510based upon the NullPointerException, but I think the issue is really something different. Please try it again with Jenkins 2.282+ (when released) and agent version 4.7.