Details
-
Bug
-
Status: Open (View Workflow)
-
Critical
-
Resolution: Unresolved
-
Docker image based on jenkins/jenkins:2.204.5-jdk11
Both with and without Nginx 1.17.6 as reverse proxy
Ubuntu 18.04
Description
Investigating a spike in builds queue size we've found out that TcpSlaveAgent listener thread was dead with the following logs:
2019-10-23 09:02:17.236+0000 [id=200815] SEVERE h.TcpSlaveAgentListener$ConnectionHandler#lambda$new$0: Uncaught exception in TcpSlaveAgentListener ConnectionHandler Thread[TCP agent connection handler #1715 with /10.125.100.99:47700,5,main] java.lang.UnsupportedOperationException: Network layer is not supposed to call isSendOpen at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:730) at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:738) at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.isSendOpen(SSLEngineFilterLayer.java:237) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:738) at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340) at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.isSendOpen(ConnectionHeadersFilterLayer.java:514) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:690) at org.jenkinsci.remoting.protocol.ApplicationLayer.write(ApplicationLayer.java:157) at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.start(ChannelApplicationLayer.java:230) at org.jenkinsci.remoting.protocol.ProtocolStack.init(ProtocolStack.java:201) at org.jenkinsci.remoting.protocol.ProtocolStack.access$700(ProtocolStack.java:106) at org.jenkinsci.remoting.protocol.ProtocolStack$Builder.build(ProtocolStack.java:554) at org.jenkinsci.remoting.engine.JnlpProtocol4Handler.handle(JnlpProtocol4Handler.java:153) at jenkins.slaves.JnlpSlaveAgentProtocol4.handle(JnlpSlaveAgentProtocol4.java:203) at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:271) 2019-10-23 09:02:17.237+0000 [id=200815] WARNING hudson.TcpSlaveAgentListener$1#run: Connection handler failed, restarting listener java.lang.UnsupportedOperationException: Network layer is not supposed to call isSendOpen at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:730) at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:738) at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.isSendOpen(SSLEngineFilterLayer.java:237) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:738) at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340) at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.isSendOpen(ConnectionHeadersFilterLayer.java:514) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:690) at org.jenkinsci.remoting.protocol.ApplicationLayer.write(ApplicationLayer.java:157) at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.start(ChannelApplicationLayer.java:230) at org.jenkinsci.remoting.protocol.ProtocolStack.init(ProtocolStack.java:201) at org.jenkinsci.remoting.protocol.ProtocolStack.access$700(ProtocolStack.java:106) at org.jenkinsci.remoting.protocol.ProtocolStack$Builder.build(ProtocolStack.java:554) at org.jenkinsci.remoting.engine.JnlpProtocol4Handler.handle(JnlpProtocol4Handler.java:153) at jenkins.slaves.JnlpSlaveAgentProtocol4.handle(JnlpSlaveAgentProtocol4.java:203) at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:271)
Followed by logs from nodes created by Jenkins Kubernetes Plugin:
SEVERE: http://jenkins-master.example.com/ provided port:50000 is not reachable java.io.IOException: http://jenkins-master.example.com/ provided port:50000 is not reachable at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:287) at hudson.remoting.Engine.innerRun(Engine.java:523) at hudson.remoting.Engine.run(Engine.java:474)
Changing JNLP port from 50000 to 50001 and back in Jenkins settings helped to restore connection and then nodes were able to connect to master again.
Attachments
Activity
Carsten Pfeiffer
added a comment - Unfortunately not, I'm only relaying this from a colleague who didn't copy any logs that might be helpful. They're now using 2.264.3 patched with remoting 4.13 to benefit from JENKINS-66915 (https://github.com/jenkinsci/remoting/commit/31096eff5efbe4390cc6f4b070ee337a99e0110e).
AFAIK there haven't been any JNLP-issues since then. We can try websocket mode again after updating to a new Jenkins version.
Carsten Pfeiffer
added a comment - Unfortunately not, I'm only relaying this from a colleague who didn't copy any logs that might be helpful. They're now using 2.264.3 patched with remoting 4.13 to benefit from JENKINS-66915 ( https://github.com/jenkinsci/remoting/commit/31096eff5efbe4390cc6f4b070ee337a99e0110e ).
AFAIK there haven't been any JNLP-issues since then. We can try websocket mode again after updating to a new Jenkins version. 
ASHOK MOHANTY
added a comment - We are also getting the same error (we are with core version 2.303.3). On coincident getting this error since we upgraded the Kub8 plugin from 1.24.1 to 1.30.11..
Can you please provide additional info related to the 'Patch remoting v.4.13".!! How to apply, any document to follow for the same.. And/OR by moving out from 'jnlp-slave' to 'inbound-agent' will it be helpful here.!!
{{}}
ASHOK MOHANTY
added a comment - We are also getting the same error (we are with core version 2.303.3). On coincident getting this error since we upgraded the Kub8 plugin from 1.24.1 to 1.30.11..
Can you please provide additional info related to the 'Patch remoting v.4.13".!! How to apply, any document to follow for the same.. And/OR by moving out from 'jnlp-slave' to 'inbound-agent' will it be helpful here.!!
{{}} Carsten Pfeiffer
added a comment - akmjenkins In the meantime, we experienced this problem also with remoting 4.13, so I cannot recommend this as a fix. We will try Jenkins 2.346.2 with remoting 4.13.2, latest kubernetes-plugin and websockets next.
Carsten Pfeiffer
added a comment - akmjenkins In the meantime, we experienced this problem also with remoting 4.13, so I cannot recommend this as a fix. We will try Jenkins 2.346.2 with remoting 4.13.2, latest kubernetes-plugin and websockets next. 
capf can you provide more details on the error that you saw when running in websocket mode?