Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59910

Java 11 agent disconnection: UnsupportedOperationException from ProtocolStack$Ptr.isSendOpen

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Critical
    • Resolution: Unresolved
    • core, kubernetes-plugin
    • Docker image based on jenkins/jenkins:2.204.5-jdk11
      Both with and without Nginx 1.17.6 as reverse proxy
      Ubuntu 18.04

    Description

      Investigating a spike in builds queue size we've found out that TcpSlaveAgent listener thread was dead with the following logs:

      2019-10-23 09:02:17.236+0000 [id=200815]        SEVERE  h.TcpSlaveAgentListener$ConnectionHandler#lambda$new$0: Uncaught exception in TcpSlaveAgentListener ConnectionHandler Thread[TCP agent connection handler #1715 with /10.125.100.99:47700,5,main]
      java.lang.UnsupportedOperationException: Network layer is not supposed to call isSendOpen
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:730)
              at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:738)
              at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
              at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.isSendOpen(SSLEngineFilterLayer.java:237)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:738)
              at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
              at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.isSendOpen(ConnectionHeadersFilterLayer.java:514)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:690)
              at org.jenkinsci.remoting.protocol.ApplicationLayer.write(ApplicationLayer.java:157)
              at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.start(ChannelApplicationLayer.java:230)
              at org.jenkinsci.remoting.protocol.ProtocolStack.init(ProtocolStack.java:201)
              at org.jenkinsci.remoting.protocol.ProtocolStack.access$700(ProtocolStack.java:106)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Builder.build(ProtocolStack.java:554)
              at org.jenkinsci.remoting.engine.JnlpProtocol4Handler.handle(JnlpProtocol4Handler.java:153)
              at jenkins.slaves.JnlpSlaveAgentProtocol4.handle(JnlpSlaveAgentProtocol4.java:203)
              at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:271)
      2019-10-23 09:02:17.237+0000 [id=200815]        WARNING hudson.TcpSlaveAgentListener$1#run: Connection handler failed, restarting listener
      java.lang.UnsupportedOperationException: Network layer is not supposed to call isSendOpen
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:730)
              at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:738)
              at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
              at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.isSendOpen(SSLEngineFilterLayer.java:237)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:738)
              at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
              at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.isSendOpen(ConnectionHeadersFilterLayer.java:514)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:690)
              at org.jenkinsci.remoting.protocol.ApplicationLayer.write(ApplicationLayer.java:157)
              at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.start(ChannelApplicationLayer.java:230)
              at org.jenkinsci.remoting.protocol.ProtocolStack.init(ProtocolStack.java:201)
              at org.jenkinsci.remoting.protocol.ProtocolStack.access$700(ProtocolStack.java:106)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Builder.build(ProtocolStack.java:554)
              at org.jenkinsci.remoting.engine.JnlpProtocol4Handler.handle(JnlpProtocol4Handler.java:153)
              at jenkins.slaves.JnlpSlaveAgentProtocol4.handle(JnlpSlaveAgentProtocol4.java:203)
              at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:271) 

      Followed by logs from nodes created by Jenkins Kubernetes Plugin:

      SEVERE: http://jenkins-master.example.com/ provided port:50000 is not reachable
      java.io.IOException: http://jenkins-master.example.com/ provided port:50000 is not reachable
              at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:287)
              at hudson.remoting.Engine.innerRun(Engine.java:523)
              at hudson.remoting.Engine.run(Engine.java:474)
       

      Changing JNLP port from 50000 to 50001 and back in Jenkins settings helped to restore connection and then nodes were able to connect to master again.

      Attachments

        Activity

          markewaite Mark Waite added a comment -

          capf can you provide more details on the error that you saw when running in websocket mode?

          markewaite Mark Waite added a comment - capf can you provide more details on the error that you saw when running in websocket mode?

          Unfortunately not, I'm only relaying this from a colleague who didn't copy any logs that might be helpful. They're now using 2.264.3 patched with remoting 4.13 to benefit from JENKINS-66915 (https://github.com/jenkinsci/remoting/commit/31096eff5efbe4390cc6f4b070ee337a99e0110e).

          AFAIK there haven't been any JNLP-issues since then. We can try websocket mode again after updating to a new Jenkins version.

          capf Carsten Pfeiffer added a comment - Unfortunately not, I'm only relaying this from a colleague who didn't copy any logs that might be helpful. They're now using 2.264.3 patched with remoting 4.13 to benefit from JENKINS-66915 ( https://github.com/jenkinsci/remoting/commit/31096eff5efbe4390cc6f4b070ee337a99e0110e ). AFAIK there haven't been any JNLP-issues since then. We can try websocket mode again after updating to a new Jenkins version.
          akmjenkins ASHOK MOHANTY added a comment -

          We are also getting the same error (we are with core version 2.303.3). On coincident getting this error since we upgraded the Kub8 plugin from 1.24.1 to 1.30.11..

          Can you please provide additional info related to the 'Patch remoting v.4.13".!! How to apply, any document to follow for the same.. And/OR by moving out from 'jnlp-slave' to 'inbound-agent' will it be helpful here.!!

          {{}}

          akmjenkins ASHOK MOHANTY added a comment - We are also getting the same error (we are with core version 2.303.3). On coincident getting this error since we upgraded the Kub8 plugin from 1.24.1 to 1.30.11.. Can you please provide additional info related to the 'Patch remoting v.4.13".!! How to apply, any document to follow for the same.. And/OR by moving out from 'jnlp-slave' to 'inbound-agent' will it be helpful here.!! {{}}

          akmjenkins In the meantime, we experienced this problem also with remoting 4.13, so I cannot recommend this as a fix. We will try Jenkins 2.346.2 with remoting 4.13.2, latest kubernetes-plugin and websockets next.

          capf Carsten Pfeiffer added a comment - akmjenkins In the meantime, we experienced this problem also with remoting 4.13, so I cannot recommend this as a fix. We will try Jenkins 2.346.2 with remoting 4.13.2, latest kubernetes-plugin and websockets next.
          abrandel Anthony added a comment -

          We're seeing this same issue on jenkins 2.346.2, so I don't suspect upgrading that will help. 

          abrandel Anthony added a comment - We're seeing this same issue on jenkins 2.346.2, so I don't suspect upgrading that will help. 

          People

            Unassigned Unassigned
            oxygenxo Andrey Babushkin
            Votes:
            9 Vote for this issue
            Watchers:
            21 Start watching this issue

            Dates

              Created:
              Updated: