Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59910

Java 11 agent disconnection: UnsupportedOperationException from ProtocolStack$Ptr.isSendOpen

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • core, remoting
    • Docker image based on jenkins/jenkins:2.204.5-jdk11
      Both with and without Nginx 1.17.6 as reverse proxy
      Ubuntu 18.04

      Investigating a spike in builds queue size we've found out that TcpSlaveAgent listener thread was dead with the following logs:

      2019-10-23 09:02:17.236+0000 [id=200815]        SEVERE  h.TcpSlaveAgentListener$ConnectionHandler#lambda$new$0: Uncaught exception in TcpSlaveAgentListener ConnectionHandler Thread[TCP agent connection handler #1715 with /10.125.100.99:47700,5,main]
      java.lang.UnsupportedOperationException: Network layer is not supposed to call isSendOpen
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:730)
              at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:738)
              at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
              at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.isSendOpen(SSLEngineFilterLayer.java:237)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:738)
              at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
              at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.isSendOpen(ConnectionHeadersFilterLayer.java:514)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:690)
              at org.jenkinsci.remoting.protocol.ApplicationLayer.write(ApplicationLayer.java:157)
              at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.start(ChannelApplicationLayer.java:230)
              at org.jenkinsci.remoting.protocol.ProtocolStack.init(ProtocolStack.java:201)
              at org.jenkinsci.remoting.protocol.ProtocolStack.access$700(ProtocolStack.java:106)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Builder.build(ProtocolStack.java:554)
              at org.jenkinsci.remoting.engine.JnlpProtocol4Handler.handle(JnlpProtocol4Handler.java:153)
              at jenkins.slaves.JnlpSlaveAgentProtocol4.handle(JnlpSlaveAgentProtocol4.java:203)
              at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:271)
      2019-10-23 09:02:17.237+0000 [id=200815]        WARNING hudson.TcpSlaveAgentListener$1#run: Connection handler failed, restarting listener
      java.lang.UnsupportedOperationException: Network layer is not supposed to call isSendOpen
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:730)
              at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:738)
              at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
              at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.isSendOpen(SSLEngineFilterLayer.java:237)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:738)
              at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
              at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.isSendOpen(ConnectionHeadersFilterLayer.java:514)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:690)
              at org.jenkinsci.remoting.protocol.ApplicationLayer.write(ApplicationLayer.java:157)
              at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.start(ChannelApplicationLayer.java:230)
              at org.jenkinsci.remoting.protocol.ProtocolStack.init(ProtocolStack.java:201)
              at org.jenkinsci.remoting.protocol.ProtocolStack.access$700(ProtocolStack.java:106)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Builder.build(ProtocolStack.java:554)
              at org.jenkinsci.remoting.engine.JnlpProtocol4Handler.handle(JnlpProtocol4Handler.java:153)
              at jenkins.slaves.JnlpSlaveAgentProtocol4.handle(JnlpSlaveAgentProtocol4.java:203)
              at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:271) 

      Followed by logs from nodes created by Jenkins Kubernetes Plugin:

      SEVERE: http://jenkins-master.example.com/ provided port:50000 is not reachable
      java.io.IOException: http://jenkins-master.example.com/ provided port:50000 is not reachable
              at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:287)
              at hudson.remoting.Engine.innerRun(Engine.java:523)
              at hudson.remoting.Engine.run(Engine.java:474)
       

      Changing JNLP port from 50000 to 50001 and back in Jenkins settings helped to restore connection and then nodes were able to connect to master again.

          [JENKINS-59910] Java 11 agent disconnection: UnsupportedOperationException from ProtocolStack$Ptr.isSendOpen

          Andrey Babushkin created issue -
          Andrey Babushkin made changes -
          Environment Original: Official Docker image jenkins/jenkins:2.190.1-jdk11
          No HTTPS enabled
          Ubuntu 18.04
          New: Official Docker image jenkins/jenkins:2.190.1-jdk11
          Both with and without Nginx 1.17.6 as reverse proxy
          Ubuntu 18.04
          Andrey Babushkin made changes -
          Environment Original: Official Docker image jenkins/jenkins:2.190.1-jdk11
          Both with and without Nginx 1.17.6 as reverse proxy
          Ubuntu 18.04
          New: Docker image based on jenkins/jenkins:2.204.5-jdk11
          Both with and without Nginx 1.17.6 as reverse proxy
          Ubuntu 18.04
          Andrey Babushkin made changes -
          Description Original: Investigating a spike in builds queue size we've found out that TcpSlaveAgent listener thread was dead with the following logs:
          {code:java}
          2019-10-23 09:02:17.236+0000 [id=200815] SEVERE h.TcpSlaveAgentListener$ConnectionHandler#lambda$new$0: Uncaught exception in TcpSlaveAgentListener ConnectionHandler Thread[TCP agent connection handler #1715 with /10.125.100.99:47700,5,main]
          java.lang.UnsupportedOperationException: Network layer is not supposed to call isSendOpen
                  at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:730)
                  at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
                  at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:738)
                  at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
                  at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.isSendOpen(SSLEngineFilterLayer.java:237)
                  at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:738)
                  at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
                  at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.isSendOpen(ConnectionHeadersFilterLayer.java:514)
                  at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:690)
                  at org.jenkinsci.remoting.protocol.ApplicationLayer.write(ApplicationLayer.java:157)
                  at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.start(ChannelApplicationLayer.java:230)
                  at org.jenkinsci.remoting.protocol.ProtocolStack.init(ProtocolStack.java:201)
                  at org.jenkinsci.remoting.protocol.ProtocolStack.access$700(ProtocolStack.java:106)
                  at org.jenkinsci.remoting.protocol.ProtocolStack$Builder.build(ProtocolStack.java:554)
                  at org.jenkinsci.remoting.engine.JnlpProtocol4Handler.handle(JnlpProtocol4Handler.java:153)
                  at jenkins.slaves.JnlpSlaveAgentProtocol4.handle(JnlpSlaveAgentProtocol4.java:203)
                  at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:271)
          2019-10-23 09:02:17.237+0000 [id=200815] WARNING hudson.TcpSlaveAgentListener$1#run: Connection handler failed, restarting listener
          java.lang.UnsupportedOperationException: Network layer is not supposed to call isSendOpen
                  at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:730)
                  at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
                  at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:738)
                  at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
                  at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.isSendOpen(SSLEngineFilterLayer.java:237)
                  at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:738)
                  at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
                  at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.isSendOpen(ConnectionHeadersFilterLayer.java:514)
                  at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:690)
                  at org.jenkinsci.remoting.protocol.ApplicationLayer.write(ApplicationLayer.java:157)
                  at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.start(ChannelApplicationLayer.java:230)
                  at org.jenkinsci.remoting.protocol.ProtocolStack.init(ProtocolStack.java:201)
                  at org.jenkinsci.remoting.protocol.ProtocolStack.access$700(ProtocolStack.java:106)
                  at org.jenkinsci.remoting.protocol.ProtocolStack$Builder.build(ProtocolStack.java:554)
                  at org.jenkinsci.remoting.engine.JnlpProtocol4Handler.handle(JnlpProtocol4Handler.java:153)
                  at jenkins.slaves.JnlpSlaveAgentProtocol4.handle(JnlpSlaveAgentProtocol4.java:203)
                  at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:271) {code}
          Followed by logs from nodes created by Jenkins Kubernetes Plugin:
          {code:java}
          SEVERE: http://jenkins-master.example.com/ provided port:50000 is not reachable
          java.io.IOException: http://jenkins-master.example.com/ provided port:50000 is not reachable
                  at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:287)
                  at hudson.remoting.Engine.innerRun(Engine.java:523)
                  at hudson.remoting.Engine.run(Engine.java:474)
           {code}
          Changing JNLP port from 50000 to 50001 and back in Jenkins settings helped to restore connection and then nodes were able to connect to master again.

          A few questions:
           # How can I debug this further?
           # Can it be an issue with Jenkins 2.190.1? (We've faced this twice after upgrade from previous LTS in September)
           # Is there some way to notify administrator about such things in logs?
          New: Investigating a spike in builds queue size we've found out that TcpSlaveAgent listener thread was dead with the following logs:
          {code:java}
          2019-10-23 09:02:17.236+0000 [id=200815] SEVERE h.TcpSlaveAgentListener$ConnectionHandler#lambda$new$0: Uncaught exception in TcpSlaveAgentListener ConnectionHandler Thread[TCP agent connection handler #1715 with /10.125.100.99:47700,5,main]
          java.lang.UnsupportedOperationException: Network layer is not supposed to call isSendOpen
                  at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:730)
                  at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
                  at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:738)
                  at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
                  at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.isSendOpen(SSLEngineFilterLayer.java:237)
                  at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:738)
                  at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
                  at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.isSendOpen(ConnectionHeadersFilterLayer.java:514)
                  at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:690)
                  at org.jenkinsci.remoting.protocol.ApplicationLayer.write(ApplicationLayer.java:157)
                  at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.start(ChannelApplicationLayer.java:230)
                  at org.jenkinsci.remoting.protocol.ProtocolStack.init(ProtocolStack.java:201)
                  at org.jenkinsci.remoting.protocol.ProtocolStack.access$700(ProtocolStack.java:106)
                  at org.jenkinsci.remoting.protocol.ProtocolStack$Builder.build(ProtocolStack.java:554)
                  at org.jenkinsci.remoting.engine.JnlpProtocol4Handler.handle(JnlpProtocol4Handler.java:153)
                  at jenkins.slaves.JnlpSlaveAgentProtocol4.handle(JnlpSlaveAgentProtocol4.java:203)
                  at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:271)
          2019-10-23 09:02:17.237+0000 [id=200815] WARNING hudson.TcpSlaveAgentListener$1#run: Connection handler failed, restarting listener
          java.lang.UnsupportedOperationException: Network layer is not supposed to call isSendOpen
                  at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:730)
                  at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
                  at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:738)
                  at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
                  at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.isSendOpen(SSLEngineFilterLayer.java:237)
                  at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:738)
                  at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
                  at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.isSendOpen(ConnectionHeadersFilterLayer.java:514)
                  at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:690)
                  at org.jenkinsci.remoting.protocol.ApplicationLayer.write(ApplicationLayer.java:157)
                  at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.start(ChannelApplicationLayer.java:230)
                  at org.jenkinsci.remoting.protocol.ProtocolStack.init(ProtocolStack.java:201)
                  at org.jenkinsci.remoting.protocol.ProtocolStack.access$700(ProtocolStack.java:106)
                  at org.jenkinsci.remoting.protocol.ProtocolStack$Builder.build(ProtocolStack.java:554)
                  at org.jenkinsci.remoting.engine.JnlpProtocol4Handler.handle(JnlpProtocol4Handler.java:153)
                  at jenkins.slaves.JnlpSlaveAgentProtocol4.handle(JnlpSlaveAgentProtocol4.java:203)
                  at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:271) {code}
          Followed by logs from nodes created by Jenkins Kubernetes Plugin:
          {code:java}
          SEVERE: http://jenkins-master.example.com/ provided port:50000 is not reachable
          java.io.IOException: http://jenkins-master.example.com/ provided port:50000 is not reachable
                  at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:287)
                  at hudson.remoting.Engine.innerRun(Engine.java:523)
                  at hudson.remoting.Engine.run(Engine.java:474)
           {code}
          Changing JNLP port from 50000 to 50001 and back in Jenkins settings helped to restore connection and then nodes were able to connect to master again.
          Jesse Glick made changes -
          Labels New: remoting
          Jesse Glick made changes -
          Labels Original: remoting New: java11-compatibility remoting
          Jesse Glick made changes -
          Summary Original: Nodes can't connect to master after Uncaught exception in TcpSlaveAgentListener ConnectionHandler Thread New: Java 11 agent disconnection: UnsupportedOperationException from ProtocolStack$Ptr.isSendOpen
          Mark Waite made changes -
          Epic Link New: JENKINS-67688 [ 214193 ]
          Basil Crow made changes -
          Labels Original: java11-compatibility remoting New: java11
          Vincent Latombe made changes -
          Component/s New: remoting [ 15489 ]
          Component/s Original: kubernetes-plugin [ 20639 ]
          Basil Crow made changes -
          Epic Link Original: JENKINS-67688 [ 214193 ]

            Unassigned Unassigned
            oxygenxo Andrey Babushkin
            Votes:
            22 Vote for this issue
            Watchers:
            40 Start watching this issue

              Created:
              Updated: