Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59910

Nodes can't connect to master after Uncaught exception in TcpSlaveAgentListener ConnectionHandler Thread

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Investigating a spike in builds queue size we've found out that TcpSlaveAgent listener thread was dead with the following logs:

      2019-10-23 09:02:17.236+0000 [id=200815]        SEVERE  h.TcpSlaveAgentListener$ConnectionHandler#lambda$new$0: Uncaught exception in TcpSlaveAgentListener ConnectionHandler Thread[TCP agent connection handler #1715 with /10.125.100.99:47700,5,main]
      java.lang.UnsupportedOperationException: Network layer is not supposed to call isSendOpen
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:730)
              at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:738)
              at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
              at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.isSendOpen(SSLEngineFilterLayer.java:237)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:738)
              at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
              at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.isSendOpen(ConnectionHeadersFilterLayer.java:514)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:690)
              at org.jenkinsci.remoting.protocol.ApplicationLayer.write(ApplicationLayer.java:157)
              at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.start(ChannelApplicationLayer.java:230)
              at org.jenkinsci.remoting.protocol.ProtocolStack.init(ProtocolStack.java:201)
              at org.jenkinsci.remoting.protocol.ProtocolStack.access$700(ProtocolStack.java:106)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Builder.build(ProtocolStack.java:554)
              at org.jenkinsci.remoting.engine.JnlpProtocol4Handler.handle(JnlpProtocol4Handler.java:153)
              at jenkins.slaves.JnlpSlaveAgentProtocol4.handle(JnlpSlaveAgentProtocol4.java:203)
              at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:271)
      2019-10-23 09:02:17.237+0000 [id=200815]        WARNING hudson.TcpSlaveAgentListener$1#run: Connection handler failed, restarting listener
      java.lang.UnsupportedOperationException: Network layer is not supposed to call isSendOpen
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:730)
              at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:738)
              at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
              at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.isSendOpen(SSLEngineFilterLayer.java:237)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.isSendOpen(ProtocolStack.java:738)
              at org.jenkinsci.remoting.protocol.FilterLayer.isSendOpen(FilterLayer.java:340)
              at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.isSendOpen(ConnectionHeadersFilterLayer.java:514)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:690)
              at org.jenkinsci.remoting.protocol.ApplicationLayer.write(ApplicationLayer.java:157)
              at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.start(ChannelApplicationLayer.java:230)
              at org.jenkinsci.remoting.protocol.ProtocolStack.init(ProtocolStack.java:201)
              at org.jenkinsci.remoting.protocol.ProtocolStack.access$700(ProtocolStack.java:106)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Builder.build(ProtocolStack.java:554)
              at org.jenkinsci.remoting.engine.JnlpProtocol4Handler.handle(JnlpProtocol4Handler.java:153)
              at jenkins.slaves.JnlpSlaveAgentProtocol4.handle(JnlpSlaveAgentProtocol4.java:203)
              at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:271) 

      Followed by logs from nodes created by Jenkins Kubernetes Plugin:

      SEVERE: http://jenkins-master.example.com/ provided port:50000 is not reachable
      java.io.IOException: http://jenkins-master.example.com/ provided port:50000 is not reachable
              at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:287)
              at hudson.remoting.Engine.innerRun(Engine.java:523)
              at hudson.remoting.Engine.run(Engine.java:474)
       

      Changing JNLP port from 50000 to 50001 and back in Jenkins settings helped to restore connection and then nodes were able to connect to master again.

        Attachments

          Activity

          Hide
          sithmein Thorsten Meinl added a comment -

          The very same thing just happened here with the latest LTS 2.303.1 (for the first time I have to say). Anything we can do to help tracking down the issue?

          Show
          sithmein Thorsten Meinl added a comment - The very same thing just happened here with the latest LTS 2.303.1 (for the first time I have to say). Anything we can do to help tracking down the issue?
          Hide
          sithmein Thorsten Meinl added a comment -

          The same problem occurred again, twice in two days now, right after we updated from a Java 8-based Jenkins to Java 11. This is a strong indication that is has something to do with Java 11 since we never had this issue before.

          Show
          sithmein Thorsten Meinl added a comment - The same problem occurred again, twice in two days now, right after we updated from a Java 8-based Jenkins to Java 11. This is a strong indication that is has something to do with Java 11 since we never had this issue before.
          Hide
          oxygenxo Andrey Babushkin added a comment -

          Thorsten Meinl hmm, I don't remember exactly, but there's a chance it has started when we've switched to JDK 11 official Docker images.
          We've switched to WebSocket connection recently, but I want to check if the issue will persist with JDK 8 Docker images. I'll post our observations here.

          Show
          oxygenxo Andrey Babushkin added a comment - Thorsten Meinl hmm, I don't remember exactly, but there's a chance it has started when we've switched to JDK 11 official Docker images. We've switched to WebSocket connection recently, but I want to check if the issue will persist with JDK 8 Docker images. I'll post our observations here.
          Hide
          oxygenxo Andrey Babushkin added a comment -

          Well, we've switched Docker image for our production Jenkins instance back to JDK 8. The issues hasn't appeared so far

          Show
          oxygenxo Andrey Babushkin added a comment - Well, we've switched Docker image for our production Jenkins instance back to JDK 8. The issues hasn't appeared so far
          Hide
          sithmein Thorsten Meinl added a comment -

          After switching back to Java 8 two weeks ago the problem did not occur any more. This is a very strong indication that something is wrong with the Java 11 version.

          Show
          sithmein Thorsten Meinl added a comment - After switching back to Java 8 two weeks ago the problem did not occur any more. This is a very strong indication that something is wrong with the Java 11 version.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            oxygenxo Andrey Babushkin
            Votes:
            7 Vote for this issue
            Watchers:
            12 Start watching this issue

              Dates

              Created:
              Updated: