Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-24213

Jenkins slaves repeatable disconnect and connect during startup, related to remoting/nio or swarm plugin

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • core, remoting
    • Jenkins LTS 1.565.1
      OpenJDK Java 1.7.0_65 on Debian wheezy 7.6, amd64
      Swarm plugin v1.16

      I'm not yet sure whether that's related to JENKINS-18781, and JENKINS-22758 seems to be solved already. I'm experiencing strange issues with the connection between Jenkins master and its slaves (connected using the swarm plugin) during startup. All the >25 slaves repeatable disconnect and automatically reconnect for quite some time:

      1. grep -c 'WARNING: Channel reader thread: .* terminated' ~log/jenkins/jenkins.log
        2882

      The data for the slaves at $JENKINS_URL/computer/ is missing for quite some time, but finally they persistent and then they seem to work fine. Was there some timeout decreased recently which might explain that issue?

      Quoting Jenkins master's log:

      Aug 12, 2014 12:43:08 AM jenkins.slaves.JnlpSlaveAgentProtocol$Handler$1 onClosed
      WARNING: Channel reader thread: docker1 for + docker1 terminated
      java.nio.channels.AsynchronousCloseException
      at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:205)
      at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:412)
      at hudson.remoting.SocketChannelStream$1.read(SocketChannelStream.java:33)
      at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
      at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
      at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
      at java.io.InputStream.read(InputStream.java:101)
      at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:81)
      at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82)
      at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2293)
      at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2586)
      at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2596)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1318)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
      at hudson.remoting.Command.readFrom(Command.java:92)
      at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:70)
      at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

      On the client that's in the logs:

      Aug 12, 2014 12:43:06 AM hudson.remoting.SynchronousCommandTransport$ReaderThread run
      SEVERE: I/O error in channel channel
      java.io.IOException: Unexpected termination of the channel
      at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
      Caused by: java.io.EOFException
      at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2598)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1318)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
      at hudson.remoting.Command.readFrom(Command.java:92)
      at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
      at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

          [JENKINS-24213] Jenkins slaves repeatable disconnect and connect during startup, related to remoting/nio or swarm plugin

          Michael Prokop created issue -
          Daniel Beck made changes -
          Labels New: remoting
          Kohsuke Kawaguchi made changes -
          Description Original: I'm not yet sure whether that's related to https://issues.jenkins-ci.org/browse/JENKINS-18781, and https://issues.jenkins-ci.org/browse/JENKINS-22758 seems to be solved already. I'm experiencing strange issues with the connection between Jenkins master and its slaves (connected using the swarm plugin) during startup. All the >25 slaves repeatable disconnect and automatically reconnect for quite some time:

          {quote}
          # grep -c 'WARNING: Channel reader thread: .* terminated' ~log/jenkins/jenkins.log
          2882
          {quote}

          The data for the slaves at $JENKINS_URL/computer/ is missing for quite some time, but finally they persistent and then they seem to work fine. Was there some timeout decreased recently which might explain that issue?

          Quoting Jenkins master's log:

          {quote}
          Aug 12, 2014 12:43:08 AM jenkins.slaves.JnlpSlaveAgentProtocol$Handler$1 onClosed
          WARNING: Channel reader thread: docker1 for + docker1 terminated
          java.nio.channels.AsynchronousCloseException
                  at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:205)
                  at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:412)
                  at hudson.remoting.SocketChannelStream$1.read(SocketChannelStream.java:33)
                  at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
                  at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
                  at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
                  at java.io.InputStream.read(InputStream.java:101)
                  at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:81)
                  at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82)
                  at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2293)
                  at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2586)
                  at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2596)
                  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1318)
                  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
                  at hudson.remoting.Command.readFrom(Command.java:92)
                  at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:70)
                  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
          {quote}

          On the client that's in the logs:

          {quote}
          Aug 12, 2014 12:43:06 AM hudson.remoting.SynchronousCommandTransport$ReaderThread run
          SEVERE: I/O error in channel channel
          java.io.IOException: Unexpected termination of the channel
                  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
          Caused by: java.io.EOFException
                  at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2598)
                  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1318)
                  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
                  at hudson.remoting.Command.readFrom(Command.java:92)
                  at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
                  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
          {quote}
          New: I'm not yet sure whether that's related to JENKINS-18781, and JENKINS-22758 seems to be solved already. I'm experiencing strange issues with the connection between Jenkins master and its slaves (connected using the swarm plugin) during startup. All the >25 slaves repeatable disconnect and automatically reconnect for quite some time:

          {quote}
          # grep -c 'WARNING: Channel reader thread: .* terminated' ~log/jenkins/jenkins.log
          2882
          {quote}

          The data for the slaves at $JENKINS_URL/computer/ is missing for quite some time, but finally they persistent and then they seem to work fine. Was there some timeout decreased recently which might explain that issue?

          Quoting Jenkins master's log:

          {quote}
          Aug 12, 2014 12:43:08 AM jenkins.slaves.JnlpSlaveAgentProtocol$Handler$1 onClosed
          WARNING: Channel reader thread: docker1 for + docker1 terminated
          java.nio.channels.AsynchronousCloseException
                  at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:205)
                  at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:412)
                  at hudson.remoting.SocketChannelStream$1.read(SocketChannelStream.java:33)
                  at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
                  at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
                  at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
                  at java.io.InputStream.read(InputStream.java:101)
                  at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:81)
                  at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82)
                  at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2293)
                  at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2586)
                  at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2596)
                  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1318)
                  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
                  at hudson.remoting.Command.readFrom(Command.java:92)
                  at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:70)
                  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
          {quote}

          On the client that's in the logs:

          {quote}
          Aug 12, 2014 12:43:06 AM hudson.remoting.SynchronousCommandTransport$ReaderThread run
          SEVERE: I/O error in channel channel
          java.io.IOException: Unexpected termination of the channel
                  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
          Caused by: java.io.EOFException
                  at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2598)
                  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1318)
                  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
                  at hudson.remoting.Command.readFrom(Command.java:92)
                  at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
                  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
          {quote}
          R. Tyler Croy made changes -
          Workflow Original: JNJira [ 157093 ] New: JNJira + In-Review [ 179505 ]
          Oleg Nenashev made changes -
          Component/s New: remoting [ 15489 ]

            Unassigned Unassigned
            mika Michael Prokop
            Votes:
            5 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated: