I'm not yet sure whether that's related to JENKINS-18781, and JENKINS-22758 seems to be solved already. I'm experiencing strange issues with the connection between Jenkins master and its slaves (connected using the swarm plugin) during startup. All the >25 slaves repeatable disconnect and automatically reconnect for quite some time:
- grep -c 'WARNING: Channel reader thread: .* terminated' ~log/jenkins/jenkins.log
2882
The data for the slaves at $JENKINS_URL/computer/ is missing for quite some time, but finally they persistent and then they seem to work fine. Was there some timeout decreased recently which might explain that issue?
Quoting Jenkins master's log:
Aug 12, 2014 12:43:08 AM jenkins.slaves.JnlpSlaveAgentProtocol$Handler$1 onClosed
WARNING: Channel reader thread: docker1 for + docker1 terminated
java.nio.channels.AsynchronousCloseException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:205)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:412)
at hudson.remoting.SocketChannelStream$1.read(SocketChannelStream.java:33)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at java.io.InputStream.read(InputStream.java:101)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:81)
at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82)
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2293)
at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2586)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2596)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1318)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at hudson.remoting.Command.readFrom(Command.java:92)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:70)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
On the client that's in the logs:
Aug 12, 2014 12:43:06 AM hudson.remoting.SynchronousCommandTransport$ReaderThread run
SEVERE: I/O error in channel channel
java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2598)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1318)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at hudson.remoting.Command.readFrom(Command.java:92)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
[JENKINS-24213] Jenkins slaves repeatable disconnect and connect during startup, related to remoting/nio or swarm plugin
Labels | New: remoting |
Description |
Original:
I'm not yet sure whether that's related to https://issues.jenkins-ci.org/browse/JENKINS-18781, and https://issues.jenkins-ci.org/browse/JENKINS-22758 seems to be solved already. I'm experiencing strange issues with the connection between Jenkins master and its slaves (connected using the swarm plugin) during startup. All the >25 slaves repeatable disconnect and automatically reconnect for quite some time: {quote} # grep -c 'WARNING: Channel reader thread: .* terminated' ~log/jenkins/jenkins.log 2882 {quote} The data for the slaves at $JENKINS_URL/computer/ is missing for quite some time, but finally they persistent and then they seem to work fine. Was there some timeout decreased recently which might explain that issue? Quoting Jenkins master's log: {quote} Aug 12, 2014 12:43:08 AM jenkins.slaves.JnlpSlaveAgentProtocol$Handler$1 onClosed WARNING: Channel reader thread: docker1 for + docker1 terminated java.nio.channels.AsynchronousCloseException at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:205) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:412) at hudson.remoting.SocketChannelStream$1.read(SocketChannelStream.java:33) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) at java.io.InputStream.read(InputStream.java:101) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:81) at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82) at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2293) at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2586) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2596) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1318) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:70) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) {quote} On the client that's in the logs: {quote} Aug 12, 2014 12:43:06 AM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: I/O error in channel channel java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2598) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1318) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) {quote} |
New:
I'm not yet sure whether that's related to JENKINS-18781, and {quote} # grep -c 'WARNING: Channel reader thread: .* terminated' ~log/jenkins/jenkins.log 2882 {quote} The data for the slaves at $JENKINS_URL/computer/ is missing for quite some time, but finally they persistent and then they seem to work fine. Was there some timeout decreased recently which might explain that issue? Quoting Jenkins master's log: {quote} Aug 12, 2014 12:43:08 AM jenkins.slaves.JnlpSlaveAgentProtocol$Handler$1 onClosed WARNING: Channel reader thread: docker1 for + docker1 terminated java.nio.channels.AsynchronousCloseException at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:205) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:412) at hudson.remoting.SocketChannelStream$1.read(SocketChannelStream.java:33) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) at java.io.InputStream.read(InputStream.java:101) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:81) at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82) at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2293) at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2586) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2596) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1318) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:70) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) {quote} On the client that's in the logs: {quote} Aug 12, 2014 12:43:06 AM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: I/O error in channel channel java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2598) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1318) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) {quote} |
Workflow | Original: JNJira [ 157093 ] | New: JNJira + In-Review [ 179505 ] |
Component/s | New: remoting [ 15489 ] |