[JENKINS-48850] Random java.io.IOException: Unexpected termination of the channel

Rick Liu created issue - 2018-01-08 21:18

Rick Liu made changes - 2018-01-08 21:22

Description

Original: Related to: ~~JENKINS-25858~~ and ~~JENKINS-48810~~

Per suggestion from [~oleg_nenashev],
I'm openning a separate bug ticket for further investigation.

h2. Jenkins Server log:
{code}
Dec 21, 2017 12:17:09 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
SEVERE: I/O error in channel jenkins-smoke-slave03(192.168.100.94)
java.io.IOException: Unexpected termination of the channel
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
Caused by: java.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
        at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
        at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
{code}

h3. Jenkins Slave log:
{code:java}
Dec 21, 2017 12:15:09 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 381.9±905.3/sec; rate(5min) = 363.6±923.4/sec; rate(15min) = 335.3±927.4/sec; rate(total) = 100.3±521.0/sec; N = 35,086
Dec 21, 2017 12:16:09 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 272.0±705.3/sec; rate(5min) = 324.8±863.5/sec; rate(15min) = 322.8±905.9/sec; rate(total) = 100.3±521.0/sec; N = 35,098
Dec 21, 2017 12:17:09 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 321.9±768.9/sec; rate(5min) = 333.2±865.8/sec; rate(15min) = 326.3±905.0/sec; rate(total) = 100.4±521.2/sec; N = 35,110
ERROR: Connection terminated
ESC[8mha:////4Cm+u8BY/EgsbhzNlnUfOXWprV5tRETZDv4u6647BaROAAAAVx+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0mV3NWzufebKBsTA0NFEYMUmgZnCA1SyAABjCCFBQC2xNaiYAAAAA==ESC[0mjava.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
        at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
        at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
ERROR: Socket connection to SSH server was lost
ESC[8mha:////4Cm+u8BY/EgsbhzNlnUfOXWprV5tRETZDv4u6647BaROAAAAVx+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0mV3NWzufebKBsTA0NFEYMUmgZnCA1SyAABjCCFBQC2xNaiYAAAAA==ESC[0mjava.io.IOException: Peer sent DISCONNECT message (reason code 2): Packet corrupt
        at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:779)
        at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:502)
        at java.lang.Thread.run(Thread.java:748)
Slave JVM has not reported exit code before the socket was lost
[12/21/17 12:17:09] [SSH] Connection closed.
{code}

This "*Unexpected termination of the channel*" has happened everyday (3 days in a roll) to any of slaves randomly since I updated the Jenkins core and all the plugins to the latest on *Dec 19. 2017*.

The previous Jenkins core and plugin were updated back on *April 2017*:
{quote}
Jenkins Core: 2.46.2
SSH-slave puglin: 1.16
{quote}

Due to the more than usual of the random "*Unexpected termination of the channel*",
I downgraded Jenkins Core and SSH-slave plugin to:
{quote}
Jenkins Core: 2.60.3 (which remoting should be the same as 2.46.2 based on changelog)
SSH-slave puglin: 1.16
{quote}

New: Related to: ~~JENKINS-25858~~ and ~~JENKINS-48810~~

Per suggestion from [~oleg_nenashev],
I'm openning a separate bug ticket for further investigation.

h2. Jenkins Server log:
{code}
Dec 21, 2017 12:17:09 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
SEVERE: I/O error in channel jenkins-smoke-slave03(192.168.100.94)
java.io.IOException: Unexpected termination of the channel
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
Caused by: java.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
        at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
        at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
{code}

h3. Jenkins Slave log:
{code:java}
Dec 21, 2017 12:15:09 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 381.9±905.3/sec; rate(5min) = 363.6±923.4/sec; rate(15min) = 335.3±927.4/sec; rate(total) = 100.3±521.0/sec; N = 35,086
Dec 21, 2017 12:16:09 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 272.0±705.3/sec; rate(5min) = 324.8±863.5/sec; rate(15min) = 322.8±905.9/sec; rate(total) = 100.3±521.0/sec; N = 35,098
Dec 21, 2017 12:17:09 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 321.9±768.9/sec; rate(5min) = 333.2±865.8/sec; rate(15min) = 326.3±905.0/sec; rate(total) = 100.4±521.2/sec; N = 35,110
ERROR: Connection terminated
ESC[8mha:////4Cm+u8BY/EgsbhzNlnUfOXWprV5tRETZDv4u6647BaROAAAAVx+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0mV3NWzufebKBsTA0NFEYMUmgZnCA1SyAABjCCFBQC2xNaiYAAAAA==ESC[0mjava.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
        at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
        at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
ERROR: Socket connection to SSH server was lost
ESC[8mha:////4Cm+u8BY/EgsbhzNlnUfOXWprV5tRETZDv4u6647BaROAAAAVx+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0mV3NWzufebKBsTA0NFEYMUmgZnCA1SyAABjCCFBQC2xNaiYAAAAA==ESC[0mjava.io.IOException: Peer sent DISCONNECT message (reason code 2): Packet corrupt
        at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:779)
        at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:502)
        at java.lang.Thread.run(Thread.java:748)
Slave JVM has not reported exit code before the socket was lost
[12/21/17 12:17:09] [SSH] Connection closed.
{code}

This "*Unexpected termination of the channel*" has happened everyday (3 days in a roll) to any of slaves randomly since I updated the Jenkins core and all the plugins to the latest on *Dec 19. 2017*.

The previous Jenkins core and plugin were updated back on *April 2017*:
{quote}
Jenkins Core: 2.46.2
SSH-slave puglin: 1.16
{quote}

Due to the more than usual of the random "*Unexpected termination of the channel*",
I downgraded Jenkins Core and SSH-slave plugin to:
{quote}
Jenkins Core: 2.60.3 (which remoting should be the same as 2.46.2 based on changelog)
SSH-slave puglin: 1.16
{quote}

The issue has been eased since the downgrade,
but the random "*Unexpected termination of the channel*" still happened a couple time so far.

Rick Liu made changes - 2018-01-08 21:37

Description

Original: Related to: ~~JENKINS-25858~~ and ~~JENKINS-48810~~

Per suggestion from [~oleg_nenashev],
I'm openning a separate bug ticket for further investigation.

h2. Jenkins Server log:
{code}
Dec 21, 2017 12:17:09 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
SEVERE: I/O error in channel jenkins-smoke-slave03(192.168.100.94)
java.io.IOException: Unexpected termination of the channel
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
Caused by: java.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
        at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
        at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
{code}

h3. Jenkins Slave log:
{code:java}
Dec 21, 2017 12:15:09 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 381.9±905.3/sec; rate(5min) = 363.6±923.4/sec; rate(15min) = 335.3±927.4/sec; rate(total) = 100.3±521.0/sec; N = 35,086
Dec 21, 2017 12:16:09 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 272.0±705.3/sec; rate(5min) = 324.8±863.5/sec; rate(15min) = 322.8±905.9/sec; rate(total) = 100.3±521.0/sec; N = 35,098
Dec 21, 2017 12:17:09 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 321.9±768.9/sec; rate(5min) = 333.2±865.8/sec; rate(15min) = 326.3±905.0/sec; rate(total) = 100.4±521.2/sec; N = 35,110
ERROR: Connection terminated
ESC[8mha:////4Cm+u8BY/EgsbhzNlnUfOXWprV5tRETZDv4u6647BaROAAAAVx+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0mV3NWzufebKBsTA0NFEYMUmgZnCA1SyAABjCCFBQC2xNaiYAAAAA==ESC[0mjava.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
        at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
        at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
ERROR: Socket connection to SSH server was lost
ESC[8mha:////4Cm+u8BY/EgsbhzNlnUfOXWprV5tRETZDv4u6647BaROAAAAVx+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0mV3NWzufebKBsTA0NFEYMUmgZnCA1SyAABjCCFBQC2xNaiYAAAAA==ESC[0mjava.io.IOException: Peer sent DISCONNECT message (reason code 2): Packet corrupt
        at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:779)
        at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:502)
        at java.lang.Thread.run(Thread.java:748)
Slave JVM has not reported exit code before the socket was lost
[12/21/17 12:17:09] [SSH] Connection closed.
{code}

This "*Unexpected termination of the channel*" has happened everyday (3 days in a roll) to any of slaves randomly since I updated the Jenkins core and all the plugins to the latest on *Dec 19. 2017*.

The previous Jenkins core and plugin were updated back on *April 2017*:
{quote}
Jenkins Core: 2.46.2
SSH-slave puglin: 1.16
{quote}

Due to the more than usual of the random "*Unexpected termination of the channel*",
I downgraded Jenkins Core and SSH-slave plugin to:
{quote}
Jenkins Core: 2.60.3 (which remoting should be the same as 2.46.2 based on changelog)
SSH-slave puglin: 1.16
{quote}

The issue has been eased since the downgrade,
but the random "*Unexpected termination of the channel*" still happened a couple time so far.

New: Related to: ~~JENKINS-25858~~ and ~~JENKINS-48810~~

Per suggestion from [~oleg_nenashev],
I'm openning a separate bug ticket for further investigation.

h2. Jenkins Server log:
{code}
Dec 21, 2017 12:17:09 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
SEVERE: I/O error in channel jenkins-smoke-slave03(192.168.100.94)
java.io.IOException: Unexpected termination of the channel
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
Caused by: java.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
        at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
        at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
{code}

h3. Jenkins Slave log:
{code:java}
Dec 21, 2017 12:15:09 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 381.9±905.3/sec; rate(5min) = 363.6±923.4/sec; rate(15min) = 335.3±927.4/sec; rate(total) = 100.3±521.0/sec; N = 35,086
Dec 21, 2017 12:16:09 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 272.0±705.3/sec; rate(5min) = 324.8±863.5/sec; rate(15min) = 322.8±905.9/sec; rate(total) = 100.3±521.0/sec; N = 35,098
Dec 21, 2017 12:17:09 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 321.9±768.9/sec; rate(5min) = 333.2±865.8/sec; rate(15min) = 326.3±905.0/sec; rate(total) = 100.4±521.2/sec; N = 35,110
ERROR: Connection terminated
ESC[8mha:////4Cm+u8BY/EgsbhzNlnUfOXWprV5tRETZDv4u6647BaROAAAAVx+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0mV3NWzufebKBsTA0NFEYMUmgZnCA1SyAABjCCFBQC2xNaiYAAAAA==ESC[0mjava.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
        at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
        at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
ERROR: Socket connection to SSH server was lost
ESC[8mha:////4Cm+u8BY/EgsbhzNlnUfOXWprV5tRETZDv4u6647BaROAAAAVx+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0mV3NWzufebKBsTA0NFEYMUmgZnCA1SyAABjCCFBQC2xNaiYAAAAA==ESC[0mjava.io.IOException: Peer sent DISCONNECT message (reason code 2): Packet corrupt
        at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:779)
        at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:502)
        at java.lang.Thread.run(Thread.java:748)
Slave JVM has not reported exit code before the socket was lost
[12/21/17 12:17:09] [SSH] Connection closed.
{code}

This "*Unexpected termination of the channel*" has happened everyday (3 days in a roll) to any of slaves randomly since I updated the Jenkins core and all the plugins to the latest on *Dec 19. 2017*.

The previous Jenkins core and plugin were updated back on *April 2017*:
{quote}
Jenkins Core: 2.46.2
SSH-slave puglin: 1.16
{quote}

Due to the more than usual of the random "*Unexpected termination of the channel*",
on "*Dec 22. 2017*" I downgraded Jenkins Core and SSH-slave plugin to:
{quote}
Jenkins Core: 2.60.3 (which remoting should be the same as 2.46.2 based on changelog)
SSH-slave puglin: 1.16
{quote}

The issue has been eased since the downgrade,
but the random "*Unexpected termination of the channel*" still happened a couple time so far.

Rick Liu added a comment - 2018-01-08 22:01 - edited

After the downgrade,

Jenkins Core: 2.60.3
SSH-slave puglin: 1.16

one randome SSH disconnection again on "Jan 04, 2018 @ 12:39:45 PM":

Server Log:

Jan 04, 2018 12:37:16 PM hudson.model.Run execute
INFO: FortiOS_Automated_Test_6.0/FOS_Release/PLATFORM=FGT_81E,TYPE=cli-syntax-collection #49 main build action completed: SUCCESS
Jan 04, 2018 12:38:14 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
INFO: TinderBox/FortiOS/Build_Steps/trunk_Chroot_Build #76057 completed: SUCCESS
Jan 04, 2018 12:38:14 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
INFO: TinderBox/FortiOS/Build_Steps/trunk_Chroot_Build #76046 completed: SUCCESS
Jan 04, 2018 12:38:21 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
INFO: TinderBox/FortiOS/Build_Steps/trunk_Chroot_Build #76036 completed: SUCCESS
Jan 04, 2018 12:38:31 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
INFO: TinderBox/FortiOS/Build_Steps/trunk_Chroot_Build #76056 completed: SUCCESS
Jan 04, 2018 12:38:39 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
INFO: TinderBox/FortiOS/Build_Steps/trunk_Chroot_Build #76059 completed: SUCCESS
Jan 04, 2018 12:39:03 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
INFO: FortiRobot/pipeline_FortiRobot_Test #14366 completed: SUCCESS
Jan 04, 2018 12:39:05 PM hudson.model.Run execute
INFO: FortiOS_Automated_Test_6.0/FOS_Release/PLATFORM=FGT_61E,TYPE=cli-syntax-collection #49 main build action completed: SUCCESS
Jan 04, 2018 12:39:11 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
INFO: FGT_VM_QA/low_level_job #2631 completed: SUCCESS
Jan 04, 2018 12:39:45 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
SEVERE: I/O error in channel jenkins-smoke-slave01(172.16.182.121)
java.net.SocketException: Broken pipe (Write failed)
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
        at com.trilead.ssh2.crypto.cipher.CipherOutputStream.flush(CipherOutputStream.java:75)
        at com.trilead.ssh2.transport.TransportConnection.sendMessage(TransportConnection.java:193)
        at com.trilead.ssh2.transport.TransportConnection.sendMessage(TransportConnection.java:107)
        at com.trilead.ssh2.transport.TransportManager.sendMessage(TransportManager.java:690)
        at com.trilead.ssh2.channel.Channel.freeupWindow(Channel.java:407)
        at com.trilead.ssh2.channel.Channel.freeupWindow(Channel.java:347)
        at com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:943)
        at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58)
        at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:79)
        at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:86)
        at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72)
        at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103)
        at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
        at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)

Jan 04, 2018 12:39:50 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
INFO: FortiRobot/pipeline_FortiRobot_Test #14359 completed: SUCCESS
Jan 04, 2018 12:39:51 PM hudson.model.Run execute
INFO: FortiOS_Automated_Test_6.0/FOS_Release/PLATFORM=FGT_60E,TYPE=cli-syntax-collection #49 main build action completed: SUCCESS
Jan 04, 2018 12:40:29 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
INFO: TinderBox/FortiOS/Build_Steps/trunk_Chroot_Build #76035 completed: SUCCESS
Jan 04, 2018 12:40:35 PM org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$4$1 load
WARNING: took 375ms to load/not load groovy.util.BUILD_URL from classLoader hudson.PluginManager$UberClassLoader
Jan 04, 2018 12:40:35 PM org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$4$1 load
WARNING: took 323ms to load/not load MINOR_VERSION from classLoader hudson.PluginManager$UberClassLoader
Jan 04, 2018 12:40:36 PM org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$4$1 load
WARNING: took 294ms to load/not load java.util.WorkflowScript$EXECUTOR_NUMBER from classLoader hudson.PluginManager$UberClassLoader
Jan 04, 2018 12:40:36 PM org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$4$1 load
WARNING: took 317ms to load/not load groovy.lang.GroovyObject$groovy$lang$CHROOT_NAME from classLoader hudson.PluginManager$UberClassLoader
Jan 04, 2018 12:40:37 PM org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$4$1 load
WARNING: took 331ms to load/not load java.io.Serializable$java$lang$PATCH_VERSION from classLoader hudson.PluginManager$UberClassLoader
Jan 04, 2018 12:40:45 PM hudson.slaves.SlaveComputer tryReconnect
INFO: Attempting to reconnect jenkins-smoke-slave01(172.16.182.121)
[01/04/18 12:40:51] SSH Launch of jenkins-smoke-slave01(172.16.182.121) on 172.16.182.121 completed in 6,006 ms
Jan 04, 2018 12:41:03 PM hudson.model.Run execute
INFO: FortiRobot/Performance_Tests/Jenkins_Performance_Staging #1946 main build action completed: SUCCESS

Slave Log:

Jan 04, 2018 12:34:45 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 257.3±1023.6/sec; rate(5min) = 346.4±1058.6/sec; rate(15min) = 364.6±1013.0/sec; rate(total) = 117.6±534.6/sec; N = 16,077
Jan 04, 2018 12:35:45 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 318.8±816.5/sec; rate(5min) = 352.9±1001.2/sec; rate(15min) = 365.9±995.2/sec; rate(total) = 117.8±534.8/sec; N = 16,089
Jan 04, 2018 12:36:45 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 415.6±921.6/sec; rate(5min) = 362.4±987.4/sec; rate(15min) = 367.7±990.0/sec; rate(total) = 118.0±535.2/sec; N = 16,101
Jan 04, 2018 12:37:45 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 491.1±1299.9/sec; rate(5min) = 371.4±1054.2/sec; rate(15min) = 369.3±1011.6/sec; rate(total) = 118.2±536.2/sec; N = 16,113
Jan 04, 2018 12:38:45 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 545.2±1291.1/sec; rate(5min) = 380.2±1066.8/sec; rate(15min) = 371.0±1017.1/sec; rate(total) = 118.4±536.8/sec; N = 16,125
Jan 04, 2018 12:39:45 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 458.2±1158.7/sec; rate(5min) = 364.5±1043.4/sec; rate(15min) = 365.0±1010.3/sec; rate(total) = 118.5±537.2/sec; N = 16,137
ERROR: Connection terminated
ha:////4Cm+u8BY/EgsbhzNlnUfOXWprV5tRETZDv4u6647BaROAAAAVx+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0mV3NWzufebKBsTA0NFEYMUmgZnCA1SyAABjCCFBQC2xNaiYAAAAA==java.net.SocketException: Broken pipe (Write failed)
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
	at com.trilead.ssh2.crypto.cipher.CipherOutputStream.flush(CipherOutputStream.java:75)
	at com.trilead.ssh2.transport.TransportConnection.sendMessage(TransportConnection.java:193)
	at com.trilead.ssh2.transport.TransportConnection.sendMessage(TransportConnection.java:107)
	at com.trilead.ssh2.transport.TransportManager.sendMessage(TransportManager.java:690)
	at com.trilead.ssh2.channel.Channel.freeupWindow(Channel.java:407)
	at com.trilead.ssh2.channel.Channel.freeupWindow(Channel.java:347)
	at com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:943)
	at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58)
	at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:79)
	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:86)
	at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72)
	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103)
	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
ERROR: Socket connection to SSH server was lost
ha:////4Cm+u8BY/EgsbhzNlnUfOXWprV5tRETZDv4u6647BaROAAAAVx+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0mV3NWzufebKBsTA0NFEYMUmgZnCA1SyAABjCCFBQC2xNaiYAAAAA==java.net.SocketException: Broken pipe (Write failed)
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
	at com.trilead.ssh2.crypto.cipher.CipherOutputStream.flush(CipherOutputStream.java:75)
	at com.trilead.ssh2.transport.TransportConnection.sendMessage(TransportConnection.java:193)
	at com.trilead.ssh2.transport.TransportConnection.sendMessage(TransportConnection.java:107)
	at com.trilead.ssh2.transport.TransportManager.sendMessage(TransportManager.java:690)
	at com.trilead.ssh2.channel.Channel.freeupWindow(Channel.java:407)
	at com.trilead.ssh2.channel.Channel.freeupWindow(Channel.java:347)
	at com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:943)
	at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58)
	at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:79)
	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:86)
	at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72)
	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103)
	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
Slave JVM has not reported exit code before the socket was lost
[01/04/18 12:39:48] [SSH] Connection closed.

Rick Liu added a comment - 2018-01-08 22:01 - edited After the downgrade, Jenkins Core: 2.60.3 SSH-slave puglin: 1.16 one randome SSH disconnection again on " Jan 04, 2018 @ 12:39:45 PM ": Server Log: Jan 04, 2018 12:37:16 PM hudson.model.Run execute INFO: FortiOS_Automated_Test_6.0/FOS_Release/PLATFORM=FGT_81E,TYPE=cli-syntax-collection #49 main build action completed: SUCCESS Jan 04, 2018 12:38:14 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish INFO: TinderBox/FortiOS/Build_Steps/trunk_Chroot_Build #76057 completed: SUCCESS Jan 04, 2018 12:38:14 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish INFO: TinderBox/FortiOS/Build_Steps/trunk_Chroot_Build #76046 completed: SUCCESS Jan 04, 2018 12:38:21 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish INFO: TinderBox/FortiOS/Build_Steps/trunk_Chroot_Build #76036 completed: SUCCESS Jan 04, 2018 12:38:31 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish INFO: TinderBox/FortiOS/Build_Steps/trunk_Chroot_Build #76056 completed: SUCCESS Jan 04, 2018 12:38:39 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish INFO: TinderBox/FortiOS/Build_Steps/trunk_Chroot_Build #76059 completed: SUCCESS Jan 04, 2018 12:39:03 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish INFO: FortiRobot/pipeline_FortiRobot_Test #14366 completed: SUCCESS Jan 04, 2018 12:39:05 PM hudson.model.Run execute INFO: FortiOS_Automated_Test_6.0/FOS_Release/PLATFORM=FGT_61E,TYPE=cli-syntax-collection #49 main build action completed: SUCCESS Jan 04, 2018 12:39:11 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish INFO: FGT_VM_QA/low_level_job #2631 completed: SUCCESS Jan 04, 2018 12:39:45 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: I/O error in channel jenkins-smoke-slave01(172.16.182.121) java.net.SocketException: Broken pipe (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at com.trilead.ssh2.crypto.cipher.CipherOutputStream.flush(CipherOutputStream.java:75) at com.trilead.ssh2.transport.TransportConnection.sendMessage(TransportConnection.java:193) at com.trilead.ssh2.transport.TransportConnection.sendMessage(TransportConnection.java:107) at com.trilead.ssh2.transport.TransportManager.sendMessage(TransportManager.java:690) at com.trilead.ssh2.channel.Channel.freeupWindow(Channel.java:407) at com.trilead.ssh2.channel.Channel.freeupWindow(Channel.java:347) at com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:943) at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58) at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:79) at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:86) at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72) at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103) at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59) Jan 04, 2018 12:39:50 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish INFO: FortiRobot/pipeline_FortiRobot_Test #14359 completed: SUCCESS Jan 04, 2018 12:39:51 PM hudson.model.Run execute INFO: FortiOS_Automated_Test_6.0/FOS_Release/PLATFORM=FGT_60E,TYPE=cli-syntax-collection #49 main build action completed: SUCCESS Jan 04, 2018 12:40:29 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish INFO: TinderBox/FortiOS/Build_Steps/trunk_Chroot_Build #76035 completed: SUCCESS Jan 04, 2018 12:40:35 PM org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$4$1 load WARNING: took 375ms to load/not load groovy.util.BUILD_URL from classLoader hudson.PluginManager$UberClassLoader Jan 04, 2018 12:40:35 PM org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$4$1 load WARNING: took 323ms to load/not load MINOR_VERSION from classLoader hudson.PluginManager$UberClassLoader Jan 04, 2018 12:40:36 PM org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$4$1 load WARNING: took 294ms to load/not load java.util.WorkflowScript$EXECUTOR_NUMBER from classLoader hudson.PluginManager$UberClassLoader Jan 04, 2018 12:40:36 PM org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$4$1 load WARNING: took 317ms to load/not load groovy.lang.GroovyObject$groovy$lang$CHROOT_NAME from classLoader hudson.PluginManager$UberClassLoader Jan 04, 2018 12:40:37 PM org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxResolvingClassLoader$4$1 load WARNING: took 331ms to load/not load java.io.Serializable$java$lang$PATCH_VERSION from classLoader hudson.PluginManager$UberClassLoader Jan 04, 2018 12:40:45 PM hudson.slaves.SlaveComputer tryReconnect INFO: Attempting to reconnect jenkins-smoke-slave01(172.16.182.121) [01/04/18 12:40:51] SSH Launch of jenkins-smoke-slave01(172.16.182.121) on 172.16.182.121 completed in 6,006 ms Jan 04, 2018 12:41:03 PM hudson.model.Run execute INFO: FortiRobot/Performance_Tests/Jenkins_Performance_Staging #1946 main build action completed: SUCCESS Slave Log: Jan 04, 2018 12:34:45 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats INFO: rate(1min) = 257.3±1023.6/sec; rate(5min) = 346.4±1058.6/sec; rate(15min) = 364.6±1013.0/sec; rate(total) = 117.6±534.6/sec; N = 16,077 Jan 04, 2018 12:35:45 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats INFO: rate(1min) = 318.8±816.5/sec; rate(5min) = 352.9±1001.2/sec; rate(15min) = 365.9±995.2/sec; rate(total) = 117.8±534.8/sec; N = 16,089 Jan 04, 2018 12:36:45 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats INFO: rate(1min) = 415.6±921.6/sec; rate(5min) = 362.4±987.4/sec; rate(15min) = 367.7±990.0/sec; rate(total) = 118.0±535.2/sec; N = 16,101 Jan 04, 2018 12:37:45 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats INFO: rate(1min) = 491.1±1299.9/sec; rate(5min) = 371.4±1054.2/sec; rate(15min) = 369.3±1011.6/sec; rate(total) = 118.2±536.2/sec; N = 16,113 Jan 04, 2018 12:38:45 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats INFO: rate(1min) = 545.2±1291.1/sec; rate(5min) = 380.2±1066.8/sec; rate(15min) = 371.0±1017.1/sec; rate(total) = 118.4±536.8/sec; N = 16,125 Jan 04, 2018 12:39:45 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats INFO: rate(1min) = 458.2±1158.7/sec; rate(5min) = 364.5±1043.4/sec; rate(15min) = 365.0±1010.3/sec; rate(total) = 118.5±537.2/sec; N = 16,137 ERROR: Connection terminated ha: ////4Cm+u8BY/EgsbhzNlnUfOXWprV5tRETZDv4u6647BaROAAAAVx+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0mV3NWzufebKBsTA0NFEYMUmgZnCA1SyAABjCCFBQC2xNaiYAAAAA==java.net.SocketException: Broken pipe (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at com.trilead.ssh2.crypto.cipher.CipherOutputStream.flush(CipherOutputStream.java:75) at com.trilead.ssh2.transport.TransportConnection.sendMessage(TransportConnection.java:193) at com.trilead.ssh2.transport.TransportConnection.sendMessage(TransportConnection.java:107) at com.trilead.ssh2.transport.TransportManager.sendMessage(TransportManager.java:690) at com.trilead.ssh2.channel.Channel.freeupWindow(Channel.java:407) at com.trilead.ssh2.channel.Channel.freeupWindow(Channel.java:347) at com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:943) at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58) at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:79) at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:86) at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72) at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103) at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59) ERROR: Socket connection to SSH server was lost ha: ////4Cm+u8BY/EgsbhzNlnUfOXWprV5tRETZDv4u6647BaROAAAAVx+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0mV3NWzufebKBsTA0NFEYMUmgZnCA1SyAABjCCFBQC2xNaiYAAAAA==java.net.SocketException: Broken pipe (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at com.trilead.ssh2.crypto.cipher.CipherOutputStream.flush(CipherOutputStream.java:75) at com.trilead.ssh2.transport.TransportConnection.sendMessage(TransportConnection.java:193) at com.trilead.ssh2.transport.TransportConnection.sendMessage(TransportConnection.java:107) at com.trilead.ssh2.transport.TransportManager.sendMessage(TransportManager.java:690) at com.trilead.ssh2.channel.Channel.freeupWindow(Channel.java:407) at com.trilead.ssh2.channel.Channel.freeupWindow(Channel.java:347) at com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:943) at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58) at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:79) at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:86) at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72) at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103) at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59) Slave JVM has not reported exit code before the socket was lost [01/04/18 12:39:48] [SSH] Connection closed.

Rick Liu made changes - 2018-01-08 22:04

Attachment

New: 2018-0104-jenkins.log.tgz [ 41077 ]

Rick Liu added a comment - 2018-01-09 00:02

Just happened again 30 mins ago~

Jenkins Core: 2.60.3
SSH-slave puglin: 1.16

On "01/08/18 15:07:02" and "01/08/18 15:39:29":

Server.log:

Jan 08, 2018 3:07:02 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
SEVERE: I/O error in channel jenkins-smoke-slave03(192.168.100.94)
java.io.IOException: Unexpected termination of the channel
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)
Caused by: java.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
        at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
        at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)

Jan 08, 2018 3:07:45 PM hudson.slaves.SlaveComputer tryReconnect
INFO: Attempting to reconnect jenkins-smoke-slave03(192.168.100.94)
[01/08/18 15:07:45] SSH Launch of jenkins-smoke-slave03(192.168.100.94) on 192.168.100.94 failed in 170 ms
Jan 08, 2018 3:09:07 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
INFO: TinderBox/FortiOS/Build_Steps/5.6_Chroot_Build #64101 completed: SUCCESS
Jan 08, 2018 3:09:45 PM hudson.slaves.SlaveComputer tryReconnect
INFO: Attempting to reconnect jenkins-smoke-slave03(192.168.100.94)
[01/08/18 15:10:11] SSH Launch of jenkins-smoke-slave03(192.168.100.94) on 192.168.100.94 completed in 25,748 ms

Jan 08, 2018 3:39:09 PM hudson.scm.SubversionSCM buildEnvironment
WARNING: no revision found corresponding to $svnurl; known: [https://scm-yvr.fortinet.com/svn/svnfos/FortiOS/trunk]
Jan 08, 2018 3:39:09 PM hudson.scm.SubversionSCM buildEnvironment
WARNING: no revision found corresponding to $svnurl; known: [https://scm-yvr.fortinet.com/svn/svnfos/FortiOS/trunk]
Jan 08, 2018 3:39:29 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
SEVERE: I/O error in channel jenkins-smoke-slave03(192.168.100.94)
java.io.IOException: Unexpected termination of the channel
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)
Caused by: java.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
        at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
        at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)

Jan 08, 2018 3:40:45 PM hudson.slaves.SlaveComputer tryReconnect
INFO: Attempting to reconnect jenkins-smoke-slave03(192.168.100.94)
Jan 08, 2018 3:41:02 PM hudson.model.Run execute
INFO: FortiRobot/Performance_Tests/Jenkins_Performance_Staging #1979 main build action completed: SUCCESS
[01/08/18 15:41:07] SSH Launch of jenkins-smoke-slave03(192.168.100.94) on 192.168.100.94 completed in 22,161 ms

Slave.log:

Jan 08, 2018 3:04:43 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 33.9±241.7/sec; rate(5min) = 102.7±461.3/sec; rate(15min) = 174.3±644.9/sec; rate(total) = 50.4±359.9/sec; N = 85,304
Jan 08, 2018 3:05:43 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 288.4±817.5/sec; rate(5min) = 141.6±560.3/sec; rate(15min) = 182.4±659.5/sec; rate(total) = 50.4±360.0/sec; N = 85,316
Jan 08, 2018 3:06:43 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 141.5±543.7/sec; rate(5min) = 131.2±522.5/sec; rate(15min) = 176.4±642.8/sec; rate(total) = 50.4±360.0/sec; N = 85,328
ERROR: Connection terminated
^[[8mha:////4Cm+u8BY/EgsbhzNlnUfOXWprV5tRETZDv4u6647BaROAAAAVx+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0mV3NWzufebKBsTA0NFEYMUmgZnCA1SyAABjCCFBQC2xNaiYAAAAA==^[[0mjava.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
        at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
        at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
Caused: java.io.IOException: Unexpected termination of the channel
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)
ERROR: Socket connection to SSH server was lost
^[[8mha:////4Cm+u8BY/EgsbhzNlnUfOXWprV5tRETZDv4u6647BaROAAAAVx+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0mV3NWzufebKBsTA0NFEYMUmgZnCA1SyAABjCCFBQC2xNaiYAAAAA==^[[0mjava.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:210)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at com.trilead.ssh2.crypto.cipher.CipherInputStream.fill_buffer(CipherInputStream.java:41)
        at com.trilead.ssh2.crypto.cipher.CipherInputStream.internal_read(CipherInputStream.java:52)
        at com.trilead.ssh2.crypto.cipher.CipherInputStream.getBlock(CipherInputStream.java:79)
        at com.trilead.ssh2.crypto.cipher.CipherInputStream.read(CipherInputStream.java:108)
        at com.trilead.ssh2.transport.TransportConnection.receiveMessage(TransportConnection.java:232)
        at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:706)
        at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:502)
        at java.lang.Thread.run(Thread.java:748)
Slave JVM has not reported exit code before the socket was lost
[01/08/18 15:07:02] [SSH] Connection closed.

[01/08/18 15:10:07] [SSH] Checking java version of java
[01/08/18 15:10:07] [SSH] java -version returned 1.8.0_141.
[01/08/18 15:10:07] [SSH] Starting sftp client.
[01/08/18 15:10:07] [SSH] Copying latest slave.jar...
[01/08/18 15:10:08] [SSH] Copied 719,269 bytes.
Expanded the channel window size to 4MB
[01/08/18 15:10:08] [SSH] Starting slave process: cd "/home/devops/jenkins_slave_robot" && java  -jar slave.jar
<===[JENKINS REMOTING CAPACITY]===>^@^@^@channel started
Slave.jar version: 3.7
This is a Unix agent
Evacuated stdout
Agent successfully connected and online
ERROR: Connection terminated
^[[8mha:////4Cm+u8BY/EgsbhzNlnUfOXWprV5tRETZDv4u6647BaROAAAAVx+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0mV3NWzufebKBsTA0NFEYMUmgZnCA1SyAABjCCFBQC2xNaiYAAAAA==^[[0mjava.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
        at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
        at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
Caused: java.io.IOException: Unexpected termination of the channel
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)
ERROR: Socket connection to SSH server was lost
^[[8mha:////4Cm+u8BY/EgsbhzNlnUfOXWprV5tRETZDv4u6647BaROAAAAVx+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0mV3NWzufebKBsTA0NFEYMUmgZnCA1SyAABjCCFBQC2xNaiYAAAAA==^[[0mjava.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:210)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at com.trilead.ssh2.crypto.cipher.CipherInputStream.fill_buffer(CipherInputStream.java:41)
        at com.trilead.ssh2.crypto.cipher.CipherInputStream.internal_read(CipherInputStream.java:52)
        at com.trilead.ssh2.crypto.cipher.CipherInputStream.getBlock(CipherInputStream.java:79)
        at com.trilead.ssh2.crypto.cipher.CipherInputStream.read(CipherInputStream.java:108)
        at com.trilead.ssh2.transport.TransportConnection.receiveMessage(TransportConnection.java:232)
        at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:706)
        at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:502)
        at java.lang.Thread.run(Thread.java:748)
Slave JVM has not reported exit code before the socket was lost
[01/08/18 15:39:29] [SSH] Connection closed.

Rick Liu added a comment - 2018-01-09 00:02 Just happened again 30 mins ago~ Jenkins Core: 2.60.3 SSH-slave puglin: 1.16 On " 01/08/18 15:07:02 " and " 01/08/18 15:39:29 ": Server.log: Jan 08, 2018 3:07:02 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: I/O error in channel jenkins-smoke-slave03(192.168.100.94) java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73) Caused by: java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59) Jan 08, 2018 3:07:45 PM hudson.slaves.SlaveComputer tryReconnect INFO: Attempting to reconnect jenkins-smoke-slave03(192.168.100.94) [01/08/18 15:07:45] SSH Launch of jenkins-smoke-slave03(192.168.100.94) on 192.168.100.94 failed in 170 ms Jan 08, 2018 3:09:07 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish INFO: TinderBox/FortiOS/Build_Steps/5.6_Chroot_Build #64101 completed: SUCCESS Jan 08, 2018 3:09:45 PM hudson.slaves.SlaveComputer tryReconnect INFO: Attempting to reconnect jenkins-smoke-slave03(192.168.100.94) [01/08/18 15:10:11] SSH Launch of jenkins-smoke-slave03(192.168.100.94) on 192.168.100.94 completed in 25,748 ms Jan 08, 2018 3:39:09 PM hudson.scm.SubversionSCM buildEnvironment WARNING: no revision found corresponding to $svnurl; known: [https: //scm-yvr.fortinet.com/svn/svnfos/FortiOS/trunk] Jan 08, 2018 3:39:09 PM hudson.scm.SubversionSCM buildEnvironment WARNING: no revision found corresponding to $svnurl; known: [https: //scm-yvr.fortinet.com/svn/svnfos/FortiOS/trunk] Jan 08, 2018 3:39:29 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: I/O error in channel jenkins-smoke-slave03(192.168.100.94) java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73) Caused by: java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59) Jan 08, 2018 3:40:45 PM hudson.slaves.SlaveComputer tryReconnect INFO: Attempting to reconnect jenkins-smoke-slave03(192.168.100.94) Jan 08, 2018 3:41:02 PM hudson.model.Run execute INFO: FortiRobot/Performance_Tests/Jenkins_Performance_Staging #1979 main build action completed: SUCCESS [01/08/18 15:41:07] SSH Launch of jenkins-smoke-slave03(192.168.100.94) on 192.168.100.94 completed in 22,161 ms Slave.log: Jan 08, 2018 3:04:43 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats INFO: rate(1min) = 33.9±241.7/sec; rate(5min) = 102.7±461.3/sec; rate(15min) = 174.3±644.9/sec; rate(total) = 50.4±359.9/sec; N = 85,304 Jan 08, 2018 3:05:43 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats INFO: rate(1min) = 288.4±817.5/sec; rate(5min) = 141.6±560.3/sec; rate(15min) = 182.4±659.5/sec; rate(total) = 50.4±360.0/sec; N = 85,316 Jan 08, 2018 3:06:43 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats INFO: rate(1min) = 141.5±543.7/sec; rate(5min) = 131.2±522.5/sec; rate(15min) = 176.4±642.8/sec; rate(total) = 50.4±360.0/sec; N = 85,328 ERROR: Connection terminated ^[[8mha: ////4Cm+u8BY/EgsbhzNlnUfOXWprV5tRETZDv4u6647BaROAAAAVx+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0mV3NWzufebKBsTA0NFEYMUmgZnCA1SyAABjCCFBQC2xNaiYAAAAA==^[[0mjava.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59) Caused: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73) ERROR: Socket connection to SSH server was lost ^[[8mha: ////4Cm+u8BY/EgsbhzNlnUfOXWprV5tRETZDv4u6647BaROAAAAVx+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0mV3NWzufebKBsTA0NFEYMUmgZnCA1SyAABjCCFBQC2xNaiYAAAAA==^[[0mjava.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.net.SocketInputStream.read(SocketInputStream.java:141) at com.trilead.ssh2.crypto.cipher.CipherInputStream.fill_buffer(CipherInputStream.java:41) at com.trilead.ssh2.crypto.cipher.CipherInputStream.internal_read(CipherInputStream.java:52) at com.trilead.ssh2.crypto.cipher.CipherInputStream.getBlock(CipherInputStream.java:79) at com.trilead.ssh2.crypto.cipher.CipherInputStream.read(CipherInputStream.java:108) at com.trilead.ssh2.transport.TransportConnection.receiveMessage(TransportConnection.java:232) at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:706) at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:502) at java.lang. Thread .run( Thread .java:748) Slave JVM has not reported exit code before the socket was lost [01/08/18 15:07:02] [SSH] Connection closed. [01/08/18 15:10:07] [SSH] Checking java version of java [01/08/18 15:10:07] [SSH] java -version returned 1.8.0_141. [01/08/18 15:10:07] [SSH] Starting sftp client. [01/08/18 15:10:07] [SSH] Copying latest slave.jar... [01/08/18 15:10:08] [SSH] Copied 719,269 bytes. Expanded the channel window size to 4MB [01/08/18 15:10:08] [SSH] Starting slave process: cd "/home/devops/jenkins_slave_robot" && java -jar slave.jar <===[JENKINS REMOTING CAPACITY]===>^@^@^@channel started Slave.jar version: 3.7 This is a Unix agent Evacuated stdout Agent successfully connected and online ERROR: Connection terminated ^[[8mha: ////4Cm+u8BY/EgsbhzNlnUfOXWprV5tRETZDv4u6647BaROAAAAVx+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0mV3NWzufebKBsTA0NFEYMUmgZnCA1SyAABjCCFBQC2xNaiYAAAAA==^[[0mjava.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59) Caused: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73) ERROR: Socket connection to SSH server was lost ^[[8mha: ////4Cm+u8BY/EgsbhzNlnUfOXWprV5tRETZDv4u6647BaROAAAAVx+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0mV3NWzufebKBsTA0NFEYMUmgZnCA1SyAABjCCFBQC2xNaiYAAAAA==^[[0mjava.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.net.SocketInputStream.read(SocketInputStream.java:141) at com.trilead.ssh2.crypto.cipher.CipherInputStream.fill_buffer(CipherInputStream.java:41) at com.trilead.ssh2.crypto.cipher.CipherInputStream.internal_read(CipherInputStream.java:52) at com.trilead.ssh2.crypto.cipher.CipherInputStream.getBlock(CipherInputStream.java:79) at com.trilead.ssh2.crypto.cipher.CipherInputStream.read(CipherInputStream.java:108) at com.trilead.ssh2.transport.TransportConnection.receiveMessage(TransportConnection.java:232) at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:706) at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:502) at java.lang. Thread .run( Thread .java:748) Slave JVM has not reported exit code before the socket was lost [01/08/18 15:39:29] [SSH] Connection closed.

Rick Liu made changes - 2018-01-09 00:04

Attachment

New: 2018-0108-jenkins.log.tgz [ 41078 ]

Francois Aichelbaum made changes - 2018-02-06 09:18

Assignee

Original: Oleg Nenashev [ oleg_nenashev ]

New: Francois Aichelbaum [ faichelbaum ]

Francois Aichelbaum added a comment - 2018-02-06 09:19

Hi

as I experienced the same thing, I tested a lot of different things (kernel, libs, jdk version, ..., rolled back to previous version) with no luck

Then, I realized that my issues started with the various VMware patches for Spectre/Meltdown and that most of the case here for this kind of issues did too.

Maybe worth digging that way.

Cheers,

Francois Aichelbaum added a comment - 2018-02-06 09:19 Hi as I experienced the same thing, I tested a lot of different things (kernel, libs, jdk version, ..., rolled back to previous version) with no luck Then, I realized that my issues started with the various VMware patches for Spectre/Meltdown and that most of the case here for this kind of issues did too. Maybe worth digging that way. Cheers,

Rick Liu added a comment - 2018-02-06 18:51

faichelbaum
Are you using VMware on Jenkins server side or Jenkins slave side?
My Jenkins slaves are physical baremetal Dell machines,
and only the Jenkins master is a VMware VM.

Do you have VMware's patch number you applied?

Rick Liu added a comment - 2018-02-06 18:51 faichelbaum Are you using VMware on Jenkins server side or Jenkins slave side? My Jenkins slaves are physical baremetal Dell machines, and only the Jenkins master is a VMware VM. Do you have VMware's patch number you applied?

Jenkins

Details

Description

Jenkins Server log:

Jenkins Slave log:

Attachments

Attachments

Issue Links

Activity

Collapse comment: Rick Liu added a comment - 2018-01-08 22:01, Edited by Rick Liu - 2018-01-08 22:03

Expand comment: Rick Liu added a comment - 2018-01-08 22:01, Edited by Rick Liu - 2018-01-08 22:03

Collapse comment: Rick Liu added a comment - 2018-01-09 00:02

Expand comment: Rick Liu added a comment - 2018-01-09 00:02

Collapse comment: Francois Aichelbaum added a comment - 2018-02-06 09:19

Expand comment: Francois Aichelbaum added a comment - 2018-02-06 09:19

Collapse comment: Rick Liu added a comment - 2018-02-06 18:51

Expand comment: Rick Liu added a comment - 2018-02-06 18:51

People

Dates