[JENKINS-18836] java.io.IOException: Unexpected termination of the channel - Jenkins Jira

Details

Type: Bug
Status: Resolved (View Workflow)
Priority: Major
Resolution: Fixed
Component/s: ssh-slaves-plugin
Labels:
- 1.532.1-fixed
Environment:
Ubuntu 12.04 64bit, OpenJDK 1.6.0_27 (both master and slaves)
Tried Jenkins versions 1.521, 1.522, 1.523

Similar Issues:

Show

Description

On regular occasion builds fail due to an IO error while processing the findbugs results:

09:05:12 ERROR: Publisher hudson.plugins.findbugs.FindBugsPublisher aborted due to exception
09:05:12 hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
09:05:12 	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
09:05:12 	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
09:05:12 	at hudson.remoting.Request.call(Request.java:174)
09:05:12 	at hudson.remoting.Channel.call(Channel.java:718)
09:05:12 	at hudson.FilePath.act(FilePath.java:894)
09:05:12 	at hudson.FilePath.act(FilePath.java:878)
09:05:12 	at hudson.plugins.findbugs.FindBugsPublisher.perform(FindBugsPublisher.java:161)
09:05:12 	at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:144)
09:05:12 	at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:334)
09:05:12 	at hudson.tasks.BuildStepMonitor$2.perform(BuildStepMonitor.java:27)
09:05:12 	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
09:05:12 	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:776)
09:05:12 	at hudson.model.Build$BuildExecution.post2(Build.java:183)
09:05:12 	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:726)
09:05:12 	at hudson.model.Run.execute(Run.java:1618)
09:05:12 	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
09:05:12 	at hudson.model.ResourceController.execute(ResourceController.java:88)
09:05:12 	at hudson.model.Executor.run(Executor.java:247)
09:05:12 Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
09:05:12 	at hudson.remoting.Request.abort(Request.java:299)
09:05:13 	at hudson.remoting.Channel.terminate(Channel.java:778)
09:05:15 	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
09:05:15 Caused by: java.io.IOException: Unexpected termination of the channel
09:05:15 	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
09:05:16 Caused by: java.io.EOFException
09:05:16 	at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2595)
09:05:16 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1315)
09:05:22 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
09:05:22 	at hudson.remoting.Command.readFrom(Command.java:92)
09:05:22 	at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:72)
09:05:22 	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

This doesn't happen every time, sometimes the build does finish properly. But when it fails, it's always at the same point.

None the less, I reverted back to 1.518 for now.

Attachments

Issue Links

depends on

JENKINS-20550 download artifacts is very slow

Open

is related to

JENKINS-18879 Collecting finbugs analysis results randomly fails with exception

Resolved

JENKINS-19619 Collecting findbugs analysis results occasionally causes ssh slave to go offline causing job to abort

Resolved

JENKINS-15120 Minimize round trips for slave class loading

Resolved

Activity

Ascending order - Click to sort in descending order

View 10 older comments

SCM/JIRA link daemon added a comment - 2013-11-02 10:19

Code changed in jenkins
User: Stephen Connolly
Path:
core/pom.xml
http://jenkins-ci.org/commit/jenkins/1bb06ada301496ebed6d212188d1b7c9d006317b
Log:
[FIXED JENKINS-18836][FIXED JENKINS-18879][FIXED JENKINS-19619] Upgrade trilead-ssh to version with the fix

(cherry picked from commit bb265c5e95b0fe39128720b903914236962db41b)

Conflicts:
changelog.html

SCM/JIRA link daemon added a comment - 2013-11-02 10:19 Code changed in jenkins User: Stephen Connolly Path: core/pom.xml http://jenkins-ci.org/commit/jenkins/1bb06ada301496ebed6d212188d1b7c9d006317b Log: [FIXED JENKINS-18836] [FIXED JENKINS-18879] [FIXED JENKINS-19619] Upgrade trilead-ssh to version with the fix (cherry picked from commit bb265c5e95b0fe39128720b903914236962db41b) Conflicts: changelog.html

SCM/JIRA link daemon added a comment - 2014-05-19 16:43

Code changed in jenkins
User: Stephen Connolly
Path:
src/com/trilead/ssh2/channel/Channel.java
http://jenkins-ci.org/commit/trilead-ssh2/5811ddd7ae15670a4f9ad345352613b3f2f2db97
Log:
~~JENKINS-22938~~ SSH slave connections die after the slave outputs 4MB of stderr, usually during findbugs analysis

The fix for ~~JENKINS-18836~~, ~~JENKINS-18879~~, ~~JENKINS-19619~~ was incorrect in its analysis.

There is no call to getChannelData() on the new code path, so thus you cannot have two calls of freeupWindow()
The problem with the original call to freeupWindow() is that it is on the receiver thread. You should not mix the responsibilities. Blocking the receiver thread to send a message will negatively impact performance and connection stability.
The correct solution is to push the freeupWindow onto the async queue thus the ACK gets sent and the purity of the receiving thread can be maintained.

SCM/JIRA link daemon added a comment - 2014-05-19 16:43 Code changed in jenkins User: Stephen Connolly Path: src/com/trilead/ssh2/channel/Channel.java http://jenkins-ci.org/commit/trilead-ssh2/5811ddd7ae15670a4f9ad345352613b3f2f2db97 Log: JENKINS-22938 SSH slave connections die after the slave outputs 4MB of stderr, usually during findbugs analysis The fix for JENKINS-18836 , JENKINS-18879 , JENKINS-19619 was incorrect in its analysis. There is no call to getChannelData() on the new code path, so thus you cannot have two calls of freeupWindow() The problem with the original call to freeupWindow() is that it is on the receiver thread. You should not mix the responsibilities. Blocking the receiver thread to send a message will negatively impact performance and connection stability. The correct solution is to push the freeupWindow onto the async queue thus the ACK gets sent and the purity of the receiving thread can be maintained.

Rick Liu added a comment - 2016-06-03 20:01

Ubuntu 14.04 server 64-bit
oracle-java7: 1.7.0_80
Jenkins: 1.651.1 LTS
SSH Slaves plugin: 1.11

The build sometimes randomly failed with this kind of error.

This time happened in the post-build actions:
FATAL: channel is already closed
hudson.remoting.ChannelClosedException: channel is already closed
at hudson.remoting.Channel.send(Channel.java:578)
at hudson.remoting.Request.call(Request.java:130)
at hudson.remoting.Channel.call(Channel.java:780)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:953)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:540)
at hudson.model.Run.execute(Run.java:1738)
at hudson.matrix.MatrixBuild.run(MatrixBuild.java:313)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Caused by: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

Rick Liu added a comment - 2016-06-03 20:01 Ubuntu 14.04 server 64-bit oracle-java7: 1.7.0_80 Jenkins: 1.651.1 LTS SSH Slaves plugin: 1.11 The build sometimes randomly failed with this kind of error. This time happened in the post-build actions: FATAL: channel is already closed hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:578) at hudson.remoting.Request.call(Request.java:130) at hudson.remoting.Channel.call(Channel.java:780) at hudson.Launcher$RemoteLauncher.kill(Launcher.java:953) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:540) at hudson.model.Run.execute(Run.java:1738) at hudson.matrix.MatrixBuild.run(MatrixBuild.java:313) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:410) Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50) Caused by: java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

Indra Gunawan added a comment - 2016-07-18 15:11

Master: 1.580.3
SSH Slave plugin: 1.8

Killed by signal 15.
FATAL: java.io.IOException: Unexpected termination of the channel
hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request.abort(Request.java:295)
at hudson.remoting.Channel.terminate(Channel.java:814)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
at ......remote call to sym-bld-lnx8(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356)
at hudson.remoting.Request.call(Request.java:171)
at hudson.remoting.Channel.call(Channel.java:751)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:173)
at com.sun.proxy.$Proxy73.join(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:979)
at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:137)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:770)
at hudson.model.Build$BuildExecution.build(Build.java:199)
at hudson.model.Build$BuildExecution.doRun(Build.java:160)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:533)
at hudson.model.Run.execute(Run.java:1745)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:89)
at hudson.model.Executor.run(Executor.java:240)
Caused by: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:40)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

Indra Gunawan added a comment - 2016-07-18 15:11 Master: 1.580.3 SSH Slave plugin: 1.8 Killed by signal 15. FATAL: java.io.IOException: Unexpected termination of the channel hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request.abort(Request.java:295) at hudson.remoting.Channel.terminate(Channel.java:814) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69) at ......remote call to sym-bld-lnx8(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356) at hudson.remoting.Request.call(Request.java:171) at hudson.remoting.Channel.call(Channel.java:751) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:173) at com.sun.proxy.$Proxy73.join(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:979) at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:137) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:770) at hudson.model.Build$BuildExecution.build(Build.java:199) at hudson.model.Build$BuildExecution.doRun(Build.java:160) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:533) at hudson.model.Run.execute(Run.java:1745) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:89) at hudson.model.Executor.run(Executor.java:240) Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50) Caused by: java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:40) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

Indra Gunawan added a comment - 2016-07-18 15:21

Jenkins version : 1.580.3
SSH slave plugin version: 1.8
RHEL 6.2

build log:

Killed by signal 15.
FATAL: java.io.IOException: Unexpected termination of the channel
hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request.abort(Request.java:295)
at hudson.remoting.Channel.terminate(Channel.java:814)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
at ......remote call to sym-bld-lnx8(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356)
at hudson.remoting.Request.call(Request.java:171)
at hudson.remoting.Channel.call(Channel.java:751)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:173)
at com.sun.proxy.$Proxy73.join(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:979)
at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:137)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:770)
at hudson.model.Build$BuildExecution.build(Build.java:199)
at hudson.model.Build$BuildExecution.doRun(Build.java:160)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:533)
at hudson.model.Run.execute(Run.java:1745)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:89)
at hudson.model.Executor.run(Executor.java:240)
Caused by: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:40)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

Indra Gunawan added a comment - 2016-07-18 15:21 Jenkins version : 1.580.3 SSH slave plugin version: 1.8 RHEL 6.2 build log: Killed by signal 15. FATAL: java.io.IOException: Unexpected termination of the channel hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request.abort(Request.java:295) at hudson.remoting.Channel.terminate(Channel.java:814) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69) at ......remote call to sym-bld-lnx8(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356) at hudson.remoting.Request.call(Request.java:171) at hudson.remoting.Channel.call(Channel.java:751) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:173) at com.sun.proxy.$Proxy73.join(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:979) at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:137) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:770) at hudson.model.Build$BuildExecution.build(Build.java:199) at hudson.model.Build$BuildExecution.doRun(Build.java:160) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:533) at hudson.model.Run.execute(Run.java:1745) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:89) at hudson.model.Executor.run(Executor.java:240) Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50) Caused by: java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:40) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

Jenkins

java.io.IOException: Unexpected termination of the channel

Details

Description

Attachments

Issue Links

Activity

People

Dates