Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-18836

java.io.IOException: Unexpected termination of the channel

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: ssh-slaves-plugin
    • Labels:
    • Environment:
      Ubuntu 12.04 64bit, OpenJDK 1.6.0_27 (both master and slaves)
      Tried Jenkins versions 1.521, 1.522, 1.523
    • Similar Issues:

      Description

      On regular occasion builds fail due to an IO error while processing the findbugs results:

      09:05:12 ERROR: Publisher hudson.plugins.findbugs.FindBugsPublisher aborted due to exception
09:05:12 hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
09:05:12 	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
09:05:12 	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
09:05:12 	at hudson.remoting.Request.call(Request.java:174)
09:05:12 	at hudson.remoting.Channel.call(Channel.java:718)
09:05:12 	at hudson.FilePath.act(FilePath.java:894)
09:05:12 	at hudson.FilePath.act(FilePath.java:878)
09:05:12 	at hudson.plugins.findbugs.FindBugsPublisher.perform(FindBugsPublisher.java:161)
09:05:12 	at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:144)
09:05:12 	at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:334)
09:05:12 	at hudson.tasks.BuildStepMonitor$2.perform(BuildStepMonitor.java:27)
09:05:12 	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
09:05:12 	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:776)
09:05:12 	at hudson.model.Build$BuildExecution.post2(Build.java:183)
09:05:12 	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:726)
09:05:12 	at hudson.model.Run.execute(Run.java:1618)
09:05:12 	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
09:05:12 	at hudson.model.ResourceController.execute(ResourceController.java:88)
09:05:12 	at hudson.model.Executor.run(Executor.java:247)
09:05:12 Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
09:05:12 	at hudson.remoting.Request.abort(Request.java:299)
09:05:13 	at hudson.remoting.Channel.terminate(Channel.java:778)
09:05:15 	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
09:05:15 Caused by: java.io.IOException: Unexpected termination of the channel
09:05:15 	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
09:05:16 Caused by: java.io.EOFException
09:05:16 	at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2595)
09:05:16 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1315)
09:05:22 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
09:05:22 	at hudson.remoting.Command.readFrom(Command.java:92)
09:05:22 	at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:72)
09:05:22 	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

      This doesn't happen every time, sometimes the build does finish properly. But when it fails, it's always at the same point.

      None the less, I reverted back to 1.518 for now.

        Attachments

          Issue Links

          depends on

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-20550 download artifacts is very slow

          • Critical - Crashes, loss of data, severe memory leak.
          • Open
          is related to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-18879 Collecting finbugs analysis results randomly fails with exception

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-19619 Collecting findbugs analysis results occasionally causes ssh slave to go offline causing job to abort

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. JENKINS-15120 Minimize round trips for slave class loading

          • Major - Major loss of function.
          • Resolved

            Activity

            Descending order - Click to sort in ascending order
            Hide
            Permalink
            ingunawa Indra Gunawan added a comment -

            Jenkins version : 1.580.3
            SSH slave plugin version: 1.8
            RHEL 6.2

            build log:

            Killed by signal 15.
            FATAL: java.io.IOException: Unexpected termination of the channel
            hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
            at hudson.remoting.Request.abort(Request.java:295)
            at hudson.remoting.Channel.terminate(Channel.java:814)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
            at ......remote call to sym-bld-lnx8(Native Method)
            at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356)
            at hudson.remoting.Request.call(Request.java:171)
            at hudson.remoting.Channel.call(Channel.java:751)
            at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:173)
            at com.sun.proxy.$Proxy73.join(Unknown Source)
            at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:979)
            at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:137)
            at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97)
            at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
            at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
            at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:770)
            at hudson.model.Build$BuildExecution.build(Build.java:199)
            at hudson.model.Build$BuildExecution.doRun(Build.java:160)
            at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:533)
            at hudson.model.Run.execute(Run.java:1745)
            at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
            at hudson.model.ResourceController.execute(ResourceController.java:89)
            at hudson.model.Executor.run(Executor.java:240)
            Caused by: java.io.IOException: Unexpected termination of the channel
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
            Caused by: java.io.EOFException
            at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325)
            at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794)
            at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801)
            at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
            at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:40)
            at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

            Show
            ingunawa Indra Gunawan added a comment - Jenkins version : 1.580.3 SSH slave plugin version: 1.8 RHEL 6.2 build log: Killed by signal 15. FATAL: java.io.IOException: Unexpected termination of the channel hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request.abort(Request.java:295) at hudson.remoting.Channel.terminate(Channel.java:814) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69) at ......remote call to sym-bld-lnx8(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356) at hudson.remoting.Request.call(Request.java:171) at hudson.remoting.Channel.call(Channel.java:751) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:173) at com.sun.proxy.$Proxy73.join(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:979) at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:137) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:770) at hudson.model.Build$BuildExecution.build(Build.java:199) at hudson.model.Build$BuildExecution.doRun(Build.java:160) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:533) at hudson.model.Run.execute(Run.java:1745) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:89) at hudson.model.Executor.run(Executor.java:240) Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50) Caused by: java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:40) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
            Hide
            Permalink
            ingunawa Indra Gunawan added a comment -

            Master: 1.580.3
            SSH Slave plugin: 1.8

            Killed by signal 15.
            FATAL: java.io.IOException: Unexpected termination of the channel
            hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
            at hudson.remoting.Request.abort(Request.java:295)
            at hudson.remoting.Channel.terminate(Channel.java:814)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
            at ......remote call to sym-bld-lnx8(Native Method)
            at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356)
            at hudson.remoting.Request.call(Request.java:171)
            at hudson.remoting.Channel.call(Channel.java:751)
            at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:173)
            at com.sun.proxy.$Proxy73.join(Unknown Source)
            at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:979)
            at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:137)
            at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97)
            at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
            at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
            at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:770)
            at hudson.model.Build$BuildExecution.build(Build.java:199)
            at hudson.model.Build$BuildExecution.doRun(Build.java:160)
            at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:533)
            at hudson.model.Run.execute(Run.java:1745)
            at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
            at hudson.model.ResourceController.execute(ResourceController.java:89)
            at hudson.model.Executor.run(Executor.java:240)
            Caused by: java.io.IOException: Unexpected termination of the channel
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
            Caused by: java.io.EOFException
            at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325)
            at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794)
            at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801)
            at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
            at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:40)
            at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

            Show
            ingunawa Indra Gunawan added a comment - Master: 1.580.3 SSH Slave plugin: 1.8 Killed by signal 15. FATAL: java.io.IOException: Unexpected termination of the channel hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request.abort(Request.java:295) at hudson.remoting.Channel.terminate(Channel.java:814) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69) at ......remote call to sym-bld-lnx8(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356) at hudson.remoting.Request.call(Request.java:171) at hudson.remoting.Channel.call(Channel.java:751) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:173) at com.sun.proxy.$Proxy73.join(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:979) at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:137) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:770) at hudson.model.Build$BuildExecution.build(Build.java:199) at hudson.model.Build$BuildExecution.doRun(Build.java:160) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:533) at hudson.model.Run.execute(Run.java:1745) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:89) at hudson.model.Executor.run(Executor.java:240) Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50) Caused by: java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:40) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
            Hide
            Permalink
            totoroliu Rick Liu added a comment -

            Ubuntu 14.04 server 64-bit
            oracle-java7: 1.7.0_80
            Jenkins: 1.651.1 LTS
            SSH Slaves plugin: 1.11

            The build sometimes randomly failed with this kind of error.

            This time happened in the post-build actions:
            FATAL: channel is already closed
            hudson.remoting.ChannelClosedException: channel is already closed
            at hudson.remoting.Channel.send(Channel.java:578)
            at hudson.remoting.Request.call(Request.java:130)
            at hudson.remoting.Channel.call(Channel.java:780)
            at hudson.Launcher$RemoteLauncher.kill(Launcher.java:953)
            at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:540)
            at hudson.model.Run.execute(Run.java:1738)
            at hudson.matrix.MatrixBuild.run(MatrixBuild.java:313)
            at hudson.model.ResourceController.execute(ResourceController.java:98)
            at hudson.model.Executor.run(Executor.java:410)
            Caused by: java.io.IOException: Unexpected termination of the channel
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
            Caused by: java.io.EOFException
            at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325)
            at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794)
            at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801)
            at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
            at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
            at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

            Show
            totoroliu Rick Liu added a comment - Ubuntu 14.04 server 64-bit oracle-java7: 1.7.0_80 Jenkins: 1.651.1 LTS SSH Slaves plugin: 1.11 The build sometimes randomly failed with this kind of error. This time happened in the post-build actions: FATAL: channel is already closed hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:578) at hudson.remoting.Request.call(Request.java:130) at hudson.remoting.Channel.call(Channel.java:780) at hudson.Launcher$RemoteLauncher.kill(Launcher.java:953) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:540) at hudson.model.Run.execute(Run.java:1738) at hudson.matrix.MatrixBuild.run(MatrixBuild.java:313) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:410) Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50) Caused by: java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
            Hide
            Permalink
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Stephen Connolly
            Path:
            src/com/trilead/ssh2/channel/Channel.java
            http://jenkins-ci.org/commit/trilead-ssh2/5811ddd7ae15670a4f9ad345352613b3f2f2db97
            Log:
            JENKINS-22938 SSH slave connections die after the slave outputs 4MB of stderr, usually during findbugs analysis

            The fix for JENKINS-18836, JENKINS-18879, JENKINS-19619 was incorrect in its analysis.

            • There is no call to getChannelData() on the new code path, so thus you cannot have two calls of freeupWindow()
            • The problem with the original call to freeupWindow() is that it is on the receiver thread. You should not mix the responsibilities. Blocking the receiver thread to send a message will negatively impact performance and connection stability.
            • The correct solution is to push the freeupWindow onto the async queue thus the ACK gets sent and the purity of the receiving thread can be maintained.
            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: src/com/trilead/ssh2/channel/Channel.java http://jenkins-ci.org/commit/trilead-ssh2/5811ddd7ae15670a4f9ad345352613b3f2f2db97 Log: JENKINS-22938 SSH slave connections die after the slave outputs 4MB of stderr, usually during findbugs analysis The fix for JENKINS-18836 , JENKINS-18879 , JENKINS-19619 was incorrect in its analysis. There is no call to getChannelData() on the new code path, so thus you cannot have two calls of freeupWindow() The problem with the original call to freeupWindow() is that it is on the receiver thread. You should not mix the responsibilities. Blocking the receiver thread to send a message will negatively impact performance and connection stability. The correct solution is to push the freeupWindow onto the async queue thus the ACK gets sent and the purity of the receiving thread can be maintained.
            Hide
            Permalink
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Stephen Connolly
            Path:
            core/pom.xml
            http://jenkins-ci.org/commit/jenkins/1bb06ada301496ebed6d212188d1b7c9d006317b
            Log:
            [FIXED JENKINS-18836][FIXED JENKINS-18879][FIXED JENKINS-19619] Upgrade trilead-ssh to version with the fix

            (cherry picked from commit bb265c5e95b0fe39128720b903914236962db41b)

            Conflicts:
            changelog.html

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: core/pom.xml http://jenkins-ci.org/commit/jenkins/1bb06ada301496ebed6d212188d1b7c9d006317b Log: [FIXED JENKINS-18836] [FIXED JENKINS-18879] [FIXED JENKINS-19619] Upgrade trilead-ssh to version with the fix (cherry picked from commit bb265c5e95b0fe39128720b903914236962db41b) Conflicts: changelog.html
            10 older comments

              People

              Assignee:
              kohsuke Kohsuke Kawaguchi
              Reporter:
              elmuerte Michiel Hendriks
              Votes:
              4 Vote for this issue
              Watchers:
              13 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: