Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-25858

java.io.IOException: Unexpected termination of the channel

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Minor Minor
    • remoting, ssh-slaves-plugin

      We are seeing slaves being displayed as offline with an Unexpected termination of channel IO exception. The slaves are actually running as seen in our AWS Management Console but the connection between jenkins master and the slave is broken. This behavior is fairly recent starting with maybe 1.584.

          [JENKINS-25858] java.io.IOException: Unexpected termination of the channel

          I am aslo facing the same issue of agent went offline during build.

          I am using jenkins v2.105 and jre 1.8

          I am using Linux as master and IBM AIX and windows server 2K12 as slaves. we are executing nightly builds on slaves but sometimes due to agent goes offline that build won't get complete, so anybody has any workarround for this issue then please let me know.

          Thanks in advance.

          shraddha Magar added a comment - I am aslo facing the same issue of agent went offline during build. I am using jenkins v2.105 and jre 1.8 I am using Linux as master and IBM AIX and windows server 2K12 as slaves. we are executing nightly builds on slaves but sometimes due to agent goes offline that build won't get complete, so anybody has any workarround for this issue then please let me know. Thanks in advance.

          shrapm Without the exception trace it is not possible to diagnose anything, Do you have the error trace on the Job and the corresponding error trace on the Jenkins logs? on the next version, the plugin will support by default the remoting debug I hope this would help to diagnose this kind of errors.

          Ivan Fernandez Calvo added a comment - shrapm Without the exception trace it is not possible to diagnose anything, Do you have the error trace on the Job and the corresponding error trace on the Jenkins logs? on the next version, the plugin will support by default the remoting debug I hope this would help to diagnose this kind of errors.

          Below is the error I can see in build log

          Agent went offline during the build
          ERROR: Connection was broken: java.io.EOFException
           at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2671)
           at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3146)
           at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:858)
           at java.io.ObjectInputStream.<init>(ObjectInputStream.java:354)
           at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
           at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
           at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
          Caused: java.io.IOException: Unexpected termination of the channel
           at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)

          shraddha Magar added a comment - Below is the error I can see in build log Agent went offline during the build ERROR: Connection was broken: java.io.EOFException  at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2671)  at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3146)  at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:858)  at java.io.ObjectInputStream.<init>(ObjectInputStream.java:354)  at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)  at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63) Caused: java.io.IOException: Unexpected termination of the channel  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)

          This is on Jenkins side, right? this only said that the channel is broken but not the root cause, Do you have any logs on the Agent workdir folder? also, check the latest executed build in that Agent to see if there is more info, see https://github.com/jenkinsci/remoting/blob/master/docs/logging.md to enable logging on the Agent side.

          Ivan Fernandez Calvo added a comment - This is on Jenkins side, right? this only said that the channel is broken but not the root cause, Do you have any logs on the Agent workdir folder? also, check the latest executed build in that Agent to see if there is more info, see https://github.com/jenkinsci/remoting/blob/master/docs/logging.md to enable logging on the Agent side.

          shraddha Magar added a comment - - edited

          Thanks for your quick reply.

          Yes. Those logs were from Jenkins side.

          Actually, slave goes offline for very short time and comes back online immediately. so not able to get the logs from agent. But due to that short time job is getting failed.

          shraddha Magar added a comment - - edited Thanks for your quick reply. Yes. Those logs were from Jenkins side. Actually, slave goes offline for very short time and comes back online immediately. so not able to get the logs from agent. But due to that short time job is getting failed.

          Which SSH Slaves Plugin version do you use? Did you add the `-workDir AGENT_WORK_DIR` parameter to your suffix command configuration to grab the logs?

          https://github.com/jenkinsci/remoting/blob/master/docs/workDir.md

          Ivan Fernandez Calvo added a comment - Which SSH Slaves Plugin version do you use? Did you add the `-workDir AGENT_WORK_DIR` parameter to your suffix command configuration to grab the logs? https://github.com/jenkinsci/remoting/blob/master/docs/workDir.md

          Also I have restarted Jenkins instance then agent which is configured on windows with JNLP got offline and its not coming online, its showing below error.

          Aug 29, 2018 1:43:04 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Locating server among http://x.x.x.x:x/jenkins/
          Aug 29, 2018 1:43:04 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
          INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
          Aug 29, 2018 1:43:04 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Agent discovery successful
            Agent address: x.x.x.x
            Agent port:    37091
            Identity:      51:a8:0d:a8:18:dd:db:d8:ea:18:b7:98:da:76:b2:ae
          Aug 29, 2018 1:43:04 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Handshaking
          Aug 29, 2018 1:43:04 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Connecting to x.x.x.x:37091
          Aug 29, 2018 1:43:35 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Connecting to x.x.x.x:37091 (retrying:2)
          java.io.IOException: Failed to connect to x.x.x.x:37091
           at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:242)
           at hudson.remoting.Engine.connect(Engine.java:686)
           at hudson.remoting.Engine.innerRun(Engine.java:547)
           at hudson.remoting.Engine.run(Engine.java:469)
          Caused by: java.net.ConnectException: Connection timed out: connect
           at sun.nio.ch.Net.connect0(Native Method)
           at sun.nio.ch.Net.connect(Unknown Source)
           at sun.nio.ch.Net.connect(Unknown Source)
           at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
           at java.nio.channels.SocketChannel.open(Unknown Source)
           at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:203)
           ... 3 more

           

          Could you please help me out please?

          just replaced master IP with x.x.x.x.

          shraddha Magar added a comment - Also I have restarted Jenkins instance then agent which is configured on windows with JNLP got offline and its not coming online, its showing below error. Aug 29, 2018 1:43:04 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among http://x.x.x.x:x/jenkins/ Aug 29, 2018 1:43:04 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping] Aug 29, 2018 1:43:04 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Agent discovery successful   Agent address: x.x.x.x   Agent port:    37091   Identity:      51:a8:0d:a8:18:dd:db:d8:ea:18:b7:98:da:76:b2:ae Aug 29, 2018 1:43:04 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Aug 29, 2018 1:43:04 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to x.x.x.x:37091 Aug 29, 2018 1:43:35 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to x.x.x.x:37091 (retrying:2) java.io.IOException: Failed to connect to x.x.x.x:37091  at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:242)  at hudson.remoting.Engine.connect(Engine.java:686)  at hudson.remoting.Engine.innerRun(Engine.java:547)  at hudson.remoting.Engine.run(Engine.java:469) Caused by: java.net.ConnectException: Connection timed out: connect  at sun.nio.ch.Net.connect0(Native Method)  at sun.nio.ch.Net.connect(Unknown Source)  at sun.nio.ch.Net.connect(Unknown Source)  at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)  at java.nio.channels.SocketChannel.open(Unknown Source)  at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:203)  ... 3 more   Could you please help me out please? just replaced master IP with x.x.x.x.

          This Windows Agent is a JNLP Agent, so your issue is not related to SSH Slaves plugin.

          It seems a network/configuration issue, not a bug, ask on the google user groups see https://wiki.jenkins.io/display/JENKINS/How+to+report+an+issue

          Ivan Fernandez Calvo added a comment - This Windows Agent is a JNLP Agent, so your issue is not related to SSH Slaves plugin. It seems a network/configuration issue, not a bug, ask on the google user groups see https://wiki.jenkins.io/display/JENKINS/How+to+report+an+issue

          yes. this agent is configured through JNLP.

          and other issue is for AIX machine. we have agents configured on both AIX and Windows. we are using ssh slave plugin 1.26.

          shraddha Magar added a comment - yes. this agent is configured through JNLP. and other issue is for AIX machine. we have agents configured on both AIX and Windows. we are using ssh slave plugin 1.26.

          I'm posting here because this was the most prominent result in google and I wanna share my solution in case it also helps somebody else.

          The last lines in our jenkins node log on the master instance were:

          Sep 23, 2022 10:56:28 AM null
          INFO: Launching Jenkins agent via plugin SSH: java -jar /tmp/agent.jar
          Sep 23, 2022 10:56:28 AM null
          WARNING: Error:  Exception: java.io.EOFException: unexpected stream termination

          Running the command `java -jar /tmp/agent.jar` on the node revealed that the used java version could not run the new jar - which was probably introduced by some jenkins und plugin update.

          Upgrading the node host from OpenJDK 8 to 11 fixed the problem for us.

          Michael Spiegel added a comment - I'm posting here because this was the most prominent result in google and I wanna share my solution in case it also helps somebody else. The last lines in our jenkins node log on the master instance were: Sep 23, 2022 10:56:28 AM null INFO: Launching Jenkins agent via plugin SSH: java -jar /tmp/agent.jar Sep 23, 2022 10:56:28 AM null WARNING: Error:  Exception: java.io.EOFException: unexpected stream termination Running the command `java -jar /tmp/agent.jar` on the node revealed that the used java version could not run the new jar - which was probably introduced by some jenkins und plugin update. Upgrading the node host from OpenJDK 8 to 11 fixed the problem for us.

            ifernandezcalvo Ivan Fernandez Calvo
            balsamiqluis2 Luis Arias
            Votes:
            17 Vote for this issue
            Watchers:
            28 Start watching this issue

              Created:
              Updated:
              Resolved: