Slave disconnect unexpectedly hudson.remoting.ChannelClosedException

This issue is archived. You can view it, but you can't modify it. Learn more

XMLWordPrintable

      Hi,

      Every couple of days my jenkins slave got disconnected while running builds.

      On the slave build log I have this:

      02:55:25 FATAL: command execution failed
02:55:25 java.io.EOFException
02:55:25 	at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2832)
02:55:25 	at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3307)
02:55:25 	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:934)
02:55:25 	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:396)
02:55:25 	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
02:55:25 	at hudson.remoting.Command.readFrom(Command.java:142)
02:55:25 	at hudson.remoting.Command.readFrom(Command.java:128)
02:55:25 	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
02:55:25 	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
02:55:25 Caused: java.io.IOException: Unexpected termination of the channel
02:55:25 	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
02:55:25 Caused: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@565f8d19:QA Automation": Remote call on QA Automation failed. The channel is closing down or has closed down
02:55:25 	at hudson.remoting.Channel.call(Channel.java:994)
02:55:25 	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
02:55:25 	at com.sun.proxy.$Proxy106.isAlive(Unknown Source)
02:55:25 	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1211)
02:55:25 	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1203)
02:55:25 	at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:194)
02:55:25 	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:144)
02:55:25 	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:91)
02:55:25 	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
02:55:25 	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
02:55:25 	at hudson.model.Build$BuildExecution.build(Build.java:206)
02:55:25 	at hudson.model.Build$BuildExecution.doRun(Build.java:163)
02:55:25 	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
02:55:25 	at hudson.model.Run.execute(Run.java:1894)
02:55:25 	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
02:55:25 	at hudson.model.ResourceController.execute(ResourceController.java:97)
02:55:25 	at hudson.model.Executor.run(Executor.java:428)
02:55:25 FATAL: Unable to delete script file /tmp/jenkins2698528342924960038.sh
02:55:25 java.io.EOFException
02:55:25 	at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2832)
02:55:25 	at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3307)
02:55:25 	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:934)
02:55:25 	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:396)
02:55:25 	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
02:55:25 	at hudson.remoting.Command.readFrom(Command.java:142)
02:55:25 	at hudson.remoting.Command.readFrom(Command.java:128)
02:55:25 	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
02:55:25 	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
02:55:25 Caused: java.io.IOException: Unexpected termination of the channel
02:55:25 	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
02:55:25 Caused: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@565f8d19:QA Automation": Remote call on QA Automation failed. The channel is closing down or has closed down
02:55:25 	at hudson.remoting.Channel.call(Channel.java:994)
02:55:25 	at hudson.FilePath.act(FilePath.java:1070)
02:55:25 	at hudson.FilePath.act(FilePath.java:1059)
02:55:25 	at hudson.FilePath.delete(FilePath.java:1582)
02:55:25 	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:162)
02:55:25 	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:91)
02:55:25 	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
02:55:25 	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
02:55:25 	at hudson.model.Build$BuildExecution.build(Build.java:206)
02:55:25 	at hudson.model.Build$BuildExecution.doRun(Build.java:163)
02:55:25 	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
02:55:25 	at hudson.model.Run.execute(Run.java:1894)
02:55:25 	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
02:55:25 	at hudson.model.ResourceController.execute(ResourceController.java:97)
02:55:25 	at hudson.model.Executor.run(Executor.java:428)
02:55:25 Build step 'Execute shell' marked build as failure
02:55:25 Notifying upstream projects of job completion
02:55:25 Finished: FAILURE

      On the server log I got this:

      2023-04-23 23:55:25.058+0000 [id=2242399]    INFO    h.r.SynchronousCommandTransport$ReaderThread#run: I/O error in channel QA Automation
java.io.EOFException
    at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2832)
    at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3307)
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:934)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:396)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
2023-04-23 23:55:57.079+0000 [id=38]    INFO    hudson.slaves.SlaveComputer#tryReconnect: Attempting to reconnect QA Automation
[04/24/23 02:56:39] SSH Launch of QA Automation on 54.176.197.167 completed in 42,886 ms

      Jenkins server version: 2.249.2

      SSH Agent version: 1.20

      SSH plugin version: 2.6.1

       

      I some some other tickets about this issue but with no clear solution, like:

      https://issues.jenkins.io/browse/JENKINS-25858

      https://issues.jenkins.io/browse/JENKINS-48850

       

      My builds are actully running docker, when the dissconnection happens, the builds stopped but the containers keep running.

      Can you please assist?

            Assignee:
            Jeff Thompson
            Reporter:
            Avie
            Archiver:
            Jenkins Service Account

              Created:
              Updated:
              Resolved:
              Archived: