[JENKINS-39289] Better diagnostics on "Backing channel is disconnected" - Jenkins Jira

XML

Word

Printable

Details

Type: Improvement
Status: Reopened (View Workflow)
Priority: Minor
Resolution: Unresolved
Component/s: remoting
Labels:
None

Similar Issues:

Show

Description

Often, the code that handles lost connection and report that problem is different from code that uses proxies that go over this channel. Example stack trace:

15:44:26 java.io.IOException: Backing channel is disconnected.
15:44:26        at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:184)
15:44:26        at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:249)
15:44:26        at com.sun.proxy.$Proxy104.isAlive(Unknown Source)
15:44:26        at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:996)
15:44:26        at hudson.maven.ProcessCache$MavenProcess.call(ProcessCache.java:166)
15:44:26        at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:853)
15:44:26        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
15:44:26        at hudson.model.Run.execute(Run.java:1738)
15:44:26        at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:531)
15:44:26        at hudson.model.ResourceController.execute(ResourceController.java:98)
15:44:26        at hudson.model.Executor.run(Executor.java:410)

The error diagnostics is easier if the cause of the channel loss is linked to this exception.

Attachments

Activity

Descending order - Click to sort in ascending order

Dell Green added a comment - 2018-07-25 05:39 - edited

I am also getting this almost every day with the volume of jobs we have on the server. Either that or a job hangs forever and never completes.

My slaves run in lxd containers,

jenkins 2.131,

java.io.IOException: Backing channel '005-lxd-ubuntu_16.04-x86_64' is disconnected.
at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:214)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
at com.sun.proxy.$Proxy73.isAlive(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1137)
at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1129)
at hudson.Launcher$ProcStarter.join(Launcher.java:464)
at hudson.plugins.gradle.Gradle.performTask(Gradle.java:333)
at hudson.plugins.gradle.Gradle.perform(Gradle.java:225)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
at hudson.model.Build$BuildExecution.build(Build.java:206)
at hudson.model.Build$BuildExecution.doRun(Build.java:163)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
at hudson.model.Run.execute(Run.java:1815)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2679)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3154)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
at hudson.remoting.Command.readFrom(Command.java:140)
at hudson.remoting.Command.readFrom(Command.java:126)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)

Dell Green added a comment - 2018-07-25 05:39 - edited I am also getting this almost every day with the volume of jobs we have on the server. Either that or a job hangs forever and never completes. My slaves run in lxd containers, jenkins 2.131, java.io.IOException: Backing channel '005-lxd-ubuntu_16.04-x86_64' is disconnected. at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:214) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283) at com.sun.proxy.$Proxy73.isAlive(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1137) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1129) at hudson.Launcher$ProcStarter.join(Launcher.java:464) at hudson.plugins.gradle.Gradle.performTask(Gradle.java:333) at hudson.plugins.gradle.Gradle.perform(Gradle.java:225) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744) at hudson.model.Build$BuildExecution.build(Build.java:206) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504) at hudson.model.Run.execute(Run.java:1815) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77) Caused by: java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2679) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3154) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.Command.readFrom(Command.java:140) at hudson.remoting.Command.readFrom(Command.java:126) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)

Oleg Nenashev added a comment - 2018-03-14 02:32

Unfortunately I have no capacity to work on Remoting in medium term, so I will unassign it and let others to take it. If somebody is interested to submit a pull request, I will be happy to help to get it reviewed and released.

Oleg Nenashev added a comment - 2018-03-14 02:32 Unfortunately I have no capacity to work on Remoting in medium term, so I will unassign it and let others to take it. If somebody is interested to submit a pull request, I will be happy to help to get it reviewed and released.

Oleg Nenashev added a comment - 2018-02-19 10:02

megafon why have you reopened it? Could you please clarify the status?

Oleg Nenashev added a comment - 2018-02-19 10:02 megafon why have you reopened it? Could you please clarify the status?

Jan Duris added a comment - 2018-02-07 05:51 - edited

Sometimes (randomly), my tests are failing with this error. Tests runs on docker nodes. Jenkins (or docker plugin) cancel docker node where tests are executed too early. Jenkins job can't finish because executor is terminated.

Jenkins version: 2.100 | Docker Plugin version: 1.1.2

FATAL: command execution failed
Command close created at
at hudson.remoting.Command.<init>(Command.java:62)
at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1220)
at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1218)
at hudson.remoting.Channel.close(Channel.java:1391)
at hudson.remoting.Channel.close(Channel.java:1358)
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1225)
Caused: hudson.remoting.Channel$OrderlyShutdown
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1226)
at hudson.remoting.Channel$1.handle(Channel.java:561)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:87)
Caused: java.io.IOException: Backing channel 'docker-4e1dcb53088a' is disconnected.
at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:212)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:281)
at com.sun.proxy.$Proxy87.isAlive(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1137)
at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1129)
at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:155)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
at hudson.model.Build$BuildExecution.build(Build.java:206)
at hudson.model.Build$BuildExecution.doRun(Build.java:163)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
at hudson.model.Run.execute(Run.java:1727)
at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)

Jan Duris added a comment - 2018-02-07 05:51 - edited Sometimes (randomly), my tests are failing with this error. Tests runs on docker nodes. Jenkins (or docker plugin) cancel docker node where tests are executed too early. Jenkins job can't finish because executor is terminated. Jenkins version: 2.100 | Docker Plugin version: 1.1.2 FATAL: command execution failed Command close created at at hudson.remoting.Command.<init>(Command.java:62) at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1220) at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1218) at hudson.remoting.Channel.close(Channel.java:1391) at hudson.remoting.Channel.close(Channel.java:1358) at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1225) Caused: hudson.remoting.Channel$OrderlyShutdown at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1226) at hudson.remoting.Channel$1.handle(Channel.java:561) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:87) Caused: java.io.IOException: Backing channel 'docker-4e1dcb53088a' is disconnected. at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:212) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:281) at com.sun.proxy.$Proxy87.isAlive(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1137) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1129) at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:155) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744) at hudson.model.Build$BuildExecution.build(Build.java:206) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504) at hudson.model.Run.execute(Run.java:1727) at hudson.matrix.MatrixRun.run(MatrixRun.java:146) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429)

Oleg Nenashev added a comment - 2016-11-10 21:17

The fix has been integrated towards jenkins-2.31

Oleg Nenashev added a comment - 2016-11-10 21:17 The fix has been integrated towards jenkins-2.31

View 4 older comments

People

Assignee:: Unassigned

Reporter:: Kohsuke Kawaguchi

Votes:: 0 Vote for this issue

Watchers:: 7 Start watching this issue

Dates

Created:: 2016-10-27 00:43

Updated:: 2018-07-25 06:09