If you encounter similar problems on recent versions of Jenkins and up to date slave.jar on agents, please file a new bug instead of reopening this one. It's too old to be useful. Provide as much information about your configuration as possible.

We have the same situation. Build jobs which uses the clone workspace let the slaves die with the above message. All other builds work fine and do not make any trouble.

We have a setup with three jobs:

1. clean deploy -DskipTests=true
2. test
3. site site:deploy

here is the complete log of the test-build

Gestartet durch vorgelagertes Projekt "de.wwag.infra.oocsd~commons~B-trunk~compile", Build 20
Baue auf Slave pasv0066
FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request.call(Request.java:137)
at hudson.remoting.Channel.call(Channel.java:551)
at hudson.FilePath.act(FilePath.java:736)
at hudson.FilePath.act(FilePath.java:729)
at hudson.FilePath.unzip(FilePath.java:409)
at hudson.FileSystemProvisioner$Default$WorkspaceSnapshotImpl.restoreTo(FileSystemProvisioner.java:227)
at hudson.plugins.cloneworkspace.CloneWorkspaceSCM$Snapshot.restoreTo(CloneWorkspaceSCM.java:344)
at hudson.plugins.cloneworkspace.CloneWorkspaceSCM.checkout(CloneWorkspaceSCM.java:126)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1044)
at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
at hudson.model.Run.run(Run.java:1244)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:306)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:122)
Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request.abort(Request.java:257)
at hudson.remoting.Channel.terminate(Channel.java:598)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:880)
Caused by: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Channel$ReaderThread.run(Channel.java:862)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:856)

between 'Baue auf Slave' and the error is a time frame of about one minute.

the compile-job was executed on the same slave.

This is a mere blocker!

We tried with reducing the load (only one build running), changing the jdk from 1.6.0_16 to 1.6.0_20 but nothing helped.

I looked a little bit further:

The first job runs fine an creates a workspace.zip on the master - not on the slave

It seems that the following job has problems to access this file (even if it is running on the same slave).

After a minute a timeout of hudson occurs and kills the slave.

On our testing system it is working - but this is a build which is very small. The same build on production also works.

Perhaps there is a problem with the size of the workspace.zip (e.g. buffering)?

testing (working): 6404 bytes
real build (problem): 38494759 bytes

Is there any progress on this problem?

I assume that this is a problem in hudson core.

We tried to simulate the clone workspace by doing a wget on the 'Download all files as a zip' link and unpack them. The same error occured.

I assume that the problem is on the master closing the connection and not on the client-side.

Seeing the same thing here.

I have an upstream job, "MyCode-upstream," using the Clone Workspace SCM Plug-in to publish files, and a downstream job, "MyCode-downstream," using Clone Workspace SCM Plug-in to pull the upstream job's published files. The downstream job fails with hudson.remoting.RequestAbortedException (full Console log below).

I've also set up a pair of test jobs, "foo-upstream" and "foo-downstream", which use the Clone Workspace SCM Plug-in just like the "MyCode" jobs do... and they work EVERY TIME! Also, after running the "foo" jobs the "MyCode-downstream" job starts working - WTH???

One noticeable difference is the size of the files that the Clone Workspace SCM Plug-in publishes: ~3KB for the "foo-upstream" job, and ~22MB for the "MyCode-upstream" job.

Console log from the slave machine running the "MyCode-upstream" (SCM Client) build:

Started by upstream project "MyCode-upstream" build number 12
Building remotely on ci-deploy
FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request.call(Request.java:137)
at hudson.remoting.Channel.call(Channel.java:557)
at hudson.FilePath.act(FilePath.java:742)
at hudson.FilePath.act(FilePath.java:735)
at hudson.FilePath.unzip(FilePath.java:415)
at hudson.FileSystemProvisioner$Default$WorkspaceSnapshotImpl.restoreTo(FileSystemProvisioner.java:227)
at hudson.plugins.cloneworkspace.CloneWorkspaceSCM$Snapshot.restoreTo(CloneWorkspaceSCM.java:344)
at hudson.plugins.cloneworkspace.CloneWorkspaceSCM.checkout(CloneWorkspaceSCM.java:126)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1054)
at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
at hudson.model.Run.run(Run.java:1248)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:129)
Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request.abort(Request.java:257)
at hudson.remoting.Channel.terminate(Channel.java:608)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:899)
Caused by: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Channel$ReaderThread.run(Channel.java:881)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:875)

I did some more testing with my previously mentioned "foo" jobs - I varied the size of the content packaged by the Clone Workspace plugin and noticed that when the content grew much larger than just 64KB (though even that number was not consistent) the downstream jobs (running on slaves) would fail. NOTE: the slave agents were restarted before kicking off the upstream job.

An example of the steps I used:

1. configure "foo-upstream" to only have a 256KB text file, which the Clone Workspace plugin packages up
2. restart slave (where "foo-downstream" is tied to run)
3. run "foo-upstream"
   • kicks off "foo-downstream" which fails with the above error
4. change "foo-upstream" to only have a 32K text file.
5. restart slave (where "foo-downstream" is tied to run)
6. run "foo-upstream"
   • kicks off "foo-downstream" which WORKS!
7. change "foo-upstream" back to have a 256KB text file (like in step 1)
   • DO NOT restart the slave
8. run "foo-upstream"
   • kicks off "foo-downstream" which WORKS!

This is most bizarre and I'm at a loss for what to try next. Any ideas?

i just got the same stacktrace in the warnings parser with hudson 1.380 on a win32 slave (running in virtual box), master is linux 64bit.

WARNINGS] Parsing warnings in console log...
ERROR: Publisher hudson.plugins.warnings.WarningsPublisher aborted due to exception
hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request.call(Request.java:137)
at hudson.remoting.Channel.call(Channel.java:630)
at hudson.FilePath.act(FilePath.java:742)
at hudson.FilePath.act(FilePath.java:735)
at hudson.plugins.warnings.WarningsPublisher.perform(WarningsPublisher.java:244)
at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:280)
at hudson.tasks.BuildStepMonitor$2.perform(BuildStepMonitor.java:27)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:601)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:580)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:558)
at hudson.model.Build$RunnerImpl.post2(Build.java:157)
at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:528)
at hudson.model.Run.run(Run.java:1303)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:140)
Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request.abort(Request.java:257)
at hudson.remoting.Channel.terminate(Channel.java:681)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:973)
Caused by: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Channel$ReaderThread.run(Channel.java:954)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:948)
Archiving artifacts
ERROR: Publisher hudson.tasks.Mailer aborted due to exception
java.lang.NullPointerException
at hudson.tasks.MailSender.createFailureMail(MailSender.java:271)
at hudson.tasks.MailSender.getMail(MailSender.java:146)
at hudson.tasks.MailSender.execute(MailSender.java:94)
at hudson.tasks.Mailer.perform(Mailer.java:111)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:601)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:580)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:558)
at hudson.model.Build$RunnerImpl.post2(Build.java:157)
at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:528)
at hudson.model.Run.run(Run.java:1303)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:140)
Finished: FAILURE

We have seen slave stacktrace similar to this resolved with increasing the slave JVM memory options.
For SSH launched slaves, see the JVM Options field in the Advanced section under node launch method.
At least -Xmx128m is needed if you are on a 1.6 JVM and pushing across large files or returning large result sets.

I'm getting the "IOException: Unexpected termination of the channel" issues. You might be interested in my notes in JENKINS-9576.
I'm getting "IOException: Unexpected termination of the channel" both on win7 slaves and win XP slaves. This seems to have something to due with the long-running processes that I spawn from Ant <exec> and perhaps that sometimes the hard drive in the box gets busy.

I see that Lynn Lin has posted "this may be caused by disk plugin .after i disabled this plugin,the issue gone"
I'm going to try that...

I've disabled "Hudson disk-usage plugin" and also upgraded to Jenkins ver. 1.414
Oh no, its still happening.

Definitely still happening: "IOException: Unexpected termination of the channel"

I'm starting the slaves with lines like: javaws http://192.168.1.28:8080/computer/Win7-64/slave-agent.jnlp
and its using 360M of RAM. So I don't think adding -Xmx128m would help.

Anyone have any suggestions?

I thought maybe the backup system on this machine might be hogging the hard disk and freezing things up. So I turned it off.
Although backup system is off, still got the "IOException: Unexpected termination of the channel"

Still happening over here.

I've broken all my jobs into smaller pieces, so that instead of taking 7 hrs per job (times 3 jobs) it now takes 2-3 hrs per job (times a dozen jobs). But ... I'm still periodically getting:
"IOException: Unexpected termination of the channel"

Also seeing this issue with a Windows XP slave and a Linux Master. Jobs are under 30 minutes but get frequent channel closed erros.

FATAL: Unable to delete script file C:\DOCUME~1\rbmadmin\LOCALS~1\Temp\hudson291666598280615056.bat
hudson.util.IOException2: remote file operation failed: C:\DOCUME~1\rbmadmin\LOCALS~1\Temp\hudson291666598280615056.bat at hudson.remoting.Channel@3f3611ac:WindowsClient
at hudson.FilePath.act(FilePath.java:754)
at hudson.FilePath.act(FilePath.java:740)
at hudson.FilePath.delete(FilePath.java:995)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
at hudson.plugins.m2extrasteps.M2ExtraStepsWrapper.executeBuildSteps(M2ExtraStepsWrapper.java:166)
at hudson.plugins.m2extrasteps.M2ExtraStepsWrapper.setUp(M2ExtraStepsWrapper.java:126)
at hudson.maven.MavenModuleSetBuild$RunnerImpl.doRun(MavenModuleSetBuild.java:603)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:430)
at hudson.model.Run.run(Run.java:1376)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:466)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:175)
Caused by: hudson.remoting.ChannelClosedException: channel is already closed
at hudson.remoting.Channel.send(Channel.java:486)
at hudson.remoting.Request.call(Request.java:110)
at hudson.remoting.Channel.call(Channel.java:667)
at hudson.FilePath.act(FilePath.java:747)
... 12 more
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2248)
at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2541)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2551)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1296)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1030)
ERROR: Processing failed due to a bug in the code. Please report this to jenkinsci-users@googlegroups.com
hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.net.SocketException: Socket closed
at hudson.remoting.Request.call(Request.java:149)
at hudson.remoting.Channel.call(Channel.java:667)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158)
at $Proxy21.join(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:850)
at hudson.Launcher$ProcStarter.join(Launcher.java:336)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:82)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
at hudson.plugins.m2extrasteps.M2ExtraStepsWrapper.executeBuildSteps(M2ExtraStepsWrapper.java:166)
at hudson.plugins.m2extrasteps.M2ExtraStepsWrapper.setUp(M2ExtraStepsWrapper.java:126)
at hudson.maven.MavenModuleSetBuild$RunnerImpl.doRun(MavenModuleSetBuild.java:603)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:430)
at hudson.model.Run.run(Run.java:1376)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:466)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:175)
Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Socket closed
at hudson.remoting.Request.abort(Request.java:273)
at hudson.remoting.Channel.terminate(Channel.java:718)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1059)
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2248)
at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2541)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2551)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1296)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1030)

Still happening for me, for example:

Sep 11, 2011 5:40:23 PM	 
IOException: Unexpected termination of the channel
Jenkins ver. 1.414

There have been several remoting related fixes in newer version, specifically 1.416 and 1.420. Could you try it with that?

Still happening for me, Jenkins ver 1.430. I noticed that the issue this bug was forked from has apparently been fixed.

Okay, I've upgraded to 1.431 – It's always nice to have the latest.

Page generated: Sep 24, 2011 12:47:57 AM  Jenkins ver. 1.431

Let's see what happens ...

Guess that doesn't fix the whole problem:

     [echo] 09/24/2011 10:33 PM Copy prototype source tree to build directory ...
FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel


Page generated: Sep 25, 2011 4:24:23 PM    Jenkins ver. 1.431

I'm getting this as well on 1.432.

As David mentioned, it looks like this was fixed on the fork (http://issues.hudson-ci.org/browse/HUDSON-6817). Any chance we can try this out here?

I've found same problem on Jenkins ver. 1.418 during building rpm of my application. The problem was, that the building task tried to obtain root's password and after a while, it has thrown unexpected termination of the channel. Check whether your build task is not waiting for some user input.

I tried my build using the current LTS release of Jenkins (1.409.2), and some of my shorter builds behaved better. However, some of the longer ones still get this error.

I've tried patching the source (for 1.435-SNAPSHOT) with the changes described against the issue HUDSON-6817. It was not exactly the same (I used a call to syncIO instead of Channel.current().flushPipe(), since neither Channel.current() not channel.flushPipe() are available in the Jenkins source), but it made no difference. I still got the stream corruption.

I'm starting to think it's more environment than the code.... :/

@Tomas: I like the idea that my job is waiting for input. I'll look into that more. I'm spawning an exec from Ant and there used to be an issue about input. Nonetheless, when one of my jobs gets stuck, it'll run for 14 hours or more – and I manually stop it. These failures happen much sooner than 14 hours. I like the idea that I can change my ant script though ... don't hold much hope, however.

As we were only getting this recently, I downgraded the app back to version 1.419 (the version we were using before we started getting these errors). And we no longer get the streaming errors.

So, it's all circumstantial, but for the time being I'll stay on this version.

The unreliability was showing up for me, in May, as socket errors. Upgrading to Jenkins ver. 1.413 didn't fix the problem. Neither did 1.414 At some point the unreliability converted mostly into the "IOException: Unexpected termination of the channel" messages with occasional socket errors. The socket errors are described in JENKINS-9576.
Can't say if I've tried exactly 1.419 – that would include the fixes in 1.416 but not those in 1.420
I'm interested to see what 1.419 does for hildo in the long term...

Hi John. We were using v 1.419 for a while before I attempted this upgrade to the later releases. So, we hadn't experienced this stream corruption until I upgraded. So far, since reverting back to 1.419, our jobs have been completing reliably.

I know I'm pointing to more symptoms than cures. I didn't have the time to start disassembling out jeknins server to start debugging what would have been causing the error. It was impacting my entire delivery team, so I opted for the quick fix (which was reverting to the 1.419).

I guess your mileage might vary, given that it might have more to do with which plugins are installed, etc. For us, it's behaving again.

I'm now running 1.419 and I feel like I'm a kid waiting for the cookies to come out of the oven.

Okay, here it is:

IOException: Unexpected termination of the channel
Page generated: Oct 20, 2011 9:56:32 AM
Jenkins ver. 1.419

I'm now running Jenkins ver. 1.435

My mileage varied too... I got the stream corruption error eventually... so that version isn't as safe as I was professing.

I did notice issue JENKINS-8856, which got stream corruption errors when the master JVM was 64-bit and the slave JVM was 32-bit. I have the same setup but reverse (the default JVM for the Master on Solaris is 32-bit, which the default JVM on the Linux slave is 64-bit).

So, I've updated my Jenkins Master (running in a Tomcat instance) to use a 64-bit JVM. I will see if this makes any difference.

UPDATE: Once I've ensured that both JVMs are both 64-bit, my builds have been a lot more reliable. I've even moved back onto later releases of the JVMs, as well as upgrading to the most recent version of Jenkins (1.436). And I'm not getting the Stream Corruption errors I was getting. For now, I'm cautiously optimistic that the root of my problem was the master process running within a 32-bit JVM, with the slave running within a 64-bit JVM.

We had also a couple of situations ending up in IOExceptions with Unexpected termination of the channel. In our case we used Linux slaves, which run out of memory. The oom_killer killed the java process running the jenkins agent. This ended up in Unexpacted termination of the channel. Perhaps others had the same problem...

The problem've still randomly occurred.

FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
Page generated: Jan 23, 2012 2:35:42 PMJenkins ver. 1.447

I'm now running Jenkins ver. 1.448

Hi Andrew, given this issue is assigned to you, do you have thoughts on how to evaluate it? Any specific input users can provide to help?

My slave machine was working just fine and suddenly I started to see this problem when trying to launch the service from slave. Any thoughts anyone????

I get this message

java.io.EOFException: unexpected stream termination
at hudson.remoting.Channel.<init>(Channel.java:408)
at hudson.remoting.Channel.<init>(Channel.java:366)
at hudson.remoting.Channel.<init>(Channel.java:327)
at hudson.remoting.Channel.<init>(Channel.java:323)
at hudson.remoting.Channel.<init>(Channel.java:311)
at hudson.remoting.Engine.run(Channel.java:238)

@Atiq: doesn't look like the usual crashes I get, e.g. "Unexpected termination of the channel"
So, that's good – there's probably a fix for your problem.

I'm now running

Page generated: Mar 26, 2012 8:17:27 AM
Jenkins ver. 1.456

and still having these exceptions.

I was having this problem. The node had this:

• Debian Lenny i386
• Jenkins ver 1.456
• Java is sun-java6-jdk 6-26-9lenny1 (see http://packages.qa.debian.org/s/sun-java6.html for more version info)

I was able to resolve the issue by removing the package java-gcj-compat-headless.
This may not work for everyone but perhaps it is a clue to what is going wrong.

Hi Vince,

Um, you're saying you opened up a jar file and deleted the package?
On the Jenkins server or on a node? (Or both?)
What gave you the idea that this would help?

Thanks,
-John

I had this problem today, and it turned out to be because swap space was off, and Java was trying to allocate a very large amount of virtual memory.

HTH

Hi,

I´m not sure if this is the right place to write about our observations about channel termination.
Several times a day Jenkins loses some slaves. Currently we are running Jenkins 1.454, but we saw the problem also with 1.457. The master and almost all slaves are running on Linux machines.

I suspect the problem has something to do with garbage collection.
When the problem occurs, Jenkins occupies all 24 hyper-threads of CPU almost 100%.
Here the heap usage (from space 99%):

 
Heap
 PSYoungGen      total 7202048K, used 23148K [0x0000000600000000, 0x0000000800000000, 0x0000000800000000)
  eden space 7181888K, 0% used [0x0000000600000000,0x00000006002ecb30,0x00000007b6590000)
  from space 20160K, 99% used [0x00000007b6590000,0x00000007b793e4f8,0x00000007b7940000)
  to   space 26624K, 0% used [0x00000007fe600000,0x00000007fe600000,0x0000000800000000)
 PSOldGen        total 8388608K, used 2438211K [0x0000000200000000, 0x0000000400000000, 0x0000000600000000)
  object space 8388608K, 29% used [0x0000000200000000,0x0000000294d10d10,0x0000000400000000)
 PSPermGen       total 1048576K, used 85742K [0x00000001c0000000, 0x0000000200000000, 0x0000000200000000)
  object space 1048576K, 8% used [0x00000001c0000000,0x00000001c53bb888,0x0000000200000000)

Excerpt from Jenkins log (master)

 
Apr 24, 2012 11:14:16 AM hudson.remoting.Channel$ReaderThread run
SEVERE: I/O error in channel ullteb15
java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.Channel$ReaderThread.run(Channel.java:1133)
Caused by: java.io.EOFException
	at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
	at java.io.ObjectInputStream.readObject0(Unknown Source)
	at java.io.ObjectInputStream.readObject(Unknown Source)
	at hudson.remoting.Channel$ReaderThread.run(Channel.java:1127)
Apr 24, 2012 11:14:16 AM hudson.remoting.Channel$ReaderThread run
SEVERE: I/O error in channel ullteb16
java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.Channel$ReaderThread.run(Channel.java:1133)
Caused by: java.io.EOFException
	at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
	at java.io.ObjectInputStream.readObject0(Unknown Source)
	at java.io.ObjectInputStream.readObject(Unknown Source)
	at hudson.remoting.Channel$ReaderThread.run(Channel.java:1127)
Apr 24, 2012 11:14:26 AM hudson.remoting.Request$2 run
SEVERE: Failed to send back a reply
java.io.IOException: Broken pipe
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(Unknown Source)
	at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
	at java.io.BufferedOutputStream.flush(Unknown Source)
	at java.io.ObjectOutputStream$BlockDataOutputStream.flush(Unknown Source)
	at java.io.ObjectOutputStream.flush(Unknown Source)
	at hudson.remoting.Channel.send(Channel.java:505)
	at hudson.remoting.Request$2.run(Request.java:301)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Apr 24, 2012 11:14:26 AM hudson.remoting.Channel$ReaderThread run
SEVERE: I/O error in channel ulcppit01
java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.Channel$ReaderThread.run(Channel.java:1133)
Caused by: java.io.EOFException
	at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
	at java.io.ObjectInputStream.readObject0(Unknown Source)
	at java.io.ObjectInputStream.readObject(Unknown Source)
	at hudson.remoting.Channel$ReaderThread.run(Channel.java:1127)
Apr 24, 2012 11:14:27 AM hudson.remoting.Request$2 run
SEVERE: Failed to send back a reply
java.io.IOException: Broken pipe
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(Unknown Source)
	at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
	at java.io.BufferedOutputStream.flush(Unknown Source)
	at java.io.ObjectOutputStream$BlockDataOutputStream.flush(Unknown Source)
	at java.io.ObjectOutputStream.flush(Unknown Source)
	at hudson.remoting.Channel.send(Channel.java:505)
	at hudson.remoting.Request$2.run(Request.java:301)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Apr 24, 2012 11:14:26 AM hudson.remoting.Request$2 run
SEVERE: Failed to send back a reply

And here log from slave ullteb15

 
Ping failed. Terminating
ERROR: Connection terminated
ha:AAAAWB+LCAAAAAAAAABb85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0ldFVf2c+b/lb5MDAwVRQxSaBqcITRIIQMEMIIUFgAAckCEiWAAAAA=java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.Channel$ReaderThread.run(Channel.java:1133)
Caused by: java.io.EOFException
	at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
	at java.io.ObjectInputStream.readObject0(Unknown Source)
	at java.io.ObjectInputStream.readObject(Unknown Source)
	at hudson.remoting.Channel$ReaderThread.run(Channel.java:1127)
ERROR: Process terminated with exit code 255
ha:AAAAWB+LCAAAAAAAAABb85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0ldFVf2c+b/lb5MDAwVRQxSaBqcITRIIQMEMIIUFgAAckCEiWAAAAA=

Log from slave ullteb28

Apr 24, 2012 11:13:56 AM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel.
java.util.concurrent.TimeoutException: Ping started on 1335258596049 hasn't completed at 1335258836049
	at hudson.remoting.PingThread.ping(PingThread.java:114)
	at hudson.remoting.PingThread.run(PingThread.java:81)
Caused by: java.util.concurrent.TimeoutException
	at hudson.remoting.Request$1.get(Request.java:249)
	at hudson.remoting.Request$1.get(Request.java:184)
	at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
	at hudson.remoting.PingThread.ping(PingThread.java:107)
	... 1 more
Apr 24, 2012 11:13:56 AM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel.
java.util.concurrent.TimeoutException: Ping started on 1335258596049 hasn't completed at 1335258836053
	at hudson.remoting.PingThread.ping(PingThread.java:114)
	at hudson.remoting.PingThread.run(PingThread.java:81)
Caused by: java.util.concurrent.TimeoutException
	at hudson.remoting.Request$1.get(Request.java:249)
	at hudson.remoting.Request$1.get(Request.java:184)
	at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
	at hudson.remoting.PingThread.ping(PingThread.java:107)
	... 1 more
Connection terminated
channel stopped

Hi,

I did some investigation on this problem over last two months. My case was that this was happening on different linux flavors (but not on Windows build nodes). This started to happen after intensity and number of build jobs started to increase and was happening only for Scala projects. Build nodes are not unique, environment is mixed (some nodes are local, some nodes are on AWS, using two different types of VPN connection).

So after digging more and more, I figured out actually that my kernel wasn't tuned correctly. I increased number of file descriptors and maximum number of processes (as by guys before me it was never tuned) and jobs were doing much more faster, but this started to cause more and more issues.

So, it turned up that this was issue with RedHat based distribution and SSH - no matter I set /etc/security/limits.conf not all values (including this one with nproc setting) were set after SSH connection initiated from Jenkins master server to client(s). I tested it easily by making simple bash script which will connect from master using SSH and same user we use for builds to build nodes and issue ulimit -u and the value was default (1024).

Quick fix was to propagate change to all linux nodes to .bashrc and add explicitly 'ulimit -u value' there after what this stopped to occur and didn't happen anymore for a ten days now (and was happening multiple times a day, even hour). Also, if you want to tune it more elegantly, you should add 'session required pam_lmits.so' to /etc/pam.d/login what will force every new SSH connection to use defined limits from /etc/security/limits.conf.

Maybe this is not real fix for this issue, but this resolved my problem (and now, after thinking about structure of my jobs, intensity, requirements, etc. and basic values of limits, it was quite logical that this was happening to me) and maybe this will help as well someone else.

For us, the cause of this error was our build slaves (VMs) running out of memory and self-rebooting.

We had a similar issue to this (but with Linux slaves) that was resolved by switching to Oracle's JDK from Open JDK.

Even we face these issues frequently when performing svn operations from Jenkins.
Not sure what went wrong. Can anyone Explain what may be the reason ?

Error log:
FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request.call(Request.java:174)
at hudson.remoting.Channel.call(Channel.java:672)
at hudson.FilePath.act(FilePath.java:854)
at hudson.FilePath.act(FilePath.java:838)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:743)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:685)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1342)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:683)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:588)
at hudson.model.Run.execute(Run.java:1567)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:237)
Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request.abort(Request.java:299)
at hudson.remoting.Channel.terminate(Channel.java:732)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
Caused by: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2570)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
at hudson.remoting.Command.readFrom(Command.java:92)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)
at
hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

Hi,

our team also started to get this kind of error after upgrading subversion plugin from 1.34 to 1.45, jenkins version 1.505 java jvm: hotspot, jre: 1.6.0_22 And this is strange that this started to show up after subversion plugin update. This is not showing on testing jenkins where subversion plugin was 1.40 and was updated to 1.45 and everything works fine there.

FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
 hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
 	at hudson.remoting.Request.call(Request.java:174)
 	at hudson.remoting.Channel.call(Channel.java:672)
 	at hudson.FilePath.act(FilePath.java:893)
 	at hudson.FilePath.act(FilePath.java:877)
 	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:843)
 	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:781)
 	at hudson.model.AbstractProject.checkout(AbstractProject.java:1369)
 	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:676)
 	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
 	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:581)
 	at hudson.model.Run.execute(Run.java:1575)
 	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
 	at hudson.model.ResourceController.execute(ResourceController.java:88)
 	at hudson.model.Executor.run(Executor.java:241)
 Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
 	at hudson.remoting.Request.abort(Request.java:299)
 	at hudson.remoting.Channel.terminate(Channel.java:732)
 	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
 Caused by: java.io.IOException: Unexpected termination of the channel
 	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
 Caused by: java.io.EOFException
 	at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2553)
 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1296)
 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
 	at hudson.remoting.Command.readFrom(Command.java:92)
 	at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)
 	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

May 10, 2013 1:09:25 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
SEVERE: I/O error in channel slave_server
java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
	at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2553)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1296)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
	at hudson.remoting.Command.readFrom(Command.java:92)
	at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)

I had this problem when I witch my slave node with a low end system (an embedded system, 1GHz CPU, 1G RAM, Fedora OS). All things were well before I replaced it. So I'm sure this is caused by the java memory usage.

After some searching from internet, I added one option to the salve node advanced "JVM Options" with "-Xmx256m". And this problem is gone. I don't think it is applied for others, it is depended on user's case, some one has this error because java can't get enough memory, and other may has this error because java exhaust system memory.

Sorry, my comments above(deleted) is wrong. This bug is similar with https://issues.jenkins-ci.org/browse/JENKINS-13341 , switching the slave to use Oracle's JDK (1.8.0, early access) indeed fixed it.

We are also getting this rather disconcertingly often on linux slaves. Is there a sar command that I can use to demonstrate a system resource scarcity during the execution of these builds? Because our builds are not even starting and all the SAR data that I have looked at tells me this is NOT a resource issue on the slaves. BUT I am not a sar master and perhaps I am looking at the wrong things.

Started by upstream project "dp_10.0_begin_nightly" build number 107
originally caused by:
Started by timer
Building remotely on robusta in workspace /home/hudson/ci/jenkins/slaves/robusta/workspace/dp_10.0_marker_nightly
hudson.util.IOException2: remote file operation failed: /home/hudson/ci/jenkins/slaves/robusta/workspace/dp_10.0_marker_nightly at hudson.remoting.Channel@789e3ac5:robusta
at hudson.FilePath.act(FilePath.java:900)
at hudson.FilePath.act(FilePath.java:877)
at hudson.FilePath.mkdirs(FilePath.java:1017)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1365)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:674)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:579)
at hudson.model.Run.execute(Run.java:1575)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:237)
Caused by: hudson.remoting.ChannelClosedException: channel is already closed
at hudson.remoting.Channel.send(Channel.java:494)
at hudson.remoting.Request.call(Request.java:129)
at hudson.remoting.Channel.call(Channel.java:672)
at hudson.FilePath.act(FilePath.java:893)
... 10 more
Caused by: java.io.IOException
at hudson.remoting.Channel.close(Channel.java:910)
at hudson.slaves.ChannelPinger$1.onDead(ChannelPinger.java:110)
at hudson.remoting.PingThread.ping(PingThread.java:120)
at hudson.remoting.PingThread.run(PingThread.java:81)
Caused by: java.util.concurrent.TimeoutException: Ping started on 1376069782625 hasn't completed at 1376070022626
... 2 more
Build step 'Groovy Postbuild' marked build as failure

This one almost made it:

[xUnit] [INFO] - Starting to record.
[xUnit] [INFO] - Processing JUnit
[xUnit] [INFO] - [JUnit] - 114 test report file(s) were found with the pattern 'src/system-tests/smoke/test/reports*/*.xml' relative to '/home/jenkins/jobs/workspace/bla' for the testing framework 'JUnit'.
ERROR: Publisher org.jenkinsci.plugins.xunit.XUnitPublisher aborted due to exception
hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
	at hudson.remoting.Request.call(Request.java:174)
	at hudson.remoting.Channel.call(Channel.java:714)
	at hudson.FilePath.act(FilePath.java:898)
	at hudson.FilePath.act(FilePath.java:882)
	at org.jenkinsci.plugins.xunit.XUnitPublisher.performTests(XUnitPublisher.java:184)
	at org.jenkinsci.plugins.xunit.XUnitPublisher.performXUnit(XUnitPublisher.java:119)
	at org.jenkinsci.plugins.xunit.XUnitPublisher.perform(XUnitPublisher.java:93)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:776)
	at hudson.model.Build$BuildExecution.post2(Build.java:183)
	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:726)
	at hudson.model.Run.execute(Run.java:1628)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:247)
Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.Request.abort(Request.java:299)
	at hudson.remoting.Channel.terminate(Channel.java:774)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
Caused by: java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
	at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2595)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1315)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1988)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1912)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1795)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
	at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1703)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1341)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1988)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1912)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1795)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1988)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1912)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1795)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
	at hudson.remoting.Command.readFrom(Command.java:92)
	at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
ERROR: Publisher hudson.tasks.Mailer aborted due to exception
hudson.remoting.ChannelClosedException: channel is already closed
	at hudson.remoting.Channel.send(Channel.java:516)
	at hudson.remoting.Request.call(Request.java:129)
	at hudson.remoting.Channel.call(Channel.java:714)
	at hudson.EnvVars.getRemote(EnvVars.java:212)
	at hudson.model.Computer.getEnvironment(Computer.java:908)
	at jenkins.model.CoreEnvironmentContributor.buildEnvironmentFor(CoreEnvironmentContributor.java:40)
	at hudson.model.Job.getEnvironment(Job.java:374)
	at hudson.model.AbstractProject.getEnvironment(AbstractProject.java:353)
	at hudson.model.Run.getEnvironment(Run.java:2077)
	at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:933)
	at hudson.tasks.Mailer.perform(Mailer.java:114)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:776)
	at hudson.model.Build$BuildExecution.post2(Build.java:183)
	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:726)
	at hudson.model.Run.execute(Run.java:1628)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:247)
Caused by: java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
	at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2595)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1315)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1988)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1912)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1795)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
	at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1703)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1341)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1988)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1912)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1795)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1988)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1912)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1795)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
	at hudson.remoting.Command.readFrom(Command.java:92)
	at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

Hi, for my issue investigation pointed to svnkit, in more general gnome-keyring glib is crashing VM. I found that this issue is also connected to subversion checkout lock.

Mainly problem is when couple of builds are running on slave and randomly some build checkout's from subversion( slave's VM crashes) and for other builds in the log it seams that connections was dropped

Solution is to add java vm variable:
-Dsvnkit.library.gnome-keyring.enabled=false

Info on svn lock during checkout with thread dump:
https://issues.jenkins-ci.org/browse/JENKINS-16427

P.S.
Tested on couple of Jenkins instances and linux slaves. Since introducing vm variable jenkins works stable for almost week(before, one of Jenkins instance slaves crashed everyday).

In my case, this exceptions seems to be the result of an overloaded machine. I don't think this is something you "fix" so much as I'd like Jenkins to let me increase the timeout value so it doesn't close the channel for such overloaded machines.

I did the following according to Chris Z's post:

• added -Dsvnkit.library.gnome-keyring.enabled=false to the slave launch options
• set "password-stores = kwallet keychain windows-cryptoapi" in the .subversion/config file on the master

Still ran into this...

We are seeing this constantly on our build slaves during gcc and ARM compilation:

FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
at hudson.remoting.Request.call(Request.java:174)
at hudson.remoting.Channel.call(Channel.java:722)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:167)
at com.sun.proxy.$Proxy47.join(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:925)
at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:137)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
at hudson.model.Build$BuildExecution.build(Build.java:199)
at hudson.model.Build$BuildExecution.doRun(Build.java:160)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:585)
at hudson.model.Run.execute(Run.java:1676)
at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:231)
Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request.abort(Request.java:299)
at hudson.remoting.Channel.terminate(Channel.java:782)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
Caused by: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.Command.readFrom(Command.java:92)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

I've completely turned off the Response Time check, bumped the heap size to 512MB (monitoring indicates that this is ample), ensured that the slave is using Oracle JRE (1.6.0_45-b06).

Master is Ubuntu 10.04 (fully updated) 8 core, 16GB RAM physical host with Jenkins 1.555.
Slave is Ubuntu 10.04 (fully updated) 4 core, 1GB RAM living on Suse KVM host. Monitoring indicates that CPU consumption and load average are definitely high (compilation, etc).

This is a really ugly issue. What is the status of development?

You know, I could be wrong, but it seems I get this error message less after having switched all my machines from WIFI to wired Ethernet.

This is killing me, I get this 6 times a day but only on my windows slaves. The amount of resources I give to them makes no difference to the chance of failure (they are all quad core with 6Gb of RAM now)

I've been able to "sort of" get around this by only sending IO and CPU heavy stuff to dedicated, powerful systems, reducing the number of executors on each slave to 1, reducing the number of threads to half the number of cores.

Totally lame workaround, but "done at all" is better than the stupid stack dump.

Please make sure to specify Jenkins versions when adding "me too" comments.

We have seen these issues in an unreliable network but one tweak that has helped a lot is to remove the monitoring plugin. That plugin has the unfortunate property of severing connections to slaves if they don't respond to a single ping. The connection will be severed regardless of the slave running jobs at that time.

@ritzmann: Which monitoring plugin?

Which monitoring plugin?

https://wiki.jenkins-ci.org/display/JENKINS/Monitoring

Same issue her with 1.567:

NOTE: recipe attr-native-2.4.46-r4: task do_configure: Succeeded
FATAL: hudson.remoting.RequestAbortedException: hudson.remoting.Channel$OrderlyShutdown
hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: hudson.remoting.Channel$OrderlyShutdown
at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
at hudson.remoting.Request.call(Request.java:174)
at hudson.remoting.Channel.call(Channel.java:739)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:168)
at com.sun.proxy.$Proxy70.join(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:951)
at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:137)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:756)
at hudson.model.Build$BuildExecution.build(Build.java:198)
at hudson.model.Build$BuildExecution.doRun(Build.java:159)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1710)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:234)
Caused by: hudson.remoting.RequestAbortedException: hudson.remoting.Channel$OrderlyShutdown
at hudson.remoting.Request.abort(Request.java:299)
at hudson.remoting.Channel.terminate(Channel.java:802)
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:951)
at hudson.remoting.Channel$2.handle(Channel.java:475)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)
Caused by: hudson.remoting.Channel$OrderlyShutdown
... 3 more
Caused by: Command close created at
at hudson.remoting.Command.<init>(Command.java:56)
at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:945)
at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:943)
at hudson.remoting.Channel.close(Channel.java:1026)
at hudson.remoting.Channel.close(Channel.java:1009)
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:950)
... 2 more

same here on 1.565.1, slave and master are both in the same LAN on Amazon EC2. Quite a frustrating bug.

I have seen the same with v1.583, I haven't found a solution yet but did anyone ever checked the sshd syslog?

I observed that each time a job fails due to "java.io.IOException: Unexpected termination of the channel" there is a syslog entry for some changes in the underlying ssh connection:
Oct 8 01:30:14 servername sshd[pid]: Generating new 1024 bit RSA key.
Oct 8 01:30:14 servername sshd[pid]: RSA key generation complete.

can anyone check if you see the same?

slaves OS: Scientific Linux release 6.5 (Carbon) and CentOS release 6.5 (Final)

KeyRegenerationInterval
             In protocol version 1, the ephemeral server key is automatically regenerated after this
             many seconds (if it has been used).  The purpose of regeneration is to prevent decrypting
             captured sessions by later breaking into the machine and stealing the keys.  The key is
             never stored anywhere.  If the value is 0, the key is never regenerated.  The default is
             3600 (seconds).

The default value for this setting will generate an ephemerial 1024bit RSA key for the sshd process every 3600 sec. Try setting it to zero?

I suggest doing the opposite. Set the value to a small number, like 10 seconds, to make the issue easier to reproduce and investigate a fix.

@Ian: that's fair enough but I'd expect for ssh key regeneration to be done in the background without breaking the connection.
@cowwoc: I'll try that but I'm not yet sure where to change the setting, should it be on the master node of slave? I'll do a bit of reading on how ssh does this

@Calin Marina : i am using Jenkins LTS versions and i don't have this issue ... to much . Maybe consider switching to LTS. Much more stable in a production enviroment

unfortunately I only have full access to the slaves not on the master server... master is administered by others and I can only lobby for changes for now

that is very bad...especially if you need more access to configure. I am the build manager for my team so it is simpler for all devs because i am very fast to approach

https://issues.jenkins-ci.org/browse/JENKINS-14332 helped for me.

Sometimes I have this problem when running Jenkins 1.637 (Tomcat 8.0.28 / Windows 10)

[WARNING] Couldn't clean up oid=9 from null
hudson.remoting.ChannelClosedException: channel is already closed
	at hudson.remoting.Channel.send(Channel.java:575)
	at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.cleanup(RemoteInvocationHandler.java:360)
	at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.access$700(RemoteInvocationHandler.java:319)
	at hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:420)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
	at java.lang.Thread.run(Thread.java:745)
Caused by: hudson.remoting.Channel$OrderlyShutdown
	at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1077)
	at hudson.remoting.Channel$1.handle(Channel.java:498)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)
Caused by: Command close created at
	at hudson.remoting.Command.<init>(Command.java:56)
	at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1071)
	at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1069)
	at hudson.remoting.Channel.close(Channel.java:1153)
	at hudson.remoting.Channel.close(Channel.java:1135)
	at hudson.maven.ProcessCache$MavenProcess.discard(ProcessCache.java:150)
	at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:861)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:537)
	at hudson.model.Run.execute(Run.java:1741)
	at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:531)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:410)
Finished: SUCCESS

In Jenkins ver. 1.625 this problem shows up as:

Slave went offline during the build
ERROR: Connection was broken: java.io.IOException: Connection aborted: org.jenkinsci.remoting.nio.NioChannelHub$MonoNioTransport@18bfcae9[name=Arnold]
	at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.abort(NioChannelHub.java:208)
	at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:628)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: An existing connection was forcibly closed by the remote host
	at sun.nio.ch.SocketDispatcher.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(Unknown Source)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
	at sun.nio.ch.IOUtil.read(Unknown Source)
	at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
	at org.jenkinsci.remoting.nio.FifoBuffer$Pointer.receive(FifoBuffer.java:136)
	at org.jenkinsci.remoting.nio.FifoBuffer.receive(FifoBuffer.java:306)
	at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:561)
	... 6 more

Hi John,
I had the same problem one year ago with an older version.
Try to launch the slave with a batch instead of a service and redirect the standard output to a file to check what happens.
Honestly I solved disabling the service and leaving the slave running with a scheduled batch.
Graz

Any resolution for connection timeout issues ?

Ubuntu 14.04 server 64-bit
oracle-java7: 1.7.0_80
Jenkins: 1.651.1 LTS

The build sometimes randomly failed with this kind of error.

This time happened in the post-build actions:
FATAL: channel is already closed
hudson.remoting.ChannelClosedException: channel is already closed
at hudson.remoting.Channel.send(Channel.java:578)
at hudson.remoting.Request.call(Request.java:130)
at hudson.remoting.Channel.call(Channel.java:780)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:953)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:540)
at hudson.model.Run.execute(Run.java:1738)
at hudson.matrix.MatrixBuild.run(MatrixBuild.java:313)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Caused by: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

I got similar issues as below

FATAL: hudson.remoting.Channel$OrderlyShutdown: java.util.concurrent.TimeoutException: Ping started on 1467229478054 hasn't completed at 1467229718378
hudson.remoting.RequestAbortedException: hudson.remoting.Channel$OrderlyShutdown: java.util.concurrent.TimeoutException: Ping started on 1467229478054 hasn't completed at 1467229718378
	at hudson.remoting.Request.abort(Request.java:295)
	at hudson.remoting.Channel.terminate(Channel.java:814)
	at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1029)
	at hudson.remoting.Channel$2.handle(Channel.java:483)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)
	at ......remote call to 10.79.17.177(Native Method)
	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356)
	at hudson.remoting.Request.call(Request.java:171)
	at hudson.remoting.Channel.call(Channel.java:751)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:173)
	at $Proxy36.join(Unknown Source)
	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:979)
	at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:137)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:770)
	at hudson.model.Build$BuildExecution.build(Build.java:199)
	at hudson.model.Build$BuildExecution.doRun(Build.java:160)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:533)
	at hudson.model.Run.execute(Run.java:1745)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:89)
	at hudson.model.Executor.run(Executor.java:240)
Caused by: hudson.remoting.Channel$OrderlyShutdown: java.util.concurrent.TimeoutException: Ping started on 1467229478054 hasn't completed at 1467229718378
	at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1029)
	at hudson.remoting.Channel$2.handle(Channel.java:483)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)
Caused by: Command close created at
	at hudson.remoting.Command.<init>(Command.java:56)
	at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1023)
	at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1021)
	at hudson.remoting.Channel.close(Channel.java:1104)
	at hudson.slaves.ChannelPinger$1.onDead(ChannelPinger.java:110)
	at hudson.remoting.PingThread.ping(PingThread.java:125)
	at hudson.remoting.PingThread.run(PingThread.java:86)
Caused by: java.util.concurrent.TimeoutException: Ping started on 1467229478054 hasn't completed at 1467229718378
	... 2 more    

My environment:
jenkins master runs on CentOS release 6.4 (Final). jenkins version is : 1.580.3
jenkins slave in which am facing issue is a solaris vm. This started recently and I could'nt think of any environment changes happened. When, this happened initially, I thought that swap space is not enough and I have increased the swap space of the slave. Any suggestions is much appreciated. When I hit this issue, I just reboot the slave and then one or two build works fine. Have not seen this problem after three builds, but not sure, when I will hit this.