If I understand you correctly, Hudson starts a shell (on a slave) and runs your
script, which in turn run ssh and starts a process on yet another machine?

The exception indicates that the link between the master and the slave are
terminated unexpectedly. How does your master and slave talk to each other?

Finally, I didn't follow your reasoning about ProcessTreeKiller and why that
hints a bug in the remoting code.

I apologize if I was vague. The job is just a shell execute but it must run on a
particular environment. Thus, it is tied to a slave and that slave is started
via ssh command from master.

The shell script starts by copying tables from a remote data store using mysql
client. One of those tables is very large and takes just over two hours to copy.
While it is copying there is obviously TCP activity between the slave and the
remote data store but the slave doesn't send any logging info back to the master
for the entire two hour+ period. Since upgrading hudson to 1.293 from 1.278. The
connection seems to be getting dropped at some point during this two hour period.

Before I turned off ProcessTreeKiller the underlying mysql transfer was
terminiating with the hudson job. However, now the command started by the
hudson job is completing on the slave but the slave reports failure.

adding myself as CC

It seems I'm hitting the same problem with just 5 seconds of sleep time (the job
is executing a shell script that in turn calls ant):

check-resources-library:
[echo] Javascript Library 1_2 available = true
[echo] The file is checked at:
/export/home/j2eetest/hudson/workspace/JSF-core/glassfishv3/glassfish/domains/domain1/applications/guessNumber/resources/js/1_2/validator.js
[echo] Image Library 1_2 available = true
[echo] the file is checked at:
/export/home/j2eetest/hudson/workspace/JSF-core/glassfishv3/glassfish/domains/domain1/applications/guessNumber/resources/images/1_2/wave.med.gif
[echo] Sleeping for 5 seconds...
FATAL: command execution failed
hudson.util.IOException2: Failed to join the process
at hudson.Proc$RemoteProc.join(Proc.java:297)
at hudson.Launcher$ProcStarter.join(Launcher.java:274)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:84)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
at hudson.model.Build$RunnerImpl.build(Build.java:195)
at hudson.model.Build$RunnerImpl.doRun(Build.java:151)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:272)
at hudson.model.Run.run(Run.java:928)
at hudson.model.Build.run(Build.java:112)
at hudson.model.ResourceController.execute(ResourceController.java:93)
at hudson.model.Executor.run(Executor.java:118)
Caused by: java.util.concurrent.ExecutionException:
hudson.remoting.RequestAbortedException: java.io.EOFException
at hudson.remoting.Request$1.get(Request.java:188)
at hudson.remoting.Request$1.get(Request.java:157)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55)
at hudson.Proc$RemoteProc.join(Proc.java:289)
... 10 more
Caused by: hudson.remoting.RequestAbortedException: java.io.EOFException
at hudson.remoting.Request.abort(Request.java:223)
at hudson.remoting.Channel.terminate(Channel.java:558)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:776)
Caused by: java.io.EOFException
at
java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:757)
FATAL: Unable to delete script file /tmp/hudson8537360715477296990.sh
hudson.util.IOException2: remote file operation failed
at hudson.FilePath.act(FilePath.java:645)
at hudson.FilePath.act(FilePath.java:633)
at hudson.FilePath.delete(FilePath.java:863)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:94)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
at hudson.model.Build$RunnerImpl.build(Build.java:195)
at hudson.model.Build$RunnerImpl.doRun(Build.java:151)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:272)
at hudson.model.Run.run(Run.java:928)
at hudson.model.Build.run(Build.java:112)
at hudson.model.ResourceController.execute(ResourceController.java:93)
at hudson.model.Executor.run(Executor.java:118)
Caused by: java.io.IOException: already closed
at hudson.remoting.Channel.send(Channel.java:372)
at hudson.remoting.Request.call(Request.java:104)
at hudson.remoting.Channel.call(Channel.java:511)
at hudson.FilePath.act(FilePath.java:640)
... 11 more
FATAL: already closed
java.io.IOException: already closed
at hudson.remoting.Channel.send(Channel.java:372)
at hudson.remoting.Request.call(Request.java:104)
at hudson.remoting.Channel.call(Channel.java:511)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:730)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:277)
at hudson.model.Run.run(Run.java:928)
at hudson.model.Build.run(Build.java:112)
at hudson.model.ResourceController.execute(ResourceController.java:93)
at hudson.model.Executor.run(Executor.java:118)

This job executes fine on solaris but fails on linux RH5.

adding myself to cc list

I am having the same issue on CentOS 5.

....F...............FATAL: rake execution failed
hudson.util.IOException2: Failed to join the process
at hudson.Proc$RemoteProc.join(Proc.java:297)
at hudson.plugins.rake.Rake.perform(Rake.java:101)
at
hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:56)
at hudson.model.Build$RunnerImpl.build(Build.java:195)
at hudson.model.Build$RunnerImpl.doRun(Build.java:151)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:271)
at hudson.model.Run.run(Run.java:938)
at hudson.model.Build.run(Build.java:112)
at hudson.model.ResourceController.execute(ResourceController.java:93)
at hudson.model.Executor.run(Executor.java:118)
Caused by: java.util.concurrent.ExecutionException:
hudson.remoting.RequestAbortedException: java.io.EOFException
at hudson.remoting.Request$1.get(Request.java:188)
at hudson.remoting.Request$1.get(Request.java:157)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55)
at hudson.Proc$RemoteProc.join(Proc.java:289)
... 9 more
Caused by: hudson.remoting.RequestAbortedException: java.io.EOFException
at hudson.remoting.Request.abort(Request.java:223)
at hudson.remoting.Channel.terminate(Channel.java:558)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:776)
Caused by: java.io.EOFException
at
java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2570)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:757)
FATAL: already closed
java.io.IOException: already closed
at hudson.remoting.Channel.send(Channel.java:372)
at hudson.remoting.Request.call(Request.java:104)
at hudson.remoting.Channel.call(Channel.java:511)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:730)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:276)
at hudson.model.Run.run(Run.java:938)
at hudson.model.Build.run(Build.java:112)
at hudson.model.ResourceController.execute(ResourceController.java:93)
at hudson.model.Executor.run(Executor.java:118)

When this happens, the slave log might show some record of why the communication
with the slave JVM failed. Can you please check them?

I hit this error as well on the slave. The slave log seemed to be basically empty, but the Hudson's main log had this which seemed
to correspond with the slave and time:

16/07/2009 3:32:31 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run
WARNING: Failed to monitor Worker 4 for Free Temp Space
hudson.util.IOException2: remote file operation failed
at hudson.FilePath.act(FilePath.java:548)
at hudson.node_monitors.TemporarySpaceMonitor$1.getFreeSpace(TemporarySpaceMonitor.java:71)
at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:80)
at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:43)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:161)
Caused by: java.io.IOException: Unable to serialize 229391015936
at hudson.remoting.UserRequest.serialize(UserRequest.java:134)
at hudson.remoting.UserRequest.perform(UserRequest.java:100)
at hudson.remoting.UserRequest.perform(UserRequest.java:46)
at hudson.remoting.Request$2.run(Request.java:236)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:54)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.NotSerializableException: hudson.node_monitors.DiskSpaceMonitorDescriptor$DiskSpace
at java.io.ObjectOutputStream.writeObject0(Unknown Source)
at java.io.ObjectOutputStream.writeObject(Unknown Source)
at hudson.remoting.UserRequest._serialize(UserRequest.java:123)
at hudson.remoting.UserRequest.serialize(UserRequest.java:132)
... 10 more

This stacktrace is reported here:

https://hudson.dev.java.net/issues/show_bug.cgi?id=3381, which has been fixed already, however in 1.296, where-as we are running
1.295, so we are updating now. Hopefully that will fix this issue reported here.