Francesco Chemolli added a comment - 2011-04-26 09:41 Same issue on OpenBSD 4.8, both 32- and 64-bit.

vila added a comment - 2011-05-06 10:43 Same issue here with a master running under Ubuntu natty and slaves (virtualbox VMS) running under FreedBSD, gentoo, ubuntu hardy, lucid, maverick, natty and windows but also with a real host running OSX 10.6. I think we can rule out an OS specific issue

vila added a comment - 2011-05-06 13:48 So, it seems to be more than intermittent here, anything I can do to help diagnose ? It now occurs on a regular basis across all the slaves and may even always for some of them.

vila added a comment - 2011-05-13 07:11 More evidence on a specific setup: One of the slave (macadam.local) is a real host (as opposed to a virtual machine running on the same host than the master). It's configure with 'Launch slave via execution of command on the Master' with sh -c "${BABUNE_ROOT}/bin/start-slave-via-ssh.sh macadam.local" The start-slave-via-ssh.sh is: #!/bin/bash -x This script start a slave via ssh under some assumptions: - slave.jar needs to be up to date, - respect the master ssh config and forward the agent NODE=$1 JAR_REMOTE_PATH='~/babune/slaves/${NODE}/slave.jar' We start in /, let's go to a friendler place cd ${BABUNE_ROOT} Copy the slave.jar file scp master/war/WEB-INF/slave.jar ${NODE}:${JAR_REMOTE_PATH} Start slave.jar ssh ${NODE} java -jar ${JAR_REMOTE_PATH} It's needed because I prefer to handle the ssh settings in ~/.ssh/config which is not respected when using 'Launch slave agents on UNIX machines via ssh' Now, while this slave is started and a job is running, I started a wireshark on the master with a capture filter of 'host macadam.local and port 22' and also watch the job console displaying progress as tests are passing. The suspicious thing there is that while packets are flowing (slowly but flowing) on the ssh connection (as shown by wireshark), no updates happen on the console. Tests generally takes far less than one second and even after waiting for minutes, nothing appears on the console. Does that ring some bells ? Is there some way to make jenkins a bit more verbose in this area ? This seems to indicate that the connection between the master and the slave is broken at a higher level than the ssh one. I suspect that the master finally gave up somehow but still try to use the connection to delete the remote file and fail.

Uwe Stuehler added a comment - 2011-08-26 08:54 - edited Some more evidence, yet no solution... :-| OpenBSD 5.0-current and Jenkins 1.427. In the middle of a kernel build, suddenly there is no activity in the console log for a long time and then I get this very similar stack trace. FATAL: Unable to delete script file /tmp/hudson2998673953817336628.sh hudson.util.IOException2: remote file operation failed: /tmp/hudson2998673953817336628.sh at hudson.remoting.Channel@652b32ea:openbsd at hudson.FilePath.act(FilePath.java:754) at hudson.FilePath.act(FilePath.java:740) at hudson.FilePath.delete(FilePath.java:995) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:682) at hudson.model.Build$RunnerImpl.build(Build.java:178) at hudson.model.Build$RunnerImpl.doRun(Build.java:139) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:448) at hudson.model.Run.run(Run.java:1376) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:230) Caused by: hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:486) at hudson.remoting.Request.call(Request.java:110) at hudson.remoting.Channel.call(Channel.java:668) at hudson.FilePath.act(FilePath.java:747) ... 13 more Caused by: hudson.remoting.Channel$OrderlyShutdown at hudson.remoting.Channel$CloseCommand.execute(Channel.java:813) at hudson.remoting.Channel$ReaderThread.run(Channel.java:1049) Caused by: Command close created at [...] It seems that the build process stalls trying to write to stdout because the java slave process no longer consumes the output of the build process. Before I the build was aborted, I had a chance to look at the process table $ pstree [...] | \-+= 07917 root sshd: hudson [priv] (sshd) | \-+- 26930 hudson sshd: hudson@notty (sshd) | \-+= 03191 hudson ksh -c ksh | \-+- 18833 hudson /home/hudson/jdk/bin/java -jar slave.jar | \-+- 30004 hudson /bin/sh -xe /tmp/hudson2998673953817336628.sh | \-+- 07697 hudson /bin/sh -xe /tmp/hudson2998673953817336628.sh | \--- 16885 hudson make [...] $ ps auxOwchan|grep make hudson 16885 0.0 0.2 2788 3740 ?? I 8:24AM 0:00.04 make pipewr $ sudo ktrace -p 18833 $ sudo kdump |head -15 18833 java EMUL "native" 18833 java RET read 51/0x33 18833 java CALL read(0,0x20cd76570,0x2000) 18833 java GIO fd 0 read 696 bytes "xr\0\^Whudson.remoting.Command\0\0\0\0\0\0\0\^A\^B\0\^AL\0 createdAtt\0\^ULjava/lang/Exception;xpsr\0\^^hudson.remoting.Command$Source\0\0\0\0\0\0\0\^A\^B\0\^AL\0\^Fthis$0t\0\^YLhudson/remoting/\ Command;xr\0\^Sjava.lang.Exception\M-P\M-}\^_>\^Z;\^\\M-D\^B\0\0xr\0\^Sjava.lang.Throwable\M-U\M-F5'9w\M-8\M-K\^C\0\^CL\0\^Ecauset\0\^ULjava/lang/Throwable;L\0\rdetailMessaget\0\^RLjava/lang/String;[\ \0 stackTracet\0\^^[Ljava/lang/StackTraceElement;xpq\0~\0\vpur\0\^^[Ljava.lang.StackTraceElement;\^BF*<<\M-}"9\^B\0\0xp\0\0\0\bsr\0\^[java.lang.StackTraceElementa \M-E\M^Z&6\M-]\M^E\^B\0\^DI\0 lineNumberL\0\^NdeclaringClassq\0~\0 L\0\bfileNameq\0~\0 L\0 methodNameq\0~\0 xp\0\0\0>t\0\^Whudson.remoting.Commandt\0\fCommand.javat\0\^F<init>sq\0~\0\^N\0\0\0/q\0~\0\^Pq\0~\0\^Qq\0~\0\^Rsq\0~\0\^N\0\0\^C)t\0$hudson.remoting.Channel$CloseCommandt\0\fC\ hannel.javaq\0~\0\^Rsq\0~\0\^N\0\0\^C)q\0~\0\^Uq\0~\0\^Vq\0~\0\^Rsq\0~\0\^N\0\0\^CV" 18833 java RET read 696/0x2b8 18833 java CALL gettimeofday(0x20cd77c40,0) 18833 java PSIG SIGPROF caught handler=0x2034ee1e0 mask=0x0 18833 java RET gettimeofday 0

Uwe Stuehler added a comment - 2011-08-26 09:22 More more food for thought. I was accidentally still using pthreads even though I normally replace libpthread with librthread during the build. However when the issue came up, it didn't help to restart the slave - the issue was showing up consistently in every single build. But now that I've actually enabled librthread again and rebooted the slave once more, it builds just fine, as usual.

Gerwin Jansen added a comment - 2011-09-27 12:32 I have a very similar issue when I use a hudson job to do a very time-consuming svn merge. The console doesn't produce any output when this long svn merge is pending (this is expected). After 38 minutes the below error happens. I triggered the job twice, and both failed after 38 minutes. System spec Jenkins 1.431 [root@srv-ind-dvm17zl ~]# cat /etc/*release Red Hat Enterprise Linux Server release 5.4 (Tikanga) [root@srv-ind-dvm17zl ~]# uname -a Linux srv-ind-dvm17zl.vanenburg.com 2.6.18-164.el5 #1 SMP Tue Aug 18 15:51:48 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux [root@srv-ind-dvm17zl ~]# java -version java version "1.6.0_20" Java(TM) SE Runtime Environment (build 1.6.0_20-b02) Java HotSpot(TM) Server VM (build 16.3-b01, mixed mode) Console log setendrevision: [echo] Prepare DROP-merge command || revision : revision [getmergecmd] [getmergecmd] Merge command: svn merge -r 189757:199394 "http: //srv-ind-scrat:8080/repos/bcp/branches/ private /batic/batic-wip" "/opt/jenkins/workspace/batic-wip-dropmerge/trunk" --accept postpone --non-interactive FATAL: Unable to delete script file /tmp/hudson8787918218652977262.sh hudson.util.IOException2: remote file operation failed: /tmp/hudson8787918218652977262.sh at hudson.remoting.Channel@7c1f7480:srv-ind-dvm17zl at hudson.FilePath.act(FilePath.java:754) at hudson.FilePath.act(FilePath.java:740) at hudson.FilePath.delete(FilePath.java:995) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:693) at hudson.model.Build$RunnerImpl.build(Build.java:178) at hudson.model.Build$RunnerImpl.doRun(Build.java:139) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:459) at hudson.model.Run.run(Run.java:1376) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:230) Caused by: hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:499) at hudson.remoting.Request.call(Request.java:110) at hudson.remoting.Channel.call(Channel.java:681) at hudson.FilePath.act(FilePath.java:747) ... 13 more Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Channel$ReaderThread.run(Channel.java:1093) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2570) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368) at hudson.remoting.Channel$ReaderThread.run(Channel.java:1087) FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request.call(Request.java:149) at hudson.remoting.Channel.call(Channel.java:681) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158) at $Proxy38.join(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:850) at hudson.Launcher$ProcStarter.join(Launcher.java:336) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:82) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:693) at hudson.model.Build$RunnerImpl.build(Build.java:178) at hudson.model.Build$RunnerImpl.doRun(Build.java:139) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:459) at hudson.model.Run.run(Run.java:1376) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:230) Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request.abort(Request.java:273) at hudson.remoting.Channel.terminate(Channel.java:732) at hudson.remoting.Channel$ReaderThread.run(Channel.java:1117) Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Channel$ReaderThread.run(Channel.java:1093) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2570) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368) at hudson.remoting.Channel$ReaderThread.run(Channel.java:1087)

Kishore Sasidharan added a comment - 2012-01-24 20:30 Same issue here with jenkins-1.408 on Centos - slaves on FreeBSD.