-
Bug
-
Resolution: Duplicate
-
Major
-
None
-
Windows XP master - Windows XP slave, Linux
-
Powered by SuggestiMate
When executing a free-style job in a Windows agent, this error appers:
//wc/psp/Branches/Release/PSP/content/EaTrax/MichaelFranti.wav#1 - added as P:\Release\PSP\content\EaTrax\MichaelFranti.wav //wc/psp/Branches/Release/PSP/content/EaTrax/MiikeSnow.wav#1 - added as P:\Release\PSPFATAL: command execution failed hudson.util.IOException2: Failed to join the process at hudson.Proc$RemoteProc.join(Proc.java:311) at hudson.Launcher$ProcStarter.join(Launcher.java:275) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:83) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:577) at hudson.model.Build$RunnerImpl.build(Build.java:165) at hudson.model.Build$RunnerImpl.doRun(Build.java:133) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:417) at hudson.model.Run.run(Run.java:1176) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:123) Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.io.EOFException at hudson.remoting.Request$1.get(Request.java:218) at hudson.remoting.Request$1.get(Request.java:172) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55) at hudson.Proc$RemoteProc.join(Proc.java:303) ... 12 more Caused by: hudson.remoting.RequestAbortedException: java.io.EOFException at hudson.remoting.Request.abort(Request.java:257) at hudson.remoting.Channel.terminate(Channel.java:594) at hudson.remoting.Channel$ReaderThread.run(Channel.java:872) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Channel$ReaderThread.run(Channel.java:852) FATAL: Unable to delete script file C:\DOCUME~1\pcfarm08\LOCALS~1\Temp\2\hudson2101488692771117152.bat hudson.util.IOException2: remote file operation failed at hudson.FilePath.act(FilePath.java:677) at hudson.FilePath.act(FilePath.java:665) at hudson.FilePath.delete(FilePath.java:922) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:93) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:577) at hudson.model.Build$RunnerImpl.build(Build.java:165) at hudson.model.Build$RunnerImpl.doRun(Build.java:133) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:417) at hudson.model.Run.run(Run.java:1176) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:123) Caused by: hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:408) at hudson.remoting.Request.call(Request.java:105) at hudson.remoting.Channel.call(Channel.java:547) at hudson.FilePath.act(FilePath.java:672) ... 13 more [locks-and-latches] Releasing all the locks [locks-and-latches] All the locks released FATAL: channel is already closed hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:408) at hudson.remoting.Request.call(Request.java:105) at hudson.remoting.Channel.call(Channel.java:547) at hudson.Launcher$RemoteLauncher.kill(Launcher.java:734) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:422) at hudson.model.Run.run(Run.java:1176) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:123)
The job executes a batch command and it tooks 1h 17 min to fail in Hudson 1.336.
- duplicates
-
JENKINS-1948 Intermittent slave disconnections with secondary symptoms
-
- Resolved
-
- is duplicated by
-
JENKINS-7391 "hudson.util.IOException2: Failed to join the process" during MSBuild task
-
- Resolved
-
-
JENKINS-7690 Connection to slave is dropped in the middle of a build
-
- Resolved
-
-
JENKINS-3412 For long running jobs (>2 hours) job failing with hudson.util.IOException2: Failed to join the process
-
- Closed
-
-
JENKINS-6531 FATAL: command execution failed. hudson.util.IOException2: Failed to join the process
-
- Closed
-
- is related to
-
JENKINS-8700 "Node offline during build" and problems reconnecting.
-
- Resolved
-
-
JENKINS-12235 FATAL, Unable to delete script file, IOException2, remote file operation failed, unexpected termination of channel
-
- Resolved
-
- relates to
-
JENKINS-72799 ConsoleLogFilter is not applied to all SlaveComputer logging
-
- Closed
-
[JENKINS-5073] hudson.util.IOException2: Failed to join the process - on a Windows slave
I have the same problem on other platform like ibm aix and with hudson 1.338 and with jdk 1.6
Seems to be a general bug:
FATAL: command execution failed
hudson.util.IOException2: Failed to join the process
at hudson.Proc$RemoteProc.join(Proc.java:311)
at hudson.Launcher$ProcStarter.join(Launcher.java:275)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:83)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:583)
at hudson.model.Build$RunnerImpl.build(Build.java:165)
at hudson.model.Build$RunnerImpl.doRun(Build.java:133)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:417)
at hudson.model.Run.run(Run.java:1179)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:122)
Caused by: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: hudson.remoting.ProxyOutputStream$Flush
at hudson.remoting.Channel$1.adapt(Channel.java:580)
at hudson.remoting.Channel$1.adapt(Channel.java:575)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55)
at hudson.Proc$RemoteProc.join(Proc.java:303)
... 12 more
Caused by: java.lang.NoClassDefFoundError: hudson.remoting.ProxyOutputStream$Flush
at java.lang.J9VMInternals.verifyImpl(Native Method)
at java.lang.J9VMInternals.verify(J9VMInternals.java:72)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:134)
at hudson.remoting.RemoteOutputStream.readObject(RemoteOutputStream.java:90)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:600)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:986)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1865)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1769)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1345)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1963)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1887)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1769)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1345)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:364)
at hudson.remoting.UserRequest.deserialize(UserRequest.java:168)
at hudson.remoting.UserRequest.perform(UserRequest.java:88)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:270)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:453)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:315)
at java.util.concurrent.FutureTask.run(FutureTask.java:150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:736)
Caused by: java.lang.ClassNotFoundException: hudson.remoting.ProxyOutputStream$Flush
at java.net.URLClassLoader.findClass(URLClassLoader.java:421)
at java.lang.ClassLoader.loadClass(ClassLoader.java:643)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:346)
at java.lang.ClassLoader.loadClass(ClassLoader.java:609)
... 27 more
Notifying upstream projects of job completion
Finished: FAILURE
Sorry it's not exactly the same issue.
Your connection problem was linked to:
Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.io.EOFException
And my connection problem was linked to:
Caused by: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: hudson.remoting.ProxyOutputStream$Flush
@erwan_q
Your issue looks very similar to JENKINS-5141. I'd suggest commenting there.
Thanks for your comment.
The Hudson-5141 is more oriented on "ClassNotFoundException: hudson.model.job" i think.
My problem is more on the hudson.remoting.ProxyOutputStream$Flush
I created a new bug JENKINS-5149 on this specific issue.
I'm trying to find a solution to have a stable env but no success
We have encountered this same exact issue with Windows XP master/slave (Hudson 1.342). It occurs about 2 hours into build process, but not consistently. Usually builds of the same project do succeed without any problems. Here's a snippet of our failures (omitting some of the lines, since it is exactly the same as in the original post).
FATAL: command execution failed
hudson.util.IOException2: Failed to join the process
at hudson.Proc$RemoteProc.join(Proc.java:312)
at hudson.Launcher$ProcStarter.join(Launcher.java:275)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:83)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:582)
at hudson.model.Build$RunnerImpl.build(Build.java:165)
at hudson.model.Build$RunnerImpl.doRun(Build.java:132)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416)
at hudson.model.Run.run(Run.java:1198)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:122)
Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.io.EOFException
...
FATAL: channel is already closed
hudson.remoting.ChannelClosedException: channel is already closed
at hudson.remoting.Channel.send(Channel.java:412)
at hudson.remoting.Request.call(Request.java:105)
at hudson.remoting.Channel.call(Channel.java:551)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:735)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:421)
at hudson.model.Run.run(Run.java:1198)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:122)
@decicco: Thanks for the tip about lack of output, will see if adding some output (if possible) would help.
We've seen similar join related issues on our Hudson deployment (master/slaves all running 64-bit Windows 2003 Server, slaves as Windows Service).
Usually happens on a job running for a little longer than an hour. Latest Hudson version I can report this happening is 1.356.
Output:
FATAL: command execution failed.
hudson.util.IOException2: Failed to join the process
at hudson.Proc$RemoteProc.join(Proc.java:312)
at hudson.Launcher$ProcStarter.join(Launcher.java:278)
at hudson.tasks.Ant.perform(Ant.java:212)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:600)
at hudson.model.Build$RunnerImpl.build(Build.java:174)
at hudson.model.Build$RunnerImpl.doRun(Build.java:138)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416)
at hudson.model.Run.run(Run.java:1244)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:122)
Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
at hudson.remoting.Request$1.get(Request.java:218)
at hudson.remoting.Request$1.get(Request.java:172)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55)
at hudson.Proc$RemoteProc.join(Proc.java:304)
... 11 more
Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
at hudson.remoting.Request.abort(Request.java:257)
at hudson.remoting.Channel.terminate(Channel.java:598)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:880)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:856)
FATAL: channel is already closed
hudson.remoting.ChannelClosedException: channel is already closed
at hudson.remoting.Channel.send(Channel.java:412)
at hudson.remoting.Request.call(Request.java:105)
at hudson.remoting.Channel.call(Channel.java:551)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:738)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:421)
at hudson.model.Run.run(Run.java:1244)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:122)
Is this a problem with Hudson or the Join plugin?
Same problem on ubuntu linux.
hudson.util.IOException2: Failed to join the process at hudson.Proc$RemoteProc.join(Proc.java:312) at hudson.Launcher$ProcStarter.join(Launcher.java:278) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:83) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:584) at hudson.model.Build$RunnerImpl.build(Build.java:174) at hudson.model.Build$RunnerImpl.doRun(Build.java:138) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416) at hudson.model.Run.run(Run.java:1244) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:122) Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request$1.get(Request.java:218) at hudson.remoting.Request$1.get(Request.java:172) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55) at hudson.Proc$RemoteProc.join(Proc.java:304) ... 12 more Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request.abort(Request.java:257) at hudson.remoting.Channel.terminate(Channel.java:598) at hudson.remoting.Channel$ReaderThread.run(Channel.java:880) Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Channel$ReaderThread.run(Channel.java:862) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) at hudson.remoting.Channel$ReaderThread.run(Channel.java:856) FATAL: Unable to delete script file /tmp/hudson4172490882779075596.sh hudson.util.IOException2: remote file operation failed: /tmp/hudson4172490882779075596.sh at hudson.remoting.Channel@4b4e563e:hudson_slave at hudson.FilePath.act(FilePath.java:743) at hudson.FilePath.act(FilePath.java:729) at hudson.FilePath.delete(FilePath.java:984) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:93) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:584) at hudson.model.Build$RunnerImpl.build(Build.java:174) at hudson.model.Build$RunnerImpl.doRun(Build.java:138) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416) at hudson.model.Run.run(Run.java:1244) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:122) Caused by: hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:412) at hudson.remoting.Request.call(Request.java:105) at hudson.remoting.Channel.call(Channel.java:551) at hudson.FilePath.act(FilePath.java:736) ... 13 more FATAL: channel is already closed hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:412) at hudson.remoting.Request.call(Request.java:105) at hudson.remoting.Channel.call(Channel.java:551) at hudson.Launcher$RemoteLauncher.kill(Launcher.java:738) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:421) at hudson.model.Run.run(Run.java:1244) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:122)
I occasionally get a similar error when Hudson checks out a large svn project. It happens at night when system backups are also happening on the network, so the check out may take a very long time.
I am running Hudson v.1.360. Master runs on Windows XP, slaves are Linux. Failure seen on jobs running on master or slave.
ERROR: Failed to check out https://<snip>
org.tmatesoft.svn.core.SVNException: svn: SSL peer shut down incorrectly
svn: REPORT request failed on '/subversion/zodiac/!svn/vcc/default'
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:103)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:87)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:616)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:273)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:261)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:266)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1263)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:820)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.update(SVNUpdateClient.java:558)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:934)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:742)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:660)
at hudson.FilePath.act(FilePath.java:753)
at hudson.FilePath.act(FilePath.java:735)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:653)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:601)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1044)
at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
at hudson.model.Run.run(Run.java:1241)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:124)
Caused by: javax.net.ssl.SSLException: SSL peer shut down incorrectly
at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at org.tmatesoft.svn.core.internal.util.ChunkedInputStream.read(ChunkedInputStream.java:70)
at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
at sun.nio.cs.StreamDecoder.read(Unknown Source)
at java.io.InputStreamReader.read(Unknown Source)
at org.tmatesoft.svn.core.internal.io.dav.http.XMLReader.read(XMLReader.java:39)
at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipChar(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.readData(HTTPConnection.java:726)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.readData(HTTPConnection.java:691)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPRequest.dispatch(HTTPRequest.java:216)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:345)
... 20 more
Email was triggered for: Failure
Sending email for trigger: Failure
I am seeing the same on my Windows XP master-slave setup. I am running latest Hudson ver. 1.363
I am using the close-workspace-scm plugin to copy my workspace from master to slave(150).
Started by user anonymous
Building remotely on 150
FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request.call(Request.java:137)
at hudson.remoting.Channel.call(Channel.java:555)
at hudson.FilePath.act(FilePath.java:742)
at hudson.FilePath.act(FilePath.java:735)
at hudson.FilePath.unzip(FilePath.java:415)
at hudson.FileSystemProvisioner$Default$WorkspaceSnapshotImpl.restoreTo(FileSystemProvisioner.java:227)
at hudson.plugins.cloneworkspace.CloneWorkspaceSCM$Snapshot.restoreTo(CloneWorkspaceSCM.java:344)
at hudson.plugins.cloneworkspace.CloneWorkspaceSCM.checkout(CloneWorkspaceSCM.java:126)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1044)
at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
at hudson.model.Run.run(Run.java:1253)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:127)
Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request.abort(Request.java:257)
at hudson.remoting.Channel.terminate(Channel.java:602)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:893)
Caused by: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Channel$ReaderThread.run(Channel.java:875)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:869)
I think I'm seeing the same thing, at least the same thing as levante reported.
Hudson 1.362 in Tomcat 5.5 on Server 2003 32-bit, JRE 1.6. Slave is Server 2008 R2, 64-bit, JRE 1.6, connected via hudsonslave service. When a lot of jobs come in at once the hudsonslave service crashes.
hudson.util.IOException2: Failed to join the process
at hudson.Proc$RemoteProc.join(Proc.java:312)
at hudson.Launcher$ProcStarter.join(Launcher.java:280)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:83)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:601)
at hudson.model.Build$RunnerImpl.build(Build.java:174)
at hudson.model.Build$RunnerImpl.doRun(Build.java:138)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416)
at hudson.model.Run.run(Run.java:1253)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:124)
Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
at hudson.remoting.Request$1.get(Request.java:218)
at hudson.remoting.Request$1.get(Request.java:172)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55)
at hudson.Proc$RemoteProc.join(Proc.java:304)
... 12 more
Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
at hudson.remoting.Request.abort(Request.java:257)
at hudson.remoting.Channel.terminate(Channel.java:602)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:893)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:869)
FATAL: Unable to delete script file C:\Windows\TEMP\hudson750406825412804255.bat
hudson.util.IOException2: remote file operation failed: C:\Windows\TEMP\hudson750406825412804255.bat at hudson.remoting.Channel@1db6bbe:weyoun1
at hudson.FilePath.act(FilePath.java:749)
at hudson.FilePath.act(FilePath.java:735)
at hudson.FilePath.delete(FilePath.java:990)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:93)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:601)
at hudson.model.Build$RunnerImpl.build(Build.java:174)
at hudson.model.Build$RunnerImpl.doRun(Build.java:138)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416)
at hudson.model.Run.run(Run.java:1253)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:124)
Caused by: hudson.remoting.ChannelClosedException: channel is already closed
at hudson.remoting.Channel.send(Channel.java:412)
at hudson.remoting.Request.call(Request.java:105)
at hudson.remoting.Channel.call(Channel.java:555)
at hudson.FilePath.act(FilePath.java:742)
... 13 more
FATAL: channel is already closed
hudson.remoting.ChannelClosedException: channel is already closed
at hudson.remoting.Channel.send(Channel.java:412)
at hudson.remoting.Request.call(Request.java:105)
at hudson.remoting.Channel.call(Channel.java:555)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:744)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:421)
at hudson.model.Run.run(Run.java:1253)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:124)
most of the Apache Hadoop projects started hitting the same issues.
FATAL: Unable to delete script file /tmp/hudson6048046841607430352.sh
hudson.util.IOException2: remote file operation failed: /tmp/hudson6048046841607430352.sh at hudson.remoting.Channel@af5ea0: (Ubuntu)
at hudson.FilePath.act(FilePath.java:743)
at hudson.FilePath.act(FilePath.java:729)
at hudson.FilePath.delete(FilePath.java:984)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:93)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
I click on the log like on the node and here is the info that I got.
[07/13/10 16:55:37] Launching slave agent
$ ssh <hostname>/homes/hudson/tools/java/latest/bin/java -Xmx512m -jar /grid/0/hudson/hudson-slave/slave.jar
<===[HUDSON REMOTING CAPACITY]===>���channel started
Slave.jar version: 1.364
This is a Unix slave
Copied maven-agent.jar
Copied maven-interceptor.jar
Copied maven2.1-interceptor.jar
Slave successfully connected and online
Jul 13, 2010 4:55:43 PM com.youdevise.hudson.slavestatus.SlaveListener call
INFO: Slave-status listener starting
Jul 13, 2010 4:55:44 PM com.youdevise.hudson.slavestatus.SocketHTTPListener waitForConnection
INFO: Slave-status listener ready on port 3141
I notice the same (or a similar) problem on my Debian/Linux host and Hudson 1.357. I have the remote job running on the same host started with the Launch slave via execution of command on the master:
nice -n 5 /usr/local/lib/java/jdk1.6.0/bin/java -Xdebug -Xrunjdwp:transport=dt_socket,address=8003,server=y,suspend=n -Xms16M -jar /home/argonightly/tools/WEB-INF/slave.jar
(The same problems occur also without the -Xdebug -Xrunjdwp... arguments). The purpose of starting in this way is to get some long-running jobs to run with a lower nice to allow other jobs to finish quicker.
For me this has only happened when there are two jobs running on the slave. Both jobs fail with the same error message at the same time. Then the slave(s) is restarted and new jobs are allocated on the slave. I have previously noticed that the jobs on the old slave is still running so I assume that it is just the connection between the master and the slave that is the problem.
Today I have noticed this twice and managed to capture the output from the master hudson. The first time with two jobs running on one slave and probably none on the other, the second time with just one of the slaves without any jobs running.
Here is the first time with the output from the master. Notice that both slaves (updateshow and nightly) are crashing, taken down, or failing at the same time. You can also notice that I run the master hudson with the -verbosegc argument.
[GC 127674K->117585K(160932K), 9.4880730 secs] Jul 22, 2010 6:35:43 AM hudson.model.Run run INFO: update-mirror #30764 main build action completed: SUCCESS [GC 127697K->117678K(160932K), 6.2626260 secs] Jul 22, 2010 6:44:51 AM hudson.remoting.Channel$ReaderThread run SEVERE: I/O error in channel updateshow java.io.IOException: Unexpected termination of the channel at hudson.remoting.Channel$ReaderThread.run(Channel.java:862) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputSt ream.java:2552) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) at hudson.remoting.Channel$ReaderThread.run(Channel.java:856) Jul 22, 2010 6:44:57 AM hudson.remoting.Channel$ReaderThread run SEVERE: I/O error in channel nightly java.io.IOException: Unexpected termination of the channel at hudson.remoting.Channel$ReaderThread.run(Channel.java:862) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputSt ream.java:2552) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) at hudson.remoting.Channel$ReaderThread.run(Channel.java:856) Jul 22, 2010 6:44:57 AM hudson.remoting.Request$2 run SEVERE: Failed to send back a reply java.io.IOException: Broken pipe at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:260) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65 ) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) at java.io.ObjectOutputStream$BlockDataOutputStream.flush(ObjectOutputSt ream.java:1784) at java.io.ObjectOutputStream.flush(ObjectOutputStream.java:691) at hudson.remoting.Channel.send(Channel.java:418) at hudson.remoting.Request$2.run(Request.java:282) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:44 1) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec utor.java:885) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:907) at java.lang.Thread.run(Thread.java:619) Jul 22, 2010 6:45:03 AM hudson.slaves.SlaveComputer tryReconnect INFO: Attempting to reconnect nightly Jul 22, 2010 6:45:04 AM hudson.slaves.SlaveComputer tryReconnect INFO: Attempting to reconnect updateshow [GC 127787K->118134K(160932K), 6.6658860 secs]
This is the backtrace from the log of one of the jobs on the nightly slave:
FATAL: command execution failed hudson.util.IOException2: Failed to join the process at hudson.Proc$RemoteProc.join(Proc.java:312) at hudson.Launcher$ProcStarter.join(Launcher.java:278) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:83) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:600) at hudson.model.Build$RunnerImpl.build(Build.java:174) at hudson.model.Build$RunnerImpl.doRun(Build.java:138) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416) at hudson.model.Run.run(Run.java:1241) at hudson.matrix.MatrixRun.run(MatrixRun.java:130) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:122) Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request$1.get(Request.java:218) at hudson.remoting.Request$1.get(Request.java:172) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55) at hudson.Proc$RemoteProc.join(Proc.java:304) ... 12 more Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request.abort(Request.java:257) at hudson.remoting.Channel.terminate(Channel.java:598) at hudson.remoting.Channel$ReaderThread.run(Channel.java:880) Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Channel$ReaderThread.run(Channel.java:862) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2552) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) at hudson.remoting.Channel$ReaderThread.run(Channel.java:856) FATAL: Unable to delete script file /tmp/hudson18796.sh hudson.util.IOException2: remote file operation failed: /tmp/hudson18796.sh at hudson.remoting.Channel@2e646f:nightly at hudson.FilePath.act(FilePath.java:752) at hudson.FilePath.act(FilePath.java:738) at hudson.FilePath.delete(FilePath.java:993) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:93) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:600) at hudson.model.Build$RunnerImpl.build(Build.java:174) at hudson.model.Build$RunnerImpl.doRun(Build.java:138) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416) at hudson.model.Run.run(Run.java:1241) at hudson.matrix.MatrixRun.run(MatrixRun.java:130) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:122) Caused by: hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:412) at hudson.remoting.Request.call(Request.java:105) at hudson.remoting.Channel.call(Channel.java:551) at hudson.FilePath.act(FilePath.java:745) ... 13 more FATAL: channel is already closed hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:412) at hudson.remoting.Request.call(Request.java:105) at hudson.remoting.Channel.call(Channel.java:551) at hudson.Launcher$RemoteLauncher.kill(Launcher.java:738) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:421) at hudson.model.Run.run(Run.java:1241) at hudson.matrix.MatrixRun.run(MatrixRun.java:130) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:122)
The second time today only the updateshow job got this problem and I cannot find any failed jobs. The jobs on updateshow are a lot shorter (around 5m). Here is the output from the master process:
[GC 151557K->148710K(160932K), 4.0233760 secs] Jul 22, 2010 12:23:09 PM hudson.model.Run run INFO: update-mirror #30799 main build action completed: SUCCESS [GC 158822K->150464K(161356K), 1.0420050 secs] [Full GC[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14 720] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14411] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14660] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14581] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14459] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14746] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14554] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14357] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor13929] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14437] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14747] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14707] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14743] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14384] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14754] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14496] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14383] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14508] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14664] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14346] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14468] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14397] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14712] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14549] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14419] [Unloading class Script1$_run_closure1] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14671] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14620] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14365] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14328] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14429] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14415] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14408] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor13652] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14593] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14456] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14425] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14609] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14636] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14574] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14626] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14539] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14441] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14568] [Unloading class Script1] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14639] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14694] [Unloading class sun.reflect.GeneratedConstructorAccessor334] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14667] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14577] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14550] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14650] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14351] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14633] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14744] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14364] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14725] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14356] [Unloading class sun.reflect.GeneratedMethodAccessor1350] [Unloading class Script1] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor13782] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor13807] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14463] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14360] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14576] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14560] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14515] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14462] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14349] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14489] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14647] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14579] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14538] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14524] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14482] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14445] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14398] [Unloading class Script1] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14402] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14358] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14651] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14627] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14421] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14400] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14391] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14432] [Unloading class Script1$_run_closure1] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14635] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor13571] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14481] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14572] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14734] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14336] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14440] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14467] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14732] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14713] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14629] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14377] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14715] [Unloading class sun.reflect.GeneratedConstructorAccessor333] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14632] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14724] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14418] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14617] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14361] [Unloading class sun.reflect.GeneratedConstructorAccessor322] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14460] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14392] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14599] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14691] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14368] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14613] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14705] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14587] [Unloading class Script1$_run_closure1] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14511] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor13936] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14396] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14696] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14634] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14548] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14700] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14388] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14504] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14491] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14678] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14645] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor13930] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14733] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14592] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14444] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14433] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14665] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14551] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14479] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14730] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14615] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14589] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14355] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14472] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14708] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14662] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14683] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14507] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14521] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14470] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14393] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14422] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14420] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14674] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14637] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14476] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14716] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14477] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14352] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor13805] [Unloading class sun.reflect.GeneratedMethodAccessor1352] [Unloading class sun.reflect.GeneratedConstructorAccessor327] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14516] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14403] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14719] [Unloading class sun.reflect.GeneratedConstructorAccessor329] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14416] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14466] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14692] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14711] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14565] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14685] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14698] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14677] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14575] [Unloading class sun.reflect.GeneratedConstructorAccessor332] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14333] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14571] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14412] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14717] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14679] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14654] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14638] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14652] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14604] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14457] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14534] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor13586] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14329] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14409] [Unloading class sun.reflect.GeneratedMethodAccessor1355] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14449] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14544] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14424] [Unloading class sun.reflect.GeneratedConstructorAccessor324] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14385] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14703] [Unloading class sun.reflect.GeneratedConstructorAccessor331] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14343] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14621] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14531] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14379] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14542] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14526] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14518] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14506] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14666] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14564] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14656] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14390] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14598] [Unloading class sun.reflect.GeneratedConstructorAccessor325] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14567] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14453] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14488] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14625] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14528] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14502] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14558] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14741] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14668] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14586] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14438] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14670] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14541] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14375] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14641] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14646] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14596] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14728] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14367] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14439] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14597] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14736] [Unloading class Script1] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14451] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor13538] [Unloading class Script1$_run_closure1] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14490] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14382] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14584] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14347] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14628] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14704] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14749] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor13937] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14378] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14423] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14658] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14386] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14399] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14697] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14350] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14553] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14622] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14461] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14417] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14503] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14687] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14381] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14731] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14591] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14748] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14475] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14448] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14695] [Unloading class Script1$_run_closure1] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14362] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14370] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14458] [Unloading class Script1$_run_closure1] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14363] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14582] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14753] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14334] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14478] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14610] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14492] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14480] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14513] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14473] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14607] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor13935] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14330] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14585] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor13580] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14659] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14573] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14689] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14557] [Unloading class Script1$_run_closure1] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14332] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14532] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14739] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14588] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor13583] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14680] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14605] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14616] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14623] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14369] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14341] [Unloading class sun.reflect.GeneratedMethodAccessor1351] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14738] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14514] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14436] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14465] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14642] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14740] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14723] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14742] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14520] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14500] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14517] [Unloading class sun.reflect.GeneratedConstructorAccessor321] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14452] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14331] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14547] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14373] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14701] [Unloading class Script1] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14663] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14600] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14606] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14603] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14522] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14340] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14485] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor13581] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14527] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14619] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14474] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14413] [Unloading class Script1$_run_closure1] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14611] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14630] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14454] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14446] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14536] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14512] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14750] [Unloading class Script1$_run_closure1] [Unloading class sun.reflect.GeneratedMethodAccessor1294] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14543] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14669] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor13582] [Unloading class Script1$_run_closure1] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14435] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14583] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14407] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14608] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14430] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14427] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14612] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14426] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14394] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14344] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14745] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14519] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14471] [Unloading class sun.reflect.GeneratedConstructorAccessor326] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14672] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14682] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14673] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14359] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14580] [Unloading class sun.reflect.GeneratedConstructorAccessor328] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14643] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14497] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14410] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14722] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14533] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14540] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14354] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14469] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14631] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14537] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14595] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14688] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14661] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14601] [Unloading class Script1] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14702] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14487] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14690] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14376] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14374] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor13579] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14594] [Unloading class Script1] [Unloading class Script1$_run_closure1] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor13938] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14483] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14484] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14525] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14751] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14556] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14464] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14428] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14505] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14501] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14486] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14348] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14405] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14561] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14726] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14676] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14675] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14530] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14335] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14590] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14699] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14578] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14566] [Unloading class Script1] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14395] [Unloading class sun.reflect.GeneratedMethodAccessor1354] [Unloading class sun.reflect.GeneratedMethodAccessor1290] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14510] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14404] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14727] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14406] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14618] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14657] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14494] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor13534] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14681] [Unloading class sun.reflect.GeneratedMethodAccessor1298] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14755] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14366] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14570] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14545] [Unloading class Script1] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14353] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14546] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14431] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14710] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14718] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14380] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14752] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14648] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14552] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14499] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14706] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14714] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14342] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14640] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14602] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14559] [Unloading class sun.reflect.GeneratedConstructorAccessor330] [Unloading class sun.reflect.GeneratedConstructorAccessor302] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14569] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14414] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14447] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14372] [Unloading class Script1] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14614] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14523] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor13769] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14493] [Unloading class Script1] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14684] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14371] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14649] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14562] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14450] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14389] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14729] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14535] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14655] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14624] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14509] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14737] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14735] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor13806] [Unloading class Script1$_run_closure1] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14721] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14442] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14653] [Unloading class sun.reflect.GeneratedConstructorAccessor323] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14345] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14555] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14455] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14434] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14387] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14529] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14498] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14495] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14563] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14443] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14693] [Unloading class Script1] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14401] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14644] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor13931] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14686] [Unloading class sun.reflect.GeneratedMethodAccessor1356] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14709] 150464K->51729K(161356K), 225.1859240 secs] Jul 22, 2010 12:26:56 PM hudson.remoting.Channel$ReaderThread run SEVERE: I/O error in channel updateshow java.io.IOException: Unexpected termination of the channel at hudson.remoting.Channel$ReaderThread.run(Channel.java:862) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputSt ream.java:2552) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) at hudson.remoting.Channel$ReaderThread.run(Channel.java:856) [GC 61841K->52274K(160932K), 0.0367730 secs] [GC 62386K->52083K(160932K), 0.0317730 secs] Jul 22, 2010 12:28:38 PM hudson.slaves.SlaveComputer tryReconnect INFO: Attempting to reconnect updateshow Jul 22, 2010 12:28:47 PM hudson.slaves.CommandLauncher launch INFO: slave agent launched for updateshow [GC 62195K->52411K(160932K), 0.0747990 secs]
In this second case I guess that the following has happened: The FullGC on the master takes 225 seconds (almost four minutes). During that time the master cannot answer the ping. The slave gives up on the master and aborts. Eventually the master detects this and restarts the slave. This happens after the job started at 12:20 is completed and before the job starting at 12:30 so no jobs are affected. To this, the simple fix would be to increase the TIME_OUT of the Ping (end of the hudson.remoting.PingThread class).
In the first case I don't understand what has happened.
This is a complicated problem to investigate because it just happens occasionally and the log is overwritten.
To allow us to investigate this further, let me suggest that the master-slave function is improved with a function to move or save the old log every time a slave starts so the log of the slave that crashed, failed, or terminated is preserved.
It happened again (at 04:59 AM this morning). I guess it was the long Garbage Collect (221 seconds) that made the client close because of the ping not being answered.
This time I had moved the slave-nightly.log file to another name. Unluckily, nothing was written to the file. I had hoped to see the "No Ping response received" log error. I guess the slave-nightly.log file is written by the master on the master and since the connection is already closed by the client at this point, the master cannot write anything.
I still think the most frustrating part of this is that the logs of the slave cannot be accessed. I will see if I understand how to start the slaves logging to a file.
linustolke: "To this, the simple fix would be to increase the TIME_OUT of the Ping (end of the hudson.remoting.PingThread class)."
This particular issue was fixed in 1.367.
Please see this discussion for more info:
http://hudson.361315.n4.nabble.com/Regarding-the-IOException2-ChannelClosedException-many-of-us-has-been-seeing-lately-td2280769.html
are you sure that it was fixed in 1.367? i run this version and i still see this problem. additionally, this fix was not yet mentioned in the release notes...
i'll wait for the next release...
Yes.
Change 32664 was made 09 July. However the changelog was not updated (I have noticed that many fixes actually goes in to the trunk without updating the changelog).
http://fisheye.jenkins-ci.org/changelog/Hudson/trunk/hudson?cs=32664
I think that this "fix" may only be a workaround instead of a real fix.
I experienced this problem in 1.363. Yesterday I upgrade to 1.368 and executed the same job in question - a matrix job with about 40 configurations, each of which took from 3 to 4 hours. Most of them used to fail in 1.363, but after switching to 1.368, only one of them failed. So I think the problem still exists.
yes, i think there is still something broken in this regard. i am running 1.368 on a linux64 master with two 32bit virtualbox slave VMs (win7/linux) and another linux64 node with three 64bit virtualbox slave VMs (win7/linux/osx).
below is the typical stacktrace i see regularly (2 out of 3 jobs) on the windows slaves (32bit and 64bit win7, not on linux or osx). it does not seem to be related to the host machines. the exceptions almost always occurs on longish linker, nsis or file transfer (archiving artifacts) operations.
update: the slave log does not contain anything special when it happens.
i appreciate any help to debug this.
kind regards,
simon
FATAL: command execution failed
hudson.util.IOException2: Failed to join the process
at hudson.Proc$RemoteProc.join(Proc.java:355)
at hudson.Launcher$ProcStarter.join(Launcher.java:280)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:83)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:601)
at hudson.model.Build$RunnerImpl.build(Build.java:174)
at hudson.model.Build$RunnerImpl.doRun(Build.java:138)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416)
at hudson.model.Run.run(Run.java:1257)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:129)
Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request$1.get(Request.java:218)
at hudson.remoting.Request$1.get(Request.java:172)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55)
at hudson.Proc$RemoteProc.join(Proc.java:347)
... 12 more
Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request.abort(Request.java:257)
at hudson.remoting.Channel.terminate(Channel.java:602)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:893)
Caused by: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Channel$ReaderThread.run(Channel.java:875)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:869)
FATAL: Unable to delete script file C:\Windows\TEMP\hudson1632955036060142154.bat
hudson.util.IOException2: remote file operation failed: C:\Windows\TEMP\hudson1632955036060142154.bat at hudson.remoting.Channel@72945e31:sidian_win7_32
at hudson.FilePath.act(FilePath.java:749)
at hudson.FilePath.act(FilePath.java:735)
at hudson.FilePath.delete(FilePath.java:990)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:93)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:601)
at hudson.model.Build$RunnerImpl.build(Build.java:174)
at hudson.model.Build$RunnerImpl.doRun(Build.java:138)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416)
at hudson.model.Run.run(Run.java:1257)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:129)
Caused by: hudson.remoting.ChannelClosedException: channel is already closed
at hudson.remoting.Channel.send(Channel.java:412)
at hudson.remoting.Request.call(Request.java:105)
at hudson.remoting.Channel.call(Channel.java:555)
at hudson.FilePath.act(FilePath.java:742)
... 13 more
FATAL: channel is already closed
hudson.remoting.ChannelClosedException: channel is already closed
at hudson.remoting.Channel.send(Channel.java:412)
at hudson.remoting.Request.call(Request.java:105)
at hudson.remoting.Channel.call(Channel.java:555)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:744)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:421)
at hudson.model.Run.run(Run.java:1257)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:129)
Code changed in hudson
User: : kohsuke
Path:
trunk/hudson/main/remoting/src/main/java/hudson/remoting/Channel.java
trunk/hudson/main/remoting/src/main/java/hudson/remoting/ChannelClosedException.java
http://jenkins-ci.org/commit/33537
Log:
[JENKINS-5073 JENKINS-3412] improved the error diagnostics on ChannelClosedException by having it report who/how the connection was closed.
i disabled indexing (windows search service) on my win7 vmware slaves - still the same behavior
We see this on a Linux slave, running on the same machine as the master.
Seems to happen more frequently under high load.
Stacktrace is slightly differen:
FATAL: command execution failed hudson.util.IOException2: Failed to join the process at hudson.Proc$RemoteProc.join(Proc.java:312) at hudson.Launcher$ProcStarter.join(Launcher.java:278) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:83) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:601) at hudson.model.Build$RunnerImpl.build(Build.java:174) at hudson.model.Build$RunnerImpl.doRun(Build.java:138) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416) at hudson.model.Run.run(Run.java:1241) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:124) Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request$1.get(Request.java:218) at hudson.remoting.Request$1.get(Request.java:172) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55) at hudson.Proc$RemoteProc.join(Proc.java:304) ... 12 more Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request.abort(Request.java:257) at hudson.remoting.Channel.terminate(Channel.java:598) at hudson.remoting.Channel$ReaderThread.run(Channel.java:880) Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Channel$ReaderThread.run(Channel.java:862) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:2700) at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1648) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1323) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) at hudson.remoting.Channel$ReaderThread.run(Channel.java:856) FATAL: Unable to delete script file /tmp/hudson700909144823232476.sh hudson.util.IOException2: remote file operation failed: /tmp/hudson700909144823232476.sh at hudson.remoting.Channel@7f5a75a2:hudson_slave at hudson.FilePath.act(FilePath.java:749) at hudson.FilePath.act(FilePath.java:735) at hudson.FilePath.delete(FilePath.java:990) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:93) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:601) at hudson.model.Build$RunnerImpl.build(Build.java:174) at hudson.model.Build$RunnerImpl.doRun(Build.java:138) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416) at hudson.model.Run.run(Run.java:1241) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:124) Caused by: hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:412) at hudson.remoting.Request.call(Request.java:105) at hudson.remoting.Channel.call(Channel.java:551) at hudson.FilePath.act(FilePath.java:742) ... 13 more FATAL: channel is already closed hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:412) at hudson.remoting.Request.call(Request.java:105) at hudson.remoting.Channel.call(Channel.java:551) at hudson.Launcher$RemoteLauncher.kill(Launcher.java:738) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:421) at hudson.model.Run.run(Run.java:1241) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:124)
a random guess: could this issue be related to high slave response times? on my 5 slaves i observe 5-40ms for the linux32/linux64/osx slaves and 200-600ms for the two win32/win64 slaves. the former three slaves run rock solid, on the two windows slaves i regularly (75%) get "failed to join the process".
note that our linux32/win32 and linux64/win64 slave pairs run on the same virtualbox hosts (linux64) each. i guess, this rules out the hosts as problem sources. so is this a problem of hudson slave communication in combination with virtualbox!? i'm going to try different vbox network setups, maybe the vbox nat is the offender...
i am veeery motivated to finally resolve this one
Also having the same problem. Running Hudson 1.378 on CentOS 5.5 connecting to a CentOS 5.2 slave via ssh.
Hi, we noticed that this happens just in a job that execute a job with a duration of around 2hours that has a step with no output in a long time (don't know exactly how long)
-deccico
This is when it fails for me, too. The error occurs while running a script that takes quite a while and does not produce output. However, if I make the script print to the console frequently as it's running, this error doesn't pop up.
Is it because the ssh connection seems idle and isn't using a "keep-alive"?
FATAL: command execution failed hudson.util.IOException2: Failed to join the process at hudson.Proc$RemoteProc.join(Proc.java:355) at hudson.Launcher$ProcStarter.join(Launcher.java:280) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:82) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:601) at hudson.model.Build$RunnerImpl.build(Build.java:174) at hudson.model.Build$RunnerImpl.doRun(Build.java:138) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416) at hudson.model.Run.run(Run.java:1280) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:137) Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request$1.get(Request.java:218) at hudson.remoting.Request$1.get(Request.java:172) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55) at hudson.Proc$RemoteProc.join(Proc.java:347) ... 12 more Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request.abort(Request.java:257) at hudson.remoting.Channel.terminate(Channel.java:658) at hudson.remoting.Channel$ReaderThread.run(Channel.java:950) Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Channel$ReaderThread.run(Channel.java:931) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) at hudson.remoting.Channel$ReaderThread.run(Channel.java:925) FATAL: Unable to delete script file /tmp/hudson8197175197320902892.sh hudson.util.IOException2: remote file operation failed: /tmp/hudson8197175197320902892.sh at hudson.remoting.Channel@7069b861:slave at hudson.FilePath.act(FilePath.java:749) at hudson.FilePath.act(FilePath.java:735) at hudson.FilePath.delete(FilePath.java:990) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:601) at hudson.model.Build$RunnerImpl.build(Build.java:174) at hudson.model.Build$RunnerImpl.doRun(Build.java:138) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416) at hudson.model.Run.run(Run.java:1280) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:137) Caused by: hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:451) at hudson.remoting.Request.call(Request.java:105) at hudson.remoting.Channel.call(Channel.java:607) at hudson.FilePath.act(FilePath.java:742) ... 13 more Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Channel$ReaderThread.run(Channel.java:931) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) at hudson.remoting.Channel$ReaderThread.run(Channel.java:925) FATAL: channel is already closed hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:451) at hudson.remoting.Request.call(Request.java:105) at hudson.remoting.Channel.call(Channel.java:607) at hudson.Launcher$RemoteLauncher.kill(Launcher.java:744) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:421) at hudson.model.Run.run(Run.java:1280) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:137) Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Channel$ReaderThread.run(Channel.java:931) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) at hudson.remoting.Channel$ReaderThread.run(Channel.java:925)
More recent version of Hudson, like the one Mike_L used, shows the root cause of the "hudson.remoting.ChannelClosedException: channel is already closed". So those who are experiencing this problem, please report your stack trace so that we can collect more data.
So far the error seems to indicate that the network connection between the master and the slave is terminated abruptly.
I guess I need to first improve Hudson so that it captures the failure from the slave process before we can understand the cause of the problem.
For the more info request. The issue started plaguing us about 2 weeks ago. Running 1.375 on 32-bit Win 2003 Server, with an additional slave on the same server and one on a 64-bit Windows 2008 server. Tomcat 6.0.20, java 1.6.0_06. Java 1.6.0_20 on the 64-bit 2008 server. Slave.jars are in sync with server version.
I have the Master + 3 slaves running as services on Windows. This particular issue strikes every 2-4 days, usually during the middle of actual build work where logs are actively being written. Recovery requires restaring the master and all slaves. The builds we do are large C and C++ builds, and they take a long time between 3-5 hours.
The unstableness is dwindling my dev team's confidence in Hudson's usefulness. Hopefully this issue can be addressed soon.
FATAL: command execution failed hudson.util.IOException2: Failed to join the process at hudson.Proc$RemoteProc.join(Proc.java:355) at hudson.Launcher$ProcStarter.join(Launcher.java:280) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:82) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:601) at hudson.model.Build$RunnerImpl.build(Build.java:174) at hudson.model.Build$RunnerImpl.doRun(Build.java:138) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416) at hudson.model.Run.run(Run.java:1273) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:129) Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request$1.get(Request.java:218) at hudson.remoting.Request$1.get(Request.java:172) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55) at hudson.Proc$RemoteProc.join(Proc.java:347) ... 12 more Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request.abort(Request.java:257) at hudson.remoting.Channel.terminate(Channel.java:608) at hudson.remoting.Channel$ReaderThread.run(Channel.java:899) Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Channel$ReaderThread.run(Channel.java:881) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Channel$ReaderThread.run(Channel.java:875) FATAL: Unable to delete script file C:\Users\lgbuild\AppData\Local\Temp\hudson8508007568280050491.bat hudson.util.IOException2: remote file operation failed: C:\Users\lgbuild\AppData\Local\Temp\hudson8508007568280050491.bat at hudson.remoting.Channel@367b19:Schwartz at hudson.FilePath.act(FilePath.java:749) at hudson.FilePath.act(FilePath.java:735) at hudson.FilePath.delete(FilePath.java:990) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:601) at hudson.model.Build$RunnerImpl.build(Build.java:174) at hudson.model.Build$RunnerImpl.doRun(Build.java:138) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416) at hudson.model.Run.run(Run.java:1273) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:129) Caused by: hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:414) at hudson.remoting.Request.call(Request.java:105) at hudson.remoting.Channel.call(Channel.java:557) at hudson.FilePath.act(FilePath.java:742) ... 13 more Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Channel$ReaderThread.run(Channel.java:881) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Channel$ReaderThread.run(Channel.java:875) FATAL: channel is already closed hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:414) at hudson.remoting.Request.call(Request.java:105) at hudson.remoting.Channel.call(Channel.java:557) at hudson.Launcher$RemoteLauncher.kill(Launcher.java:744) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:421) at hudson.model.Run.run(Run.java:1273) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:129) Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Channel$ReaderThread.run(Channel.java:881) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Channel$ReaderThread.run(Channel.java:875)
When executing a free-style job in a Windows agent, this error appers:
2> WINVER not defined. Defaulting to 0x0502 (Windows Server 2003)
11>CDllCall.cpp
FATAL: command execution failed
hudson.util.IOException2: Failed to join the process
at hudson.Proc$RemoteProc.join(Proc.java:355)
at hudson.Launcher$ProcStarter.join(Launcher.java:280)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:82)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:601)
at hudson.model.Build$RunnerImpl.build(Build.java:174)
at hudson.model.Build$RunnerImpl.doRun(Build.java:138)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416)
at hudson.model.Run.run(Run.java:1280)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:139)
Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
at hudson.remoting.Request$1.get(Request.java:218)
at hudson.remoting.Request$1.get(Request.java:172)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55)
at hudson.Proc$RemoteProc.join(Proc.java:347)
... 12 more
Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
at hudson.remoting.Request.abort(Request.java:257)
at hudson.remoting.Channel.terminate(Channel.java:681)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:972)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:185)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2265)
at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2558)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2568)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:948)
FATAL: Unable to delete script file C:\WINDOWS\TEMP\hudson7211599763862507406.bat
hudson.util.IOException2: remote file operation failed: C:\WINDOWS\TEMP\hudson7211599763862507406.bat at hudson.remoting.Channel@60ac71:FXBuild
at hudson.FilePath.act(FilePath.java:749)
at hudson.FilePath.act(FilePath.java:735)
at hudson.FilePath.delete(FilePath.java:990)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:601)
at hudson.model.Build$RunnerImpl.build(Build.java:174)
at hudson.model.Build$RunnerImpl.doRun(Build.java:138)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416)
at hudson.model.Run.run(Run.java:1280)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:139)
Caused by: hudson.remoting.ChannelClosedException: channel is already closed
at hudson.remoting.Channel.send(Channel.java:467)
at hudson.remoting.Request.call(Request.java:105)
at hudson.remoting.Channel.call(Channel.java:630)
at hudson.FilePath.act(FilePath.java:742)
... 13 more
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:185)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2265)
at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2558)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2568)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:948)
FATAL: channel is already closed
hudson.remoting.ChannelClosedException: channel is already closed
at hudson.remoting.Channel.send(Channel.java:467)
at hudson.remoting.Request.call(Request.java:105)
at hudson.remoting.Channel.call(Channel.java:630)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:744)
at hudson.plugins.cygpath.CygpathLauncherDecorator$1.kill(CygpathLauncherDecorator.java:76)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:421)
at hudson.model.Run.run(Run.java:1280)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:139)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:185)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2265)
at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2558)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2568)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:948)
This may be the same as http://issues.jenkins-ci.org/browse/JENKINS-7690
Which has a stack trace mentioning:
FATAL: channel is already closed
hudson.remoting.ChannelClosedException: channel is already closed
Hello! We also frequently experience such problem . We tried hudson 1.386 and 1.387, jdk-6.0.19 and jdk-6.0.22, Master host: Linux Gentoo, Slave host: Linux Ubuntu.
Here is log:
FATAL: command execution failed hudson.util.IOException2: Failed to join the process at hudson.Proc$RemoteProc.join(Proc.java:355) at hudson.Launcher$ProcStarter.join(Launcher.java:280) at com.g2one.hudson.grails.GrailsBuilder.perform(GrailsBuilder.java:184) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:603) at hudson.model.Build$RunnerImpl.build(Build.java:172) at hudson.model.Build$RunnerImpl.doRun(Build.java:137) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:417) at hudson.model.Run.run(Run.java:1325) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:139) Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid type code: 55 at hudson.remoting.Request$1.get(Request.java:218) at hudson.remoting.Request$1.get(Request.java:172) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55) at hudson.Proc$RemoteProc.join(Proc.java:347) ... 11 more Caused by: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid type code: 55 at hudson.remoting.Request.abort(Request.java:257) at hudson.remoting.Channel.terminate(Channel.java:680) at hudson.remoting.Channel$ReaderThread.run(Channel.java:971) Caused by: java.io.StreamCorruptedException: invalid type code: 55 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1355) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350) at hudson.remoting.Channel$ReaderThread.run(Channel.java:947) FATAL: channel is already closed hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:466) at hudson.remoting.Request.call(Request.java:105) at hudson.remoting.Channel.call(Channel.java:629) at hudson.Launcher$RemoteLauncher.kill(Launcher.java:744) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:422) at hudson.model.Run.run(Run.java:1325) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:139) Caused by: java.io.StreamCorruptedException: invalid type code: 55 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1355) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350) at hudson.remoting.Channel$ReaderThread.run(Channel.java:947)
Same problem.
Configuration:
Master: Red Hat Enterprise Linux Server release 5.5 (Tikanga), x86-64
Slave: Debian Lenny, x86-64
Freestyle job running on slave fails randomly when trying to launch a Jython script from a shell. Error message is:
FATAL: command execution failed hudson.util.IOException2: Failed to join the process at hudson.Proc$RemoteProc.join(Proc.java:355) at hudson.Launcher$ProcStarter.join(Launcher.java:280) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:82) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.plugins.m2extrasteps.M2ExtraStepsWrapper.executeBuildSteps(M2ExtraStepsWrapper.java:166) at hudson.plugins.m2extrasteps.M2ExtraStepsWrapper.access$200(M2ExtraStepsWrapper.java:43) at hudson.plugins.m2extrasteps.M2ExtraStepsWrapper$1.tearDown(M2ExtraStepsWrapper.java:137) at hudson.maven.MavenModuleSetBuild$RunnerImpl.doRun(MavenModuleSetBuild.java:494) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416) at hudson.model.Run.run(Run.java:1280) at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:293) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:139) Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request$1.get(Request.java:218) at hudson.remoting.Request$1.get(Request.java:172) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55) at hudson.Proc$RemoteProc.join(Proc.java:347) ... 12 more Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request.abort(Request.java:257) at hudson.remoting.Channel.terminate(Channel.java:681) at hudson.remoting.Channel$ReaderThread.run(Channel.java:972) Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Channel$ReaderThread.run(Channel.java:954) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) at hudson.remoting.Channel$ReaderThread.run(Channel.java:948) FATAL: Unable to delete script file /tmp/hudson1272208775265435208.sh hudson.util.IOException2: remote file operation failed: /tmp/hudson1272208775265435208.sh at hudson.remoting.Channel@35267641:vmlnxbld1buch-axhudson-x64 at hudson.FilePath.act(FilePath.java:749) at hudson.FilePath.act(FilePath.java:735) at hudson.FilePath.delete(FilePath.java:990) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.plugins.m2extrasteps.M2ExtraStepsWrapper.executeBuildSteps(M2ExtraStepsWrapper.java:166) at hudson.plugins.m2extrasteps.M2ExtraStepsWrapper.access$200(M2ExtraStepsWrapper.java:43) at hudson.plugins.m2extrasteps.M2ExtraStepsWrapper$1.tearDown(M2ExtraStepsWrapper.java:137) at hudson.maven.MavenModuleSetBuild$RunnerImpl.doRun(MavenModuleSetBuild.java:494) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416) at hudson.model.Run.run(Run.java:1280) at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:293) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:139) Caused by: hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:467) at hudson.remoting.Request.call(Request.java:105) at hudson.remoting.Channel.call(Channel.java:630) at hudson.FilePath.act(FilePath.java:742) ... 13 more Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Channel$ReaderThread.run(Channel.java:954) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) at hudson.remoting.Channel$ReaderThread.run(Channel.java:948) FATAL: channel is already closed hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:467) at hudson.remoting.Request.call(Request.java:105) at hudson.remoting.Channel.call(Channel.java:630) at hudson.Launcher$RemoteLauncher.kill(Launcher.java:744) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:421) at hudson.model.Run.run(Run.java:1280) at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:293) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:139) Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Channel$ReaderThread.run(Channel.java:954) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) at hudson.remoting.Channel$ReaderThread.run(Channel.java:948)
Does not happen all the time, seems random so far, but is highly annoying, and blocks releases sometimes.
Code changed in hudson
User: Kohsuke Kawaguchi
Path:
changelog.html
core/src/main/java/hudson/slaves/CommandLauncher.java
core/src/main/java/hudson/slaves/ComputerLauncher.java
core/src/main/java/hudson/slaves/SlaveComputer.java
core/src/main/java/hudson/util/io/ReopenableFileOutputStream.java
https://github.com/hudson/hudson/commit/0ddb0c5cca757a4a34e0c162bbca6504f5cec25f
Log:
JENKINS-5073 Hudson was failing to record the connection termination problem in slave logs.
This change might reveal additional information about why the connection is disrupted.
Code changed in hudson
User: Kohsuke Kawaguchi
Path:
core/src/main/java/hudson/model/AbstractBuild.java
core/src/main/java/hudson/model/TaskListener.java
https://github.com/hudson/hudson/commit/c6ffab117de489e7a728d47dab5c9f5a3b92e19c
Log:
JENKINS-5073 if the build fails because the node went offline, point the user to the slave log where the details are.
Code changed in hudson
User: Kohsuke Kawaguchi
Path:
core/src/main/java/hudson/model/AbstractBuild.java
https://github.com/hudson/hudson/commit/5f29d615a8d8536c59111937df2e36941fb12d18
Log:
JENKINS-5073 point the user to the slave log if the build failed because the slave went offline during the build.
Code changed in hudson
User: Kohsuke Kawaguchi
Path:
changelog.html
https://github.com/hudson/hudson/commit/7fbb893fd6831fa8c19b3be98df5655e1281e0da
Log:
noting these changes that are related to JENKINS-5073
Integrated in hudson_main_trunk #450
JENKINS-5073 Hudson was failing to record the connection termination problem in slave logs.
JENKINS-5073 if the build fails because the node went offline, point the user to the slave log where the details are.
JENKINS-5073 point the user to the slave log if the build failed because the slave went offline during the build.
noting these changes that are related to JENKINS-5073
Kohsuke Kawaguchi :
Files :
- core/src/main/java/hudson/slaves/CommandLauncher.java
- core/src/main/java/hudson/slaves/ComputerLauncher.java
- core/src/main/java/hudson/slaves/SlaveComputer.java
- core/src/main/java/hudson/util/io/ReopenableFileOutputStream.java
- changelog.html
Kohsuke Kawaguchi :
Files :
- core/src/main/java/hudson/model/TaskListener.java
- core/src/main/java/hudson/model/AbstractBuild.java
Kohsuke Kawaguchi :
Files :
- core/src/main/java/hudson/model/AbstractBuild.java
Kohsuke Kawaguchi :
Files :
- changelog.html
We upgraded to Hudson 1.395 (Master host: Linux Gentoo, Slave host: Linux Ubuntu.), so now our logs are:
FATAL: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid type code: 55 hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid type code: 55 at hudson.remoting.Request.call(Request.java:137) at hudson.remoting.Channel.call(Channel.java:629) at hudson.FilePath.act(FilePath.java:745) at hudson.FilePath.act(FilePath.java:738) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:683) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:632) at hudson.model.AbstractProject.checkout(AbstractProject.java:1171) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:499) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:415) at hudson.model.Run.run(Run.java:1362) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:145) Caused by: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid type code: 55 at hudson.remoting.Request.abort(Request.java:257) at hudson.remoting.Channel.terminate(Channel.java:680) at hudson.remoting.Channel$ReaderThread.run(Channel.java:971) Caused by: java.io.StreamCorruptedException: invalid type code: 55 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1355) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350) at hudson.remoting.Channel$ReaderThread.run(Channel.java:947)
I was looking at the Proc class, and there seems to be a bug in the StdinCopyThread that could be cause the associated LocalProc to close its stream too early.
See :
@Override public void run() { try { try { byte[] buf = new byte[8192]; int len; while ((len = in.read(buf)) > 0) { out.write(buf, 0, len); out.flush(); } } finally { in.close(); out.close(); } } catch (IOException e) { // TODO: what to do? } }
If the read returns 0, e.g. because of a buffering delay etc, then the while loop exists before we are done.
Same for StreamCopyThread. There may be other occurences in the code. I will search for them.
Not sure how these classes are used but it could definitively affect a long build, explaining the lack of reproducibility.
I created JENKINS-8686 to track and deal with this issue. We will see if that solves this problem or not. Otherwise we might have to ask for some logs (there are a few well placed log calls in that part of the code).
[Edit 2: We only had the issues with JNLP-Slaves. Instead, we now installed Cygwin+OpenSSH on the Windows slave and connect using the SSH Slaves plugin. This works fine.]
We see similar problems.
Master: Ubuntu x64, 2.6.32-29-server
Slave: Windows 7 SP1 x64, VM running in VirtualBox 4.0.4, Java 1.6.0_24-x64 (also tried 32bit JVM, same problem).
[Edit: i disabled antivirus, Windows Defender, UAC, I run the jenkins slave as administrator. These don't help.]
We also have a Windows XP 32 bit VM, similar configuration, and there it never happens. On the Win7 slave it happens at every build.
Here are 5 build, the stack traces are exactly identical 1 2 3 4 5
It looks like it does not happen at random points, but at certain points in the build.
FATAL: Unable to delete script file C:\Windows\TEMP\hudson182664989132439196.bat hudson.util.IOException2: remote file operation failed: C:\Windows\TEMP\hudson182664989132439196.bat at hudson.remoting.Channel@42728b7:win7 at hudson.FilePath.act(FilePath.java:753) at hudson.FilePath.act(FilePath.java:739) at hudson.FilePath.delete(FilePath.java:994) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:649) at hudson.model.Build$RunnerImpl.build(Build.java:177) at hudson.model.Build$RunnerImpl.doRun(Build.java:139) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:423) at hudson.model.Run.run(Run.java:1362) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:145) Caused by: hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:466) at hudson.remoting.Request.call(Request.java:105) at hudson.remoting.Channel.call(Channel.java:630) at hudson.FilePath.act(FilePath.java:746) ... 13 more Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Channel$ReaderThread.run(Channel.java:985) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2553) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1296) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350) at hudson.remoting.Channel$ReaderThread.run(Channel.java:979) FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request.call(Request.java:137) at hudson.remoting.Channel.call(Channel.java:630) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158) at $Proxy26.join(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:850) at hudson.Launcher$ProcStarter.join(Launcher.java:336) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:82) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:649) at hudson.model.Build$RunnerImpl.build(Build.java:177) at hudson.model.Build$RunnerImpl.doRun(Build.java:139) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:423) at hudson.model.Run.run(Run.java:1362) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:145) Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request.abort(Request.java:257) at hudson.remoting.Channel.terminate(Channel.java:681) at hudson.remoting.Channel$ReaderThread.run(Channel.java:1003) Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Channel$ReaderThread.run(Channel.java:985) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2553) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1296) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350) at hudson.remoting.Channel$ReaderThread.run(Channel.java:979)
Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
core/src/main/java/hudson/tasks/CommandInterpreter.java
http://jenkins-ci.org/commit/jenkins/1b22559581abf87344c57ac0dcf21d1612f2c80b
Log:
JENKINS-5073
Suppressed a confusing 2nd error message.
Compare: https://github.com/jenkinsci/jenkins/compare/786c1d899367...1b22559581ab
Integrated in jenkins_main_trunk #2887
JENKINS-5073 (Revision 1b22559581abf87344c57ac0dcf21d1612f2c80b)
Result = SUCCESS
kohsuke : 1b22559581abf87344c57ac0dcf21d1612f2c80b
Files :
- core/src/main/java/hudson/tasks/CommandInterpreter.java
Hi, we noticed that this happens just in a job that execute a job with a duration of around 2hours that has a step with no output in a long time (don't know exactly how long)
thanks