-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
None
-
Ubuntu Server 10.04 64-bit
I don't know why this happens, but my slaves have begun to hang when they get to the Archiving Artifacts portion of my job:
Archiving artifacts ERROR: Failed to archive artifacts: dist/** hudson.util.IOException2: Failed to extract /mnt/hudsonslave/workspace/simplegeo-puppet-manifests/dist/** at hudson.FilePath.readFromTar(FilePath.java:1577) at hudson.FilePath.copyRecursiveTo(FilePath.java:1491) at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:117) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:601) at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:580) at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:558) at hudson.model.Build$RunnerImpl.post2(Build.java:157) at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:528) at hudson.model.Run.run(Run.java:1303) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:137) Caused by: java.io.IOException at hudson.remoting.FastPipedInputStream.read(FastPipedInputStream.java:173) at hudson.util.HeadBufferingStream.read(HeadBufferingStream.java:61) at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:221) at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:141) at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:92) at org.apache.tools.tar.TarBuffer.readBlock(TarBuffer.java:257) at org.apache.tools.tar.TarBuffer.readRecord(TarBuffer.java:223) at hudson.org.apache.tools.tar.TarInputStream.read(TarInputStream.java:345) at java.io.FilterInputStream.read(FilterInputStream.java:90) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1025) at org.apache.commons.io.IOUtils.copy(IOUtils.java:999) at hudson.util.IOUtils.copy(IOUtils.java:33) at hudson.FilePath.readFromTar(FilePath.java:1565) ... 12 more
I aborted this job, so I don't know if the error is related or not. The job runs fine, but it hangs during archiving. If I restart the connection to the node (in /computer/; not restarting Hudson or the node itself), jobs will build successfully again for a while.
- is related to
-
-
- Resolved
-
-
-
- Resolved
-
-
-
- Resolved
-
[JENKINS-7641] Slaves hang when archiving artifacts
ieure
created issue -
For what it's worth, we are seeing this too with 1.425 on ubuntu 10.04.03 x86_64. Strangely, it's only started being a problem in the past 2-3 weeks, but it happens quite frequently now, as often as once every other matrix build. I did a threadDump but didn't see anything particularly obvious. Would be happy to provide more information.
What we see is jobs which should normally take no longer than 20-30 minutes getting stuck like this:
Started 3 hr 10 min ago on $build_node
The console shows this at the end:
+ exit 0 Archiving artifacts (spinner)
If I cancel the job, I get this:
+ exit 0
Archiving artifacts
ERROR: Failed to archive artifacts: artifacts/**
hudson.util.IOException2: hudson.util.IOException2: Failed to extract /var/lib/jenkins/workspace/lustre-reviews/arch/x86_64/build_type/client/distro/el6/ib_stack/inkernel/artifacts/**
at hudson.FilePath.readFromTar(FilePath.java:1662)
at hudson.FilePath.copyRecursiveTo(FilePath.java:1580)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:116)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:682)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:657)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:635)
at hudson.model.Build$RunnerImpl.post2(Build.java:161)
at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:604)
at hudson.model.Run.run(Run.java:1400)
at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:175)
Caused by: java.io.IOException
at hudson.remoting.FastPipedInputStream.read(FastPipedInputStream.java:175)
at hudson.util.HeadBufferingStream.read(HeadBufferingStream.java:61)
at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:238)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:109)
at org.apache.tools.tar.TarBuffer.readBlock(TarBuffer.java:257)
at org.apache.tools.tar.TarBuffer.readRecord(TarBuffer.java:223)
at hudson.org.apache.tools.tar.TarInputStream.read(TarInputStream.java:345)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1025)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:999)
at hudson.util.IOUtils.copy(IOUtils.java:36)
at hudson.FilePath.readFromTar(FilePath.java:1654)
... 12 more
at hudson.FilePath.copyRecursiveTo(FilePath.java:1587)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:116)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:682)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:657)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:635)
at hudson.model.Build$RunnerImpl.post2(Build.java:161)
at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:604)
at hudson.model.Run.run(Run.java:1400)
at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:175)
Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.net.SocketException: Socket closed
at hudson.remoting.Request$1.get(Request.java:252)
at hudson.remoting.Request$1.get(Request.java:184)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
at hudson.FilePath.copyRecursiveTo(FilePath.java:1583)
... 11 more
Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Socket closed
at hudson.remoting.Request.abort(Request.java:273)
at hudson.remoting.Channel.terminate(Channel.java:719)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1060)
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:146)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2282)
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2295)
at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3035)
at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:2836)
at java.io.ObjectInputStream.readString(ObjectInputStream.java:1616)
at java.io.ObjectInputStream.readTypeString(ObjectInputStream.java:1418)
at java.io.ObjectStreamClass.readNonProxy(ObjectStreamClass.java:667)
at java.io.ObjectInputStream.readClassDescriptor(ObjectInputStream.java:826)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1582)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1513)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1600)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1513)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1749)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1346)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1031)
Finished: SUCCESS
The job is indeed marked as successful, and life goes on, until the next one gets stuck...
mjmac
added a comment - - edited For what it's worth, we are seeing this too with 1.425 on ubuntu 10.04.03 x86_64. Strangely, it's only started being a problem in the past 2-3 weeks, but it happens quite frequently now, as often as once every other matrix build. I did a threadDump but didn't see anything particularly obvious. Would be happy to provide more information.
What we see is jobs which should normally take no longer than 20-30 minutes getting stuck like this:
Started 3 hr 10 min ago
on $build_node
The console shows this at the end:
+ exit 0
Archiving artifacts
(spinner)
If I cancel the job, I get this:
+ exit 0
Archiving artifacts
ERROR: Failed to archive artifacts: artifacts/**
hudson.util.IOException2: hudson.util.IOException2: Failed to extract / var /lib/jenkins/workspace/lustre-reviews/arch/x86_64/build_type/client/distro/el6/ib_stack/inkernel/artifacts/**
at hudson.FilePath.readFromTar(FilePath.java:1662)
at hudson.FilePath.copyRecursiveTo(FilePath.java:1580)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:116)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:682)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:657)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:635)
at hudson.model.Build$RunnerImpl.post2(Build.java:161)
at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:604)
at hudson.model.Run.run(Run.java:1400)
at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:175)
Caused by: java.io.IOException
at hudson.remoting.FastPipedInputStream.read(FastPipedInputStream.java:175)
at hudson.util.HeadBufferingStream.read(HeadBufferingStream.java:61)
at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:238)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:109)
at org.apache.tools.tar.TarBuffer.readBlock(TarBuffer.java:257)
at org.apache.tools.tar.TarBuffer.readRecord(TarBuffer.java:223)
at hudson.org.apache.tools.tar.TarInputStream.read(TarInputStream.java:345)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1025)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:999)
at hudson.util.IOUtils.copy(IOUtils.java:36)
at hudson.FilePath.readFromTar(FilePath.java:1654)
... 12 more
at hudson.FilePath.copyRecursiveTo(FilePath.java:1587)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:116)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:682)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:657)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:635)
at hudson.model.Build$RunnerImpl.post2(Build.java:161)
at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:604)
at hudson.model.Run.run(Run.java:1400)
at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:175)
Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.net.SocketException: Socket closed
at hudson.remoting.Request$1.get(Request.java:252)
at hudson.remoting.Request$1.get(Request.java:184)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
at hudson.FilePath.copyRecursiveTo(FilePath.java:1583)
... 11 more
Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Socket closed
at hudson.remoting.Request.abort(Request.java:273)
at hudson.remoting.Channel.terminate(Channel.java:719)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1060)
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:146)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2282)
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2295)
at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3035)
at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:2836)
at java.io.ObjectInputStream.readString(ObjectInputStream.java:1616)
at java.io.ObjectInputStream.readTypeString(ObjectInputStream.java:1418)
at java.io.ObjectStreamClass.readNonProxy(ObjectStreamClass.java:667)
at java.io.ObjectInputStream.readClassDescriptor(ObjectInputStream.java:826)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1582)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1513)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1600)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1513)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1749)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1346)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1031)
Finished: SUCCESS
The job is indeed marked as successful, and life goes on, until the next one gets stuck... 
Ryan Hass
added a comment - This may be directly related to JENKINS-11586 | Link |
New:
This issue is related to |
crusius
added a comment - Happening every single time on an OSX 10.4 SSH slave, Jenkins 1.467. It was not copying the artifacts at all with 1.461, and now it simply hangs. Cancelling the slave job does not work (the job does not die), I have to cancel the main job instead. I reverted to using 1.461 in the time being – it also does not work, but at least it does not hang.
crusius
added a comment - Happening every single time on an OSX 10.4 SSH slave, Jenkins 1.467 . It was not copying the artifacts at all with 1.461 , and now it simply hangs. Cancelling the slave job does not work (the job does not die), I have to cancel the main job instead. I reverted to using 1.461 in the time being – it also does not work, but at least it does not hang. 
Oliver Bock
added a comment - Same behavior here on OS X 10.4 slave (JRE 1.5.0_19, slave.jar 2.16). As described by @crusius, although I can cancel the job if I try it twice. Then I get the following, slightly different error:
Archiving artifacts ERROR: Failed to archive artifacts: build.log, dist/* hudson.util.IOException2: java.io.IOException at hudson.remoting.FastPipedInputStream.read(FastPipedInputStream.java:175) at hudson.util.HeadBufferingStream.read(HeadBufferingStream.java:61) at java.io.FilterInputStream.read(FilterInputStream.java:107) at hudson.util.HeadBufferingStream.fillSide(HeadBufferingStream.java:83) at hudson.FilePath$TarCompression$2.extract(FilePath.java:619) at hudson.FilePath.copyRecursiveTo(FilePath.java:1771) at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:116) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:717) at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:692) at hudson.model.Build$BuildExecution.post2(Build.java:183) at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:639) at hudson.model.Run.execute(Run.java:1513) at hudson.matrix.MatrixRun.run(MatrixRun.java:146) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:236) at hudson.FilePath.copyRecursiveTo(FilePath.java:1778) at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:116) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:717) at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:692) at hudson.model.Build$BuildExecution.post2(Build.java:183) at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:639) at hudson.model.Run.execute(Run.java:1513) at hudson.matrix.MatrixRun.run(MatrixRun.java:146) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:236) Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request$1.get(Request.java:278) at hudson.remoting.Request$1.get(Request.java:210) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.FilePath.copyRecursiveTo(FilePath.java:1774) ... 10 more Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request.abort(Request.java:299) at hudson.remoting.Channel.terminate(Channel.java:719) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69) Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2570) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368) at hudson.remoting.Command.readFrom(Command.java:90) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
Oliver Bock
added a comment - Same behavior here on OS X 10.4 slave (JRE 1.5.0_19, slave.jar 2.16). As described by @crusius, although I can cancel the job if I try it twice. Then I get the following, slightly different error:
Archiving artifacts
ERROR: Failed to archive artifacts: build.log, dist/*
hudson.util.IOException2: java.io.IOException
at hudson.remoting.FastPipedInputStream.read(FastPipedInputStream.java:175)
at hudson.util.HeadBufferingStream.read(HeadBufferingStream.java:61)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at hudson.util.HeadBufferingStream.fillSide(HeadBufferingStream.java:83)
at hudson.FilePath$TarCompression$2.extract(FilePath.java:619)
at hudson.FilePath.copyRecursiveTo(FilePath.java:1771)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:116)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:717)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:692)
at hudson.model.Build$BuildExecution.post2(Build.java:183)
at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:639)
at hudson.model.Run.execute(Run.java:1513)
at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:236)
at hudson.FilePath.copyRecursiveTo(FilePath.java:1778)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:116)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:717)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:692)
at hudson.model.Build$BuildExecution.post2(Build.java:183)
at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:639)
at hudson.model.Run.execute(Run.java:1513)
at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:236)
Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request$1.get(Request.java:278)
at hudson.remoting.Request$1.get(Request.java:210)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
at hudson.FilePath.copyRecursiveTo(FilePath.java:1774)
... 10 more
Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request.abort(Request.java:299)
at hudson.remoting.Channel.terminate(Channel.java:719)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
Caused by: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2570)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
at hudson.remoting.Command.readFrom(Command.java:90)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
Same thing here (RHEL 5.5). But as i've browsed the reports it seems to originate in the slow disk.
The configuration is like that:
RHEL slave with workspace on NFS. The network (100mbit) is shared so the actual throughput is even less.
Master (RHEL 5.8, Jenkins 477) has working directory on NFS too.
The copying artifacts step hasn't finished (the artifact is some 150M). The step (before i've cancelled it) worked about 15 hours, half of that at night without user activity on the net. Therefore i conclude that things went actually wrong (probably on the slave) and it's not a matter of time.
After i've cancelled the job, i see the following lines in the log:
Archiving artifacts
ERROR: Failed to archive artifacts: *.pax.gz
hudson.util.IOException2: java.io.IOException
at hudson.remoting.FastPipedInputStream.read(FastPipedInputStream.java:175)
at hudson.util.HeadBufferingStream.read(HeadBufferingStream.java:61)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at hudson.util.HeadBufferingStream.fillSide(HeadBufferingStream.java:83)
at hudson.FilePath$TarCompression$2.extract(FilePath.java:619)
at hudson.FilePath.copyRecursiveTo(FilePath.java:1771)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:116)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:717)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:692)
at hudson.model.Build$BuildExecution.post2(Build.java:183)
at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:639)
at hudson.model.Run.execute(Run.java:1527)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:236)at hudson.FilePath.copyRecursiveTo(FilePath.java:1778)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:116)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:717)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:692)
at hudson.model.Build$BuildExecution.post2(Build.java:183)
at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:639)
at hudson.model.Run.execute(Run.java:1527)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:236)
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalAccessError: class hudson.remoting.Pipe$ConnectCommand cannot access its superclass hudson.remoting.Command
at hudson.remoting.Channel$4.adapt(Channel.java:696)
at hudson.remoting.Channel$4.adapt(Channel.java:691)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
at hudson.FilePath.copyRecursiveTo(FilePath.java:1774)
... 10 more
Caused by: java.lang.IllegalAccessError: class hudson.remoting.Pipe$ConnectCommand cannot access its superclass hudson.remoting.Command
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:632)
at java.lang.ClassLoader.defineClass(ClassLoader.java:478)
at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:152)
at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131)
at java.lang.ClassLoader.loadClass(ClassLoader.java:319)
at java.lang.ClassLoader.loadClass(ClassLoader.java:264)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:332)
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Class.java:2308)
at java.lang.Class.getDeclaredField(Class.java:1897)
at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1627)
at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:69)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:442)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:430)
at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:327)
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:564)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1600)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1513)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1749)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1346)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1963)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1887)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1770)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1346)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1963)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1887)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1770)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1346)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
at hudson.remoting.UserRequest.deserialize(UserRequest.java:182)
at hudson.remoting.UserRequest.perform(UserRequest.java:98)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Triggering a new build of SVFE-Basic #1
Finished: SUCCESS
aj2
added a comment - I'm getting this error quite regularly since upgrading from v1.447.2 to v1.479. Both the master and slave are running Ubuntu 10.04 64-bit. There are just 4 artifacts being uploaded with the biggest only around 8MB.
aj2
added a comment - I'm getting this error quite regularly since upgrading from v1.447.2 to v1.479. Both the master and slave are running Ubuntu 10.04 64-bit. There are just 4 artifacts being uploaded with the biggest only around 8MB. 
hlau
added a comment - Seeing the same issue for every build on three platforms: aix, hpux and linux-ia64. Now using the latest jenkins 1.487 and still seeing the problem.
Thread dump for a linux ia64 server currently with a hung job is as follows:
Thread Dump
Channel reader thread: channel
"Channel reader thread: channel" Id=9 Group=main RUNNABLE (in native)
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:199)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
- locked java.io.BufferedInputStream@27736da0
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2248)
at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2541)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2551)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1296)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
at hudson.remoting.Command.readFrom(Command.java:90)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
main
"main" Id=1 Group=main WAITING on hudson.remoting.Channel@7a148bd3
at java.lang.Object.wait(Native Method)
- waiting on hudson.remoting.Channel@7a148bd3
at java.lang.Object.wait(Object.java:485)
at hudson.remoting.Channel.join(Channel.java:792)
at hudson.remoting.Launcher.main(Launcher.java:428)
at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:374)
at hudson.remoting.Launcher.run(Launcher.java:214)
at hudson.remoting.Launcher.main(Launcher.java:173)
Ping thread for channel hudson.remoting.Channel@7a148bd3:channel
"Ping thread for channel hudson.remoting.Channel@7a148bd3:channel" Id=10 Group=main TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at hudson.remoting.PingThread.run(PingThread.java:86)
Pipe writer thread: channel
"Pipe writer thread: channel" Id=12 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6484439e
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6484439e
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:660)
pool-1-thread-29
"pool-1-thread-29" Id=72 Group=main RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:374)
at hudson.Functions.getThreadInfos(Functions.java:889)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:660)
Number of locked synchronizers = 1
- java.util.concurrent.locks.ReentrantLock$NonfairSync@2ac4e4b9
Finalizer
"Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@b22379c
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.ReferenceQueue$Lock@b22379c
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Reference Handler
"Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@7370e879
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.Reference$Lock@7370e879
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116):1
Signal Dispatcher
"Signal Dispatcher" Id=4 Group=system RUNNABLE
Relevant portion of jenkins log for the job is as follows (it appears the jenkins thought that the job has successfully completed):
INFO: prevent-davis.linux-ia64 #65 main build action completed: SUCCESS
Oct 22, 2012 6:48:49 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
SEVERE: I/O error in channel b-linuxia64-01
java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2553)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1296)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
at hudson.remoting.Command.readFrom(Command.java:90)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
The slave on the server that processes the job does not seem to have any problem talking to the server as I had no problem displaying its System Information:
Unix slave, version 2.17
System Properties
Name ↓
Value
file.encoding ANSI_X3.4-1968
file.encoding.pkg sun.io
file.separator /
java.awt.graphicsenv sun.awt.X11GraphicsEnvironment
java.awt.printerjob sun.print.PSPrinterJob
java.class.path slave.jar
java.class.version 50.0
java.endorsed.dirs /usr/java/jdk1.6.0_26/jre/lib/endorsed
java.ext.dirs /usr/java/jdk1.6.0_26/jre/lib/ext:/usr/java/packages/lib/ext
java.home /usr/java/jdk1.6.0_26/jre
...
hlau
added a comment - Seeing the same issue for every build on three platforms: aix, hpux and linux-ia64. Now using the latest jenkins 1.487 and still seeing the problem.
Thread dump for a linux ia64 server currently with a hung job is as follows:
Thread Dump
Channel reader thread: channel
"Channel reader thread: channel" Id=9 Group=main RUNNABLE (in native)
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:199)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
locked java.io.BufferedInputStream@27736da0
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2248)
at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2541)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2551)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1296)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
at hudson.remoting.Command.readFrom(Command.java:90)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
main
"main" Id=1 Group=main WAITING on hudson.remoting.Channel@7a148bd3
at java.lang.Object.wait(Native Method)
waiting on hudson.remoting.Channel@7a148bd3
at java.lang.Object.wait(Object.java:485)
at hudson.remoting.Channel.join(Channel.java:792)
at hudson.remoting.Launcher.main(Launcher.java:428)
at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:374)
at hudson.remoting.Launcher.run(Launcher.java:214)
at hudson.remoting.Launcher.main(Launcher.java:173)
Ping thread for channel hudson.remoting.Channel@7a148bd3:channel
"Ping thread for channel hudson.remoting.Channel@7a148bd3:channel" Id=10 Group=main TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at hudson.remoting.PingThread.run(PingThread.java:86)
Pipe writer thread: channel
"Pipe writer thread: channel" Id=12 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6484439e
at sun.misc.Unsafe.park(Native Method)
waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6484439e
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:660)
pool-1-thread-29
"pool-1-thread-29" Id=72 Group=main RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:374)
at hudson.Functions.getThreadInfos(Functions.java:889)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:660)
Number of locked synchronizers = 1
java.util.concurrent.locks.ReentrantLock$NonfairSync@2ac4e4b9
Finalizer
"Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@b22379c
at java.lang.Object.wait(Native Method)
waiting on java.lang.ref.ReferenceQueue$Lock@b22379c
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Reference Handler
"Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@7370e879
at java.lang.Object.wait(Native Method)
waiting on java.lang.ref.Reference$Lock@7370e879
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116):1
Signal Dispatcher
"Signal Dispatcher" Id=4 Group=system RUNNABLE
Relevant portion of jenkins log for the job is as follows (it appears the jenkins thought that the job has successfully completed):
INFO: prevent-davis.linux-ia64 #65 main build action completed: SUCCESS
Oct 22, 2012 6:48:49 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
SEVERE: I/O error in channel b-linuxia64-01
java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2553)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1296)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
at hudson.remoting.Command.readFrom(Command.java:90)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
The slave on the server that processes the job does not seem to have any problem talking to the server as I had no problem displaying its System Information:
Unix slave, version 2.17
System Properties
Name ↓
Value
file.encoding ANSI_X3.4-1968
file.encoding.pkg sun.io
file.separator /
java.awt.graphicsenv sun.awt.X11GraphicsEnvironment
java.awt.printerjob sun.print.PSPrinterJob
java.class.path slave.jar
java.class.version 50.0
java.endorsed.dirs /usr/java/jdk1.6.0_26/jre/lib/endorsed
java.ext.dirs /usr/java/jdk1.6.0_26/jre/lib/ext:/usr/java/packages/lib/ext
java.home /usr/java/jdk1.6.0_26/jre
...
I have the same problem where some jobs hang when archiving artifacts.
...
Archiving artifacts
I am stuck on this and it would be useful to have some advice on how to proceed for exploring this issue.
For information in this particular case, the server and the slave run on the same machine.
The machine is a RedHat 3 / i686 32 bits:
$ uname -a
Linux 2.4.21-20.EL #1 Wed Aug 18 20:58:25 EDT 2004 i686 i686 i386 GNU/Linux
I had another case some weeks ago where the slave and the server were on a different machine, and the artifacts archiving was also stalled.
The slave was this time a RedHat 4 / x86_64 64 bits:
$ uname -a
Linux 2.6.9-89.0.23.ELsmp #1 SMP Fri Mar 5 23:27:13 EST 2010 x86_64 x86_64 x86_64 GNU/Linux
I have at that time upgraded the java VM from a 32 bits to a 64 bits VM and for now we didn't suffer anymore, but I suspect that it did not solve the problem fundamentally.