Major Slave processes are having file locks to files in the workspace hanging around after builds are interrupted during copy artifacts step.
I used http://file-leak-detector.kohsuke.org/ to get a stack trace of the lock:
File leak detector output
#12 C:\jenkins\workspace\Browser Tests (QA)\BrowserType\InternetExplorer32\OS\windows\TestCategory\JourneyTest02\Bin\Release\Test\Quartz.pdb by thread:pool-1-thread-1178 for channel on Thu Aug 21 07:28:35 CDT 2014 at java.io.FileOutputStream.<init>(Unknown Source) at java.io.FileOutputStream.<init>(Unknown Source) at hudson.FilePath$32.call(FilePath.java:1715) at hudson.FilePath$32.call(FilePath.java:1710) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:328) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source)
The build log for the build that corresponds to the file lock timestamp shows:
Build console log output
ERROR: Failed to copy artifacts from Build Empower with filter: Bin/*/Test/**/*.*,Build/**/*.*,Bin/*.runsettings hudson.util.IOException2: Failed to copy C:\jenkins\jobs\Build Empower\builds\2014-08-20_13-38-06\archive\Bin\Release\Test\Quartz.pdb to C:\jenkins\workspace\Browser Tests (QA)\BrowserType\InternetExplorer32\OS\windows\TestCategory\JourneyTest02\Bin\Release\Test\Quartz.pdb at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:107) at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyAll(FingerprintingCopyMethod.java:68) at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:396) at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:327) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:772) at hudson.model.Build$BuildExecution.build(Build.java:199) at hudson.model.Build$BuildExecution.doRun(Build.java:160) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:535) at hudson.model.Run.execute(Run.java:1732) at hudson.matrix.MatrixRun.run(MatrixRun.java:146) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:234) Caused by: java.io.InterruptedIOException at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.writeBlock(NioChannelHub.java:212) at hudson.remoting.AbstractByteArrayCommandTransport.write(AbstractByteArrayCommandTransport.java:83) at hudson.remoting.Channel.send(Channel.java:545) at hudson.remoting.ProxyOutputStream._write(ProxyOutputStream.java:163) at hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:109) at hudson.remoting.RemoteOutputStream.write(RemoteOutputStream.java:110) at java.security.DigestOutputStream.write(Unknown Source) at hudson.remoting.RemoteOutputStream.write(RemoteOutputStream.java:110) at hudson.Util.copyStream(Util.java:461) at hudson.FilePath$37.invoke(FilePath.java:1845) at hudson.FilePath$37.invoke(FilePath.java:1839) at hudson.FilePath.act(FilePath.java:922) at hudson.FilePath.act(FilePath.java:895) at hudson.FilePath.copyTo(FilePath.java:1839) at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:79) ... 12 more Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at org.jenkinsci.remoting.nio.FifoBuffer.write(FifoBuffer.java:336) at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.writeBlock(NioChannelHub.java:207) ... 26 more Build step 'Copy artifacts from another project' marked build as failure
I'm guessing there was a network hiccup of some sort that caused the slave to momentarily lose connection with master during the copy artifacts build step.
This is easily reproducible by simply aborting the build during this step.
