Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-24084

Slave file leak build is interrupted during Copy Artifacts build step

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • core, remoting
    • Windows Server 2012 R2
      Java 1.7.0_55-b13
      Jenkins is 1.574 (slave 2.43)

      Slave processes are having file locks to files in the workspace hanging around after builds are interrupted during copy artifacts step.

      I used http://file-leak-detector.kohsuke.org/ to get a stack trace of the lock:

      File leak detector output
      #12 C:\jenkins\workspace\Browser Tests (QA)\BrowserType\InternetExplorer32\OS\windows\TestCategory\JourneyTest02\Bin\Release\Test\Quartz.pdb by thread:pool-1-thread-1178 for channel on Thu Aug 21 07:28:35 CDT 2014
	at java.io.FileOutputStream.<init>(Unknown Source)
	at java.io.FileOutputStream.<init>(Unknown Source)
	at hudson.FilePath$32.call(FilePath.java:1715)
	at hudson.FilePath$32.call(FilePath.java:1710)
	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:328)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at hudson.remoting.Engine$1$1.run(Engine.java:63)
	at java.lang.Thread.run(Unknown Source)

      The build log for the build that corresponds to the file lock timestamp shows:

      Build console log output
      ERROR: Failed to copy artifacts from Build Empower with filter: Bin/*/Test/**/*.*,Build/**/*.*,Bin/*.runsettings
hudson.util.IOException2: Failed to copy C:\jenkins\jobs\Build Empower\builds\2014-08-20_13-38-06\archive\Bin\Release\Test\Quartz.pdb to C:\jenkins\workspace\Browser Tests (QA)\BrowserType\InternetExplorer32\OS\windows\TestCategory\JourneyTest02\Bin\Release\Test\Quartz.pdb
	at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:107)
	at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyAll(FingerprintingCopyMethod.java:68)
	at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:396)
	at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:327)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:772)
	at hudson.model.Build$BuildExecution.build(Build.java:199)
	at hudson.model.Build$BuildExecution.doRun(Build.java:160)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:535)
	at hudson.model.Run.execute(Run.java:1732)
	at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:234)
Caused by: java.io.InterruptedIOException
	at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.writeBlock(NioChannelHub.java:212)
	at hudson.remoting.AbstractByteArrayCommandTransport.write(AbstractByteArrayCommandTransport.java:83)
	at hudson.remoting.Channel.send(Channel.java:545)
	at hudson.remoting.ProxyOutputStream._write(ProxyOutputStream.java:163)
	at hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:109)
	at hudson.remoting.RemoteOutputStream.write(RemoteOutputStream.java:110)
	at java.security.DigestOutputStream.write(Unknown Source)
	at hudson.remoting.RemoteOutputStream.write(RemoteOutputStream.java:110)
	at hudson.Util.copyStream(Util.java:461)
	at hudson.FilePath$37.invoke(FilePath.java:1845)
	at hudson.FilePath$37.invoke(FilePath.java:1839)
	at hudson.FilePath.act(FilePath.java:922)
	at hudson.FilePath.act(FilePath.java:895)
	at hudson.FilePath.copyTo(FilePath.java:1839)
	at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:79)
	... 12 more
Caused by: java.lang.InterruptedException
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:503)
	at org.jenkinsci.remoting.nio.FifoBuffer.write(FifoBuffer.java:336)
	at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.writeBlock(NioChannelHub.java:207)
	... 26 more
Build step 'Copy artifacts from another project' marked build as failure

      I'm guessing there was a network hiccup of some sort that caused the slave to momentarily lose connection with master during the copy artifacts build step.

      This is easily reproducible by simply aborting the build during this step.

          [JENKINS-24084] Slave file leak build is interrupted during Copy Artifacts build step

          mrobinet created issue -

          Daniel Beck added a comment -

          How about the version of Jenkins and the slave.jar?

          Daniel Beck added a comment - How about the version of Jenkins and the slave.jar?

          mrobinet added a comment -

          Jenkins is 1.574 and slave is 2.43

          mrobinet added a comment - Jenkins is 1.574 and slave is 2.43

          Daniel Beck added a comment -

          Something opens this file for writing and doesn't close the OutputStream afterwards it seems.

          Which parts of the job config do that? MSBuild plugin? Some test plugin?

          Daniel Beck added a comment - Something opens this file for writing and doesn't close the OutputStream afterwards it seems. Which parts of the job config do that? MSBuild plugin? Some test plugin?

          mrobinet added a comment -

          Jenkins launches MSBuild and MSBuild launches vstest.console.exe to run our acceptance tests. Neither MSBuild or vstest.console.exe is running still. openfiles (http://technet.microsoft.com/en-us/library/cc732490.aspx) shows only the java.exe process holding onto it.

          mrobinet added a comment - Jenkins launches MSBuild and MSBuild launches vstest.console.exe to run our acceptance tests. Neither MSBuild or vstest.console.exe is running still. openfiles ( http://technet.microsoft.com/en-us/library/cc732490.aspx ) shows only the java.exe process holding onto it.

          Daniel Beck added a comment -

          I assume the XML file is some kind of test log/result? Do you parse these in some way to show nice test results stats in Jenkins?

          Daniel Beck added a comment - I assume the XML file is some kind of test log/result? Do you parse these in some way to show nice test results stats in Jenkins?

          mrobinet added a comment - - edited

          http://stackoverflow.com/questions/4015018/whats-the-purpose-of-xml-files-next-to-the-dll

          Just documentation for the libraries that are actually being used. It gets fingerprinted by Jenkins, so potentially that requires a lock? Seems like it wouldn't though... I don't think MSBuild or the test runner touch it though. It runs successfully if not present.

          mrobinet added a comment - - edited http://stackoverflow.com/questions/4015018/whats-the-purpose-of-xml-files-next-to-the-dll Just documentation for the libraries that are actually being used. It gets fingerprinted by Jenkins, so potentially that requires a lock? Seems like it wouldn't though... I don't think MSBuild or the test runner touch it though. It runs successfully if not present.
          mrobinet made changes -
          Component/s New: copyartifact [ 15692 ]
          Description Original: Random slave processes are having file locks to files in the workspace hanging around after builds complete.

          I used http://file-leak-detector.kohsuke.org/ to get a stack trace of the lock:
          #20 C:\jenkins\workspace\Browser Tests (Smoke)\BrowserType\Firefox\OS\windows\TestCategory\BrowserRegistration\Bin\Release\Test\System.Net.Http.Extensions.xml by thread:pool-1-thread-728 for channel on Wed Jul 30 13:23:30 CDT 2014
          at java.io.FileOutputStream.<init>(Unknown Source)
          at java.io.FileOutputStream.<init>(Unknown Source)
          at hudson.FilePath$32.call(FilePath.java:1715)
          at hudson.FilePath$32.call(FilePath.java:1710)
          at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:328)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at hudson.remoting.Engine$1$1.run(Engine.java:63)

          I can reproduce it within a few days usually, so let me know if more information is needed.           		New: Slave processes are having file locks to files in the workspace hanging around after builds are interrupted during copy artifacts step.

          I used http://file-leak-detector.kohsuke.org/ to get a stack trace of the lock:
          #12 C:\jenkins\workspace\Browser Tests (QA)\BrowserType\InternetExplorer32\OS\windows\TestCategory\JourneyTest02\Bin\Release\Test\Quartz.pdb by thread:pool-1-thread-1178 for channel on Thu Aug 21 07:28:35 CDT 2014
          at java.io.FileOutputStream.<init>(Unknown Source)
          at java.io.FileOutputStream.<init>(Unknown Source)
          at hudson.FilePath$32.call(FilePath.java:1715)
          at hudson.FilePath$32.call(FilePath.java:1710)
          at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:328)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at hudson.remoting.Engine$1$1.run(Engine.java:63)
          at java.lang.Thread.run(Unknown Source)

          The build log shows:
          ERROR: Failed to copy artifacts from Build Empower with filter: Bin/*/Test/**/*.*,Build/**/*.*,Bin/*.runsettings
          hudson.util.IOException2: Failed to copy C:\jenkins\jobs\Build Empower\builds\2014-08-20_13-38-06\archive\Bin\Release\Test\Quartz.pdb to C:\jenkins\workspace\Browser Tests (QA)\BrowserType\InternetExplorer32\OS\windows\TestCategory\JourneyTest02\Bin\Release\Test\Quartz.pdb
          at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:107)
          at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyAll(FingerprintingCopyMethod.java:68)
          at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:396)
          at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:327)
          at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
          at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:772)
          at hudson.model.Build$BuildExecution.build(Build.java:199)
          at hudson.model.Build$BuildExecution.doRun(Build.java:160)
          at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:535)
          at hudson.model.Run.execute(Run.java:1732)
          at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
          at hudson.model.ResourceController.execute(ResourceController.java:88)
          at hudson.model.Executor.run(Executor.java:234)
          Caused by: java.io.InterruptedIOException
          at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.writeBlock(NioChannelHub.java:212)
          at hudson.remoting.AbstractByteArrayCommandTransport.write(AbstractByteArrayCommandTransport.java:83)
          at hudson.remoting.Channel.send(Channel.java:545)
          at hudson.remoting.ProxyOutputStream._write(ProxyOutputStream.java:163)
          at hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:109)
          at hudson.remoting.RemoteOutputStream.write(RemoteOutputStream.java:110)
          at java.security.DigestOutputStream.write(Unknown Source)
          at hudson.remoting.RemoteOutputStream.write(RemoteOutputStream.java:110)
          at hudson.Util.copyStream(Util.java:461)
          at hudson.FilePath$37.invoke(FilePath.java:1845)
          at hudson.FilePath$37.invoke(FilePath.java:1839)
          at hudson.FilePath.act(FilePath.java:922)
          at hudson.FilePath.act(FilePath.java:895)
          at hudson.FilePath.copyTo(FilePath.java:1839)
          at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:79)
          ... 12 more
          Caused by: java.lang.InterruptedException
          at java.lang.Object.wait(Native Method)
          at java.lang.Object.wait(Object.java:503)
          at org.jenkinsci.remoting.nio.FifoBuffer.write(FifoBuffer.java:336)
          at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.writeBlock(NioChannelHub.java:207)
          ... 26 more
          Build step 'Copy artifacts from another project' marked build as failure

          I'm guessing there was a network hiccup of some sort.

          This is easily reproducible by simply aborting the build during the Copy Artifacts step.
          Summary Original: Slave file leak New: Slave file leak when interrupted builds
          mrobinet made changes -
          Summary Original: Slave file leak when interrupted builds New: Slave file leak build is interrupted during Copy Artifacts build step
          mrobinet made changes -
          Description Original: Slave processes are having file locks to files in the workspace hanging around after builds are interrupted during copy artifacts step.

          I used http://file-leak-detector.kohsuke.org/ to get a stack trace of the lock:
          #12 C:\jenkins\workspace\Browser Tests (QA)\BrowserType\InternetExplorer32\OS\windows\TestCategory\JourneyTest02\Bin\Release\Test\Quartz.pdb by thread:pool-1-thread-1178 for channel on Thu Aug 21 07:28:35 CDT 2014
          at java.io.FileOutputStream.<init>(Unknown Source)
          at java.io.FileOutputStream.<init>(Unknown Source)
          at hudson.FilePath$32.call(FilePath.java:1715)
          at hudson.FilePath$32.call(FilePath.java:1710)
          at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:328)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at hudson.remoting.Engine$1$1.run(Engine.java:63)
          at java.lang.Thread.run(Unknown Source)

          The build log shows:
          ERROR: Failed to copy artifacts from Build Empower with filter: Bin/*/Test/**/*.*,Build/**/*.*,Bin/*.runsettings
          hudson.util.IOException2: Failed to copy C:\jenkins\jobs\Build Empower\builds\2014-08-20_13-38-06\archive\Bin\Release\Test\Quartz.pdb to C:\jenkins\workspace\Browser Tests (QA)\BrowserType\InternetExplorer32\OS\windows\TestCategory\JourneyTest02\Bin\Release\Test\Quartz.pdb
          at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:107)
          at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyAll(FingerprintingCopyMethod.java:68)
          at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:396)
          at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:327)
          at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
          at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:772)
          at hudson.model.Build$BuildExecution.build(Build.java:199)
          at hudson.model.Build$BuildExecution.doRun(Build.java:160)
          at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:535)
          at hudson.model.Run.execute(Run.java:1732)
          at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
          at hudson.model.ResourceController.execute(ResourceController.java:88)
          at hudson.model.Executor.run(Executor.java:234)
          Caused by: java.io.InterruptedIOException
          at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.writeBlock(NioChannelHub.java:212)
          at hudson.remoting.AbstractByteArrayCommandTransport.write(AbstractByteArrayCommandTransport.java:83)
          at hudson.remoting.Channel.send(Channel.java:545)
          at hudson.remoting.ProxyOutputStream._write(ProxyOutputStream.java:163)
          at hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:109)
          at hudson.remoting.RemoteOutputStream.write(RemoteOutputStream.java:110)
          at java.security.DigestOutputStream.write(Unknown Source)
          at hudson.remoting.RemoteOutputStream.write(RemoteOutputStream.java:110)
          at hudson.Util.copyStream(Util.java:461)
          at hudson.FilePath$37.invoke(FilePath.java:1845)
          at hudson.FilePath$37.invoke(FilePath.java:1839)
          at hudson.FilePath.act(FilePath.java:922)
          at hudson.FilePath.act(FilePath.java:895)
          at hudson.FilePath.copyTo(FilePath.java:1839)
          at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:79)
          ... 12 more
          Caused by: java.lang.InterruptedException
          at java.lang.Object.wait(Native Method)
          at java.lang.Object.wait(Object.java:503)
          at org.jenkinsci.remoting.nio.FifoBuffer.write(FifoBuffer.java:336)
          at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.writeBlock(NioChannelHub.java:207)
          ... 26 more
          Build step 'Copy artifacts from another project' marked build as failure

          I'm guessing there was a network hiccup of some sort.

          This is easily reproducible by simply aborting the build during the Copy Artifacts step.
          		 New: Slave processes are having file locks to files in the workspace hanging around after builds are interrupted during copy artifacts step.

          I used http://file-leak-detector.kohsuke.org/ to get a stack trace of the lock:
          {code:title=File leak detector output}
          #12 C:\jenkins\workspace\Browser Tests (QA)\BrowserType\InternetExplorer32\OS\windows\TestCategory\JourneyTest02\Bin\Release\Test\Quartz.pdb by thread:pool-1-thread-1178 for channel on Thu Aug 21 07:28:35 CDT 2014
          at java.io.FileOutputStream.<init>(Unknown Source)
          at java.io.FileOutputStream.<init>(Unknown Source)
          at hudson.FilePath$32.call(FilePath.java:1715)
          at hudson.FilePath$32.call(FilePath.java:1710)
          at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:328)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at hudson.remoting.Engine$1$1.run(Engine.java:63)
          at java.lang.Thread.run(Unknown Source)
          {code}

          The build log for the build that corresponds to the file lock timestamp shows:
          {code:title=Build console log output}
          ERROR: Failed to copy artifacts from Build Empower with filter: Bin/*/Test/**/*.*,Build/**/*.*,Bin/*.runsettings
          hudson.util.IOException2: Failed to copy C:\jenkins\jobs\Build Empower\builds\2014-08-20_13-38-06\archive\Bin\Release\Test\Quartz.pdb to C:\jenkins\workspace\Browser Tests (QA)\BrowserType\InternetExplorer32\OS\windows\TestCategory\JourneyTest02\Bin\Release\Test\Quartz.pdb
          at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:107)
          at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyAll(FingerprintingCopyMethod.java:68)
          at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:396)
          at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:327)
          at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
          at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:772)
          at hudson.model.Build$BuildExecution.build(Build.java:199)
          at hudson.model.Build$BuildExecution.doRun(Build.java:160)
          at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:535)
          at hudson.model.Run.execute(Run.java:1732)
          at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
          at hudson.model.ResourceController.execute(ResourceController.java:88)
          at hudson.model.Executor.run(Executor.java:234)
          Caused by: java.io.InterruptedIOException
          at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.writeBlock(NioChannelHub.java:212)
          at hudson.remoting.AbstractByteArrayCommandTransport.write(AbstractByteArrayCommandTransport.java:83)
          at hudson.remoting.Channel.send(Channel.java:545)
          at hudson.remoting.ProxyOutputStream._write(ProxyOutputStream.java:163)
          at hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:109)
          at hudson.remoting.RemoteOutputStream.write(RemoteOutputStream.java:110)
          at java.security.DigestOutputStream.write(Unknown Source)
          at hudson.remoting.RemoteOutputStream.write(RemoteOutputStream.java:110)
          at hudson.Util.copyStream(Util.java:461)
          at hudson.FilePath$37.invoke(FilePath.java:1845)
          at hudson.FilePath$37.invoke(FilePath.java:1839)
          at hudson.FilePath.act(FilePath.java:922)
          at hudson.FilePath.act(FilePath.java:895)
          at hudson.FilePath.copyTo(FilePath.java:1839)
          at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:79)
          ... 12 more
          Caused by: java.lang.InterruptedException
          at java.lang.Object.wait(Native Method)
          at java.lang.Object.wait(Object.java:503)
          at org.jenkinsci.remoting.nio.FifoBuffer.write(FifoBuffer.java:336)
          at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.writeBlock(NioChannelHub.java:207)
          ... 26 more
          Build step 'Copy artifacts from another project' marked build as failure
          {code}

          I'm guessing there was a network hiccup of some sort that caused the slave to momentarily lose connection with master during the copy artifacts build step.

          This is easily reproducible by simply aborting the build during this step.

            Unassigned Unassigned
            mrobinet mrobinet
            Votes:
            3 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: