Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-24084

Slave file leak build is interrupted during Copy Artifacts build step

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • core, remoting
    • Windows Server 2012 R2
      Java 1.7.0_55-b13
      Jenkins is 1.574 (slave 2.43)

      Slave processes are having file locks to files in the workspace hanging around after builds are interrupted during copy artifacts step.

      I used http://file-leak-detector.kohsuke.org/ to get a stack trace of the lock:

      File leak detector output
      #12 C:\jenkins\workspace\Browser Tests (QA)\BrowserType\InternetExplorer32\OS\windows\TestCategory\JourneyTest02\Bin\Release\Test\Quartz.pdb by thread:pool-1-thread-1178 for channel on Thu Aug 21 07:28:35 CDT 2014
      	at java.io.FileOutputStream.<init>(Unknown Source)
      	at java.io.FileOutputStream.<init>(Unknown Source)
      	at hudson.FilePath$32.call(FilePath.java:1715)
      	at hudson.FilePath$32.call(FilePath.java:1710)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      	at hudson.remoting.Request$2.run(Request.java:328)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      	at java.util.concurrent.FutureTask.run(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      	at hudson.remoting.Engine$1$1.run(Engine.java:63)
      	at java.lang.Thread.run(Unknown Source)
      

      The build log for the build that corresponds to the file lock timestamp shows:

      Build console log output
      ERROR: Failed to copy artifacts from Build Empower with filter: Bin/*/Test/**/*.*,Build/**/*.*,Bin/*.runsettings
      hudson.util.IOException2: Failed to copy C:\jenkins\jobs\Build Empower\builds\2014-08-20_13-38-06\archive\Bin\Release\Test\Quartz.pdb to C:\jenkins\workspace\Browser Tests (QA)\BrowserType\InternetExplorer32\OS\windows\TestCategory\JourneyTest02\Bin\Release\Test\Quartz.pdb
      	at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:107)
      	at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyAll(FingerprintingCopyMethod.java:68)
      	at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:396)
      	at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:327)
      	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:772)
      	at hudson.model.Build$BuildExecution.build(Build.java:199)
      	at hudson.model.Build$BuildExecution.doRun(Build.java:160)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:535)
      	at hudson.model.Run.execute(Run.java:1732)
      	at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
      	at hudson.model.ResourceController.execute(ResourceController.java:88)
      	at hudson.model.Executor.run(Executor.java:234)
      Caused by: java.io.InterruptedIOException
      	at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.writeBlock(NioChannelHub.java:212)
      	at hudson.remoting.AbstractByteArrayCommandTransport.write(AbstractByteArrayCommandTransport.java:83)
      	at hudson.remoting.Channel.send(Channel.java:545)
      	at hudson.remoting.ProxyOutputStream._write(ProxyOutputStream.java:163)
      	at hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:109)
      	at hudson.remoting.RemoteOutputStream.write(RemoteOutputStream.java:110)
      	at java.security.DigestOutputStream.write(Unknown Source)
      	at hudson.remoting.RemoteOutputStream.write(RemoteOutputStream.java:110)
      	at hudson.Util.copyStream(Util.java:461)
      	at hudson.FilePath$37.invoke(FilePath.java:1845)
      	at hudson.FilePath$37.invoke(FilePath.java:1839)
      	at hudson.FilePath.act(FilePath.java:922)
      	at hudson.FilePath.act(FilePath.java:895)
      	at hudson.FilePath.copyTo(FilePath.java:1839)
      	at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:79)
      	... 12 more
      Caused by: java.lang.InterruptedException
      	at java.lang.Object.wait(Native Method)
      	at java.lang.Object.wait(Object.java:503)
      	at org.jenkinsci.remoting.nio.FifoBuffer.write(FifoBuffer.java:336)
      	at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.writeBlock(NioChannelHub.java:207)
      	... 26 more
      Build step 'Copy artifacts from another project' marked build as failure
      

      I'm guessing there was a network hiccup of some sort that caused the slave to momentarily lose connection with master during the copy artifacts build step.

      This is easily reproducible by simply aborting the build during this step.

          [JENKINS-24084] Slave file leak build is interrupted during Copy Artifacts build step

          mrobinet added a comment -

          Jenkins is 1.574 and slave is 2.43

          mrobinet added a comment - Jenkins is 1.574 and slave is 2.43

          Daniel Beck added a comment -

          Something opens this file for writing and doesn't close the OutputStream afterwards it seems.

          Which parts of the job config do that? MSBuild plugin? Some test plugin?

          Daniel Beck added a comment - Something opens this file for writing and doesn't close the OutputStream afterwards it seems. Which parts of the job config do that? MSBuild plugin? Some test plugin?

          mrobinet added a comment -

          Jenkins launches MSBuild and MSBuild launches vstest.console.exe to run our acceptance tests. Neither MSBuild or vstest.console.exe is running still. openfiles (http://technet.microsoft.com/en-us/library/cc732490.aspx) shows only the java.exe process holding onto it.

          mrobinet added a comment - Jenkins launches MSBuild and MSBuild launches vstest.console.exe to run our acceptance tests. Neither MSBuild or vstest.console.exe is running still. openfiles ( http://technet.microsoft.com/en-us/library/cc732490.aspx ) shows only the java.exe process holding onto it.

          Daniel Beck added a comment -

          I assume the XML file is some kind of test log/result? Do you parse these in some way to show nice test results stats in Jenkins?

          Daniel Beck added a comment - I assume the XML file is some kind of test log/result? Do you parse these in some way to show nice test results stats in Jenkins?

          mrobinet added a comment - - edited

          http://stackoverflow.com/questions/4015018/whats-the-purpose-of-xml-files-next-to-the-dll

          Just documentation for the libraries that are actually being used. It gets fingerprinted by Jenkins, so potentially that requires a lock? Seems like it wouldn't though... I don't think MSBuild or the test runner touch it though. It runs successfully if not present.

          mrobinet added a comment - - edited http://stackoverflow.com/questions/4015018/whats-the-purpose-of-xml-files-next-to-the-dll Just documentation for the libraries that are actually being used. It gets fingerprinted by Jenkins, so potentially that requires a lock? Seems like it wouldn't though... I don't think MSBuild or the test runner touch it though. It runs successfully if not present.

          mrobinet added a comment -

          I've updated this bug with quite a bit more information and believe I have found the root cause. I believe it is specific to the Copy Artifacts plugin.

          mrobinet added a comment - I've updated this bug with quite a bit more information and believe I have found the root cause. I believe it is specific to the Copy Artifacts plugin.

          ikedam added a comment -

          Please report the version of copyartifact.

          Anyway, it happened here:
          https://github.com/jenkinsci/copyartifact-plugin/blob/copyartifact-1.31/src/main/java/hudson/plugins/copyartifact/FingerprintingCopyMethod.java#L79

          As copyartifact closes the handle when interrupted in the finally clause, this looks the problem of Jenkins core, hudson-remoting.
          That is, even a file handle is closed in the master, it still remains open in the slave for some remoting problems.

          ikedam added a comment - Please report the version of copyartifact. Anyway, it happened here: https://github.com/jenkinsci/copyartifact-plugin/blob/copyartifact-1.31/src/main/java/hudson/plugins/copyartifact/FingerprintingCopyMethod.java#L79 As copyartifact closes the handle when interrupted in the finally clause, this looks the problem of Jenkins core, hudson-remoting. That is, even a file handle is closed in the master, it still remains open in the slave for some remoting problems.

          mrobinet added a comment -

          We are using copyartifact 1.31.

          mrobinet added a comment - We are using copyartifact 1.31.

          Simon Richter added a comment -

          Still happens with Copy Artifact 1.38.1 and Jenkins 2.46.1.

          This is a major problem for our instance, as any aborted build will leave a file that is open for writing. This file cannot be renamed or deleted, so all subsequent builds in the same workspace fail, until the slave is restarted.

          Simon Richter added a comment - Still happens with Copy Artifact 1.38.1 and Jenkins 2.46.1. This is a major problem for our instance, as any aborted build will leave a file that is open for writing. This file cannot be renamed or deleted, so all subsequent builds in the same workspace fail, until the slave is restarted.

          Oleg Nenashev added a comment -

          Unfortunately I have no capacity to work on Remoting in medium term, so I will unassign it and let others to take it. If somebody is interested to submit a pull request, I will be happy to help to get it reviewed and released.

          Oleg Nenashev added a comment - Unfortunately I have no capacity to work on Remoting in medium term, so I will unassign it and let others to take it. If somebody is interested to submit a pull request, I will be happy to help to get it reviewed and released.

            Unassigned Unassigned
            mrobinet mrobinet
            Votes:
            3 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: