Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55448

File cleaning failure, old build and plugin update

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • core

      Version 2.157 on Ubuntu with Java Wrapper to start the isntance.

      I am running into an issue where old builds are not getting cleaned up automatically or manually.

      I notice that the folder is renamed with a dot. For example, build #4 the folder is renamed from 4 to .4

      And the cleaning failed.

      Deletion of the build failedRetry deleteShow reasonjenkins.util.io.CompositeIOException: Unable to delete '/var/opt/jenkins/runner/1.0/bin/linux-x86-64/../../data/jobs/tutorial_lombok/builds/.4'. Tried 3 times (of a maximum of 3) waiting 0,1 s between attempts.
      	at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:93)
      	at hudson.Util.deleteRecursive(Util.java:267)
      	at hudson.model.Run.delete(Run.java:1581)
      	at hudson.maven.MavenModuleSetBuild.delete(MavenModuleSetBuild.java:450)
      	at hudson.model.Run.doDoDelete(Run.java:2298)
      	at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
      	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
      	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
      	at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:77)
      	at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
      	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
      	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)
      	at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:537)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
      	at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:458)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
      	at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:282)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      

       

      Moreover, when I upgraded some plugin, Jenkins failed to restart saying that the deploy plugin folder could not be deleted.

      After manually deleting the file .timestamp in the folder, it was OK.

       

       

      Seems there is an issue with file having name starting with a dot.

       

      Regards

       

      Etienne

          [JENKINS-55448] File cleaning failure, old build and plugin update

          Etienne Jouvin created issue -

          Same for us: Jenkins 2.157 running in Tomcat 8.5 under Java 1.8.0_192-b12 on Red Hat 7.3 (Maipo).

          Beginning with this version all jobs failed while trying to remove their workspace folder, which were indeed created by previous runs of that job.

          Only a downgrade to Jenkins 2.156 helped.

          Alexander Schmacks added a comment - Same for us: Jenkins 2.157 running in Tomcat 8.5 under Java 1.8.0_192-b12 on Red Hat 7.3 (Maipo). Beginning with this version all jobs failed while trying to remove their workspace folder, which were indeed created by previous runs of that job. Only a downgrade to Jenkins 2.156 helped.

          Dan Hooper added a comment -

          I can also confirm downgrading to 2.156 fixed the problem.

          Dan Hooper added a comment - I can also confirm downgrading to 2.156 fixed the problem.

          Chad Williams added a comment -

          We experience the same/related issues: This is a critical bug for us, since test result parsing via the xUnit plugin fails and successful builds are marked as failures.

          I can confirm that downgrading to 2.156 fixes the issue.

          jenkins.util.io.CompositeIOException: Unable to remove directory /foo/bar/generatedJUnitFiles/d87f0d22-c235-43cd-8272-34ce92150592/GoogleTest with directory contents: [/foo/bar/generatedJUnitFiles/d87f0d22-c235-43cd-8272-34ce92150592/GoogleTest/TEST-809dd980-e514-4260-b749-9079e250f828.xml, /foo/bar/generatedJUnitFiles/d87f0d22-c235-43cd-8272-34ce92150592/GoogleTest/TEST-2d8be771-6364-49d8-9573-c51a974a44bc.xml]
              at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:233)
              at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:191)
              at jenkins.util.io.PathRemover.forceRemoveFile(PathRemover.java:71)
              at hudson.Util.deleteFile(Util.java:255)
              at hudson.FilePath.deleteRecursive(FilePath.java:1305)
              at hudson.FilePath.access$1600(FilePath.java:213)
              at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1274)
              at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1270)
              at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
              at hudson.remoting.UserRequest.perform(UserRequest.java:212)
              at hudson.remoting.UserRequest.perform(UserRequest.java:54)
              at hudson.remoting.Request$2.run(Request.java:369)
              at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
          java.nio.file.DirectoryNotEmptyException: /foo/bar/generatedJUnitFiles/d87f0d22-c235-43cd-8272-34ce92150592/GoogleTest
              at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
              at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
              at java.nio.file.Files.deleteIfExists(Files.java:1165)
              at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:220)
              at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:191)
              at jenkins.util.io.PathRemover.forceRemoveFile(PathRemover.java:71)
              at hudson.Util.deleteFile(Util.java:255)
              at hudson.FilePath.deleteRecursive(FilePath.java:1305)
              at hudson.FilePath.access$1600(FilePath.java:213)
              at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1274)
              at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1270)
              at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
              at hudson.remoting.UserRequest.perform(UserRequest.java:212)
              at hudson.remoting.UserRequest.perform(UserRequest.java:54)
              at hudson.remoting.Request$2.run(Request.java:369)
              at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
          java.nio.file.DirectoryNotEmptyException: /foo/bar/generatedJUnitFiles/d87f0d22-c235-43cd-8272-34ce92150592/GoogleTest
              at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
              at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
              at java.nio.file.Files.deleteIfExists(Files.java:1165)
              at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:224)
              at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:191)
              at jenkins.util.io.PathRemover.forceRemoveFile(PathRemover.java:71)
              at hudson.Util.deleteFile(Util.java:255)
              at hudson.FilePath.deleteRecursive(FilePath.java:1305)
              at hudson.FilePath.access$1600(FilePath.java:213)
              at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1274)
              at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1270)
              at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
              at hudson.remoting.UserRequest.perform(UserRequest.java:212)
              at hudson.remoting.UserRequest.perform(UserRequest.java:54)
              at hudson.remoting.Request$2.run(Request.java:369)
              at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
          Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to foo
                  at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
                  at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                  at hudson.remoting.Channel.call(Channel.java:957)
                  at hudson.FilePath.act(FilePath.java:1072)
                  at hudson.FilePath.act(FilePath.java:1061)
                  at hudson.FilePath.deleteRecursive(FilePath.java:1268)
                  at org.jenkinsci.plugins.xunit.XUnitProcessor.processDeletion(XUnitProcessor.java:456)
                  at org.jenkinsci.plugins.xunit.XUnitProcessor.process(XUnitProcessor.java:166)
                  at org.jenkinsci.plugins.xunit.XUnitBuilder.perform(XUnitBuilder.java:112)
                  at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
                  at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
                  at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:51)
                  at hudson.security.ACL.impersonate(ACL.java:290)
                  at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:48)
                  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                  at java.lang.Thread.run(Thread.java:748)
          Caused: java.io.IOException: Unable to delete '/foo/bar/generatedJUnitFiles/d87f0d22-c235-43cd-8272-34ce92150592/GoogleTest'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
              at jenkins.util.io.PathRemover.forceRemoveFile(PathRemover.java:75)
              at hudson.Util.deleteFile(Util.java:255)
              at hudson.FilePath.deleteRecursive(FilePath.java:1305)
              at hudson.FilePath.access$1600(FilePath.java:213)
              at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1274)
              at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1270)
              at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
              at hudson.remoting.UserRequest.perform(UserRequest.java:212)
              at hudson.remoting.UserRequest.perform(UserRequest.java:54)
              at hudson.remoting.Request$2.run(Request.java:369)
              at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
          Finished: FAILURE
          

          Chad Williams added a comment - We experience the same/related issues: This is a critical bug for us, since test result parsing via the xUnit plugin fails and successful builds are marked as failures. I can confirm that downgrading to 2.156 fixes the issue. jenkins.util.io.CompositeIOException: Unable to remove directory /foo/bar/generatedJUnitFiles/d87f0d22-c235-43cd-8272-34ce92150592/GoogleTest with directory contents: [/foo/bar/generatedJUnitFiles/d87f0d22-c235-43cd-8272-34ce92150592/GoogleTest/TEST-809dd980-e514-4260-b749-9079e250f828.xml, /foo/bar/generatedJUnitFiles/d87f0d22-c235-43cd-8272-34ce92150592/GoogleTest/TEST-2d8be771-6364-49d8-9573-c51a974a44bc.xml] at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:233) at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:191) at jenkins.util.io.PathRemover.forceRemoveFile(PathRemover.java:71) at hudson.Util.deleteFile(Util.java:255) at hudson.FilePath.deleteRecursive(FilePath.java:1305) at hudson.FilePath.access$1600(FilePath.java:213) at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1274) at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1270) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) java.nio.file.DirectoryNotEmptyException: /foo/bar/generatedJUnitFiles/d87f0d22-c235-43cd-8272-34ce92150592/GoogleTest at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242) at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108) at java.nio.file.Files.deleteIfExists(Files.java:1165) at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:220) at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:191) at jenkins.util.io.PathRemover.forceRemoveFile(PathRemover.java:71) at hudson.Util.deleteFile(Util.java:255) at hudson.FilePath.deleteRecursive(FilePath.java:1305) at hudson.FilePath.access$1600(FilePath.java:213) at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1274) at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1270) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) java.nio.file.DirectoryNotEmptyException: /foo/bar/generatedJUnitFiles/d87f0d22-c235-43cd-8272-34ce92150592/GoogleTest at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242) at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108) at java.nio.file.Files.deleteIfExists(Files.java:1165) at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:224) at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:191) at jenkins.util.io.PathRemover.forceRemoveFile(PathRemover.java:71) at hudson.Util.deleteFile(Util.java:255) at hudson.FilePath.deleteRecursive(FilePath.java:1305) at hudson.FilePath.access$1600(FilePath.java:213) at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1274) at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1270) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to foo at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357) at hudson.remoting.Channel.call(Channel.java:957) at hudson.FilePath.act(FilePath.java:1072) at hudson.FilePath.act(FilePath.java:1061) at hudson.FilePath.deleteRecursive(FilePath.java:1268) at org.jenkinsci.plugins.xunit.XUnitProcessor.processDeletion(XUnitProcessor.java:456) at org.jenkinsci.plugins.xunit.XUnitProcessor.process(XUnitProcessor.java:166) at org.jenkinsci.plugins.xunit.XUnitBuilder.perform(XUnitBuilder.java:112) at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80) at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:51) at hudson.security.ACL.impersonate(ACL.java:290) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:48) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused: java.io.IOException: Unable to delete '/foo/bar/generatedJUnitFiles/d87f0d22-c235-43cd-8272-34ce92150592/GoogleTest'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. at jenkins.util.io.PathRemover.forceRemoveFile(PathRemover.java:75) at hudson.Util.deleteFile(Util.java:255) at hudson.FilePath.deleteRecursive(FilePath.java:1305) at hudson.FilePath.access$1600(FilePath.java:213) at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1274) at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1270) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Finished: FAILURE
          Chad Williams made changes -
          Priority Original: Minor [ 4 ] New: Critical [ 2 ]
          Wes Elliott made changes -
          Link New: This issue is related to JENKINS-55450 [ JENKINS-55450 ]

          Uwe Schindler added a comment -

          We have seen the same issue when the jenkins home folder is a symlink. The issue went away, when we changed this to be an absolute path.

          Uwe Schindler added a comment - We have seen the same issue when the jenkins home folder is a symlink. The issue went away, when we changed this to be an absolute path.

          Uwe Schindler added a comment - - edited

          Maybe that's the reason for the whole thing: If anywhere in the path up to the root folder there is a symlink, the delete directory stuff breaks. In our case the home directory of jenkins was a symlink.

          Uwe Schindler added a comment - - edited Maybe that's the reason for the whole thing: If anywhere in the path up to the root folder there is a symlink, the delete directory stuff breaks. In our case the home directory of jenkins was a symlink.
          Alexander Schmacks made changes -
          Assignee New: Alexander Schmacks [ aschmacks ]
          Alexander Schmacks made changes -
          Assignee Original: Alexander Schmacks [ aschmacks ]

            jvz Matt Sicker
            ejouvin Etienne Jouvin
            Votes:
            17 Vote for this issue
            Watchers:
            21 Start watching this issue

              Created:
              Updated:
              Resolved: