Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58674

jenkins.util.io.CompositeIOException exception on pipeline build abort

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Just recently we've encountered an issue when we want to abort Jenkins Pipeline jobs. When it happens it prevents build to use correct workspace. It's starting to select "@2" workspace instead (which breaks p4 sync'ing for an example).

      The worst thing is that the issue persists for the broken workspaces even if we manually delete "@2" versions, disconnect/restart slaves. Changing durability settings for jobs that use those workspaces doesn't help either. Only thing that appears to be helping is to restart Jenkins master.

      From what I could collect by myself it seems that durable-{id}\jenkins-log.txt is still being locked by slave's process (which is spawned in groovy by using bat step) while jenkins tries to delete it. Shouldn't it wait for the process to be killed?

      This happens with very high repro when we're using batch/python scripts (spawned in groovy by using bat step) to run child processes that write to stdout (or null) - jenkins-log.txt is being locked and not released in time for deletion.

       

      Callstack:

      *10:43:10  jenkins.util.io.CompositeIOException: Unable to delete 'D:\Hudson\workspace\jenkins_auto_buildersBK_b24_1556271982\q@tmp\durable-215a6186'. Tried 3 times (of a maximum of 3) waiting 0,1 sek between attempts.
      10:43:10  	at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:95)
      10:43:10  	at hudson.Util.deleteRecursive(Util.java:293)
      10:43:10  	at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1272)
      10:43:10  	at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1268)
      10:43:10  	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3050)
      10:43:10  	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
      10:43:10  	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      10:43:10  	at hudson.remoting.Request$2.run(Request.java:369)
      10:43:10  	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      10:43:10  	at java.util.concurrent.FutureTask.run(Unknown Source)
      10:43:10  	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      10:43:10  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      10:43:10  	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
      10:43:10  	at java.lang.Thread.run(Unknown Source)
      10:43:10  	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 172.17.41.124/172.17.41.124:49896
      10:43:10  		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
      10:43:10  		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
      10:43:10  		at hudson.remoting.Channel.call(Channel.java:957)
      10:43:10  		at hudson.FilePath.act(FilePath.java:1070)
      10:43:10  		at hudson.FilePath.act(FilePath.java:1059)
      10:43:10  		at hudson.FilePath.deleteRecursive(FilePath.java:1266)
      10:43:10  		at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.cleanup(FileMonitoringTask.java:349)
      10:43:10  		at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution$1.run(DurableTaskStep.java:417)
      10:43:10  		at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
      10:43:10  		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      10:43:10  		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      10:43:10  		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      10:43:10  		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      10:43:10  		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      10:43:10  		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      10:43:10  		at java.lang.Thread.run(Thread.java:748)
      10:43:10  jenkins.util.io.CompositeIOException: Unable to remove file D:\Hudson\workspace\jenkins_auto_buildersBK_b24_1556271982\q@tmp\durable-215a6186\jenkins-log.txt
      10:43:10  	at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:248)
      10:43:10  	at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:201)
      10:43:10  	at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:212)
      10:43:10  	at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:222)
      10:43:10  	at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:211)
      10:43:10  	at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:92)
      10:43:10  	at hudson.Util.deleteRecursive(Util.java:293)
      10:43:10  	at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1272)
      10:43:10  	at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1268)
      10:43:10  	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3050)
      10:43:10  	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
      10:43:10  	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      10:43:10  	at hudson.remoting.Request$2.run(Request.java:369)
      10:43:10  	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      10:43:10  	at java.util.concurrent.FutureTask.run(Unknown Source)
      10:43:10  	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      10:43:10  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      10:43:10  	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
      10:43:10  	at java.lang.Thread.run(Unknown Source)
      10:43:10  java.nio.file.FileSystemException: D:\Hudson\workspace\jenkins_auto_buildersBK_b24_1556271982\q@tmp\durable-215a6186\jenkins-log.txt: Proces nie może uzyskać dostępu do pliku, ponieważ jest on używany przez inny proces.
      10:43:10  
      10:43:10  	at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
      10:43:10  	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
      10:43:10  	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
      10:43:10  	at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
      10:43:10  	at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
      10:43:10  	at java.nio.file.Files.deleteIfExists(Unknown Source)
      10:43:10  	at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:233)
      10:43:10  	at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:201)
      10:43:10  	at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:212)
      10:43:10  	at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:222)
      10:43:10  	at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:211)
      10:43:10  	at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:92)
      10:43:10  	at hudson.Util.deleteRecursive(Util.java:293)
      10:43:10  	at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1272)
      10:43:10  	at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1268)
      10:43:10  	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3050)
      10:43:10  	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
      10:43:10  	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      10:43:10  	at hudson.remoting.Request$2.run(Request.java:369)
      10:43:10  	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      10:43:10  	at java.util.concurrent.FutureTask.run(Unknown Source)
      10:43:10  	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      10:43:10  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      10:43:10  	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
      10:43:10  	at java.lang.Thread.run(Unknown Source)
      10:43:10  java.nio.file.FileSystemException: D:\Hudson\workspace\jenkins_auto_buildersBK_b24_1556271982\q@tmp\durable-215a6186\jenkins-log.txt: Proces nie może uzyskać dostępu do pliku, ponieważ jest on używany przez inny proces.
      10:43:10  
      10:43:10  	at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
      10:43:10  	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
      10:43:10  	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
      10:43:10  	at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
      10:43:10  	at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
      10:43:10  	at java.nio.file.Files.deleteIfExists(Unknown Source)
      10:43:10  	at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:237)
      10:43:10  	at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:201)
      10:43:10  	at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:212)
      10:43:10  	at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:222)
      10:43:10  	at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:211)
      10:43:10  	at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:92)
      10:43:10  	at hudson.Util.deleteRecursive(Util.java:293)
      10:43:10  	at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1272)
      10:43:10  	at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1268)
      10:43:10  	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3050)
      10:43:10  	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
      10:43:10  	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      10:43:10  	at hudson.remoting.Request$2.run(Request.java:369)
      10:43:10  	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      10:43:10  	at java.util.concurrent.FutureTask.run(Unknown Source)
      10:43:10  	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      10:43:10  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      10:43:10  	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
      10:43:10  	at java.lang.Thread.run(Unknown Source)
      10:43:10  jenkins.util.io.CompositeIOException: Unable to remove directory D:\Hudson\workspace\jenkins_auto_buildersBK_b24_1556271982\q@tmp\durable-215a6186 with directory contents: [D:\Hudson\workspace\jenkins_auto_buildersBK_b24_1556271982\q@tmp\durable-215a6186\jenkins-log.txt]
      10:43:10  	at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:246)
      10:43:10  	at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:201)
      10:43:10  	at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:212)
      10:43:10  	at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:92)
      10:43:10  	at hudson.Util.deleteRecursive(Util.java:293)
      10:43:10  	at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1272)
      10:43:10  	at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1268)
      10:43:10  	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3050)
      10:43:10  	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
      10:43:10  	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      10:43:10  	at hudson.remoting.Request$2.run(Request.java:369)
      10:43:10  	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      10:43:10  	at java.util.concurrent.FutureTask.run(Unknown Source)
      10:43:10  	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      10:43:10  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      10:43:10  	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
      10:43:10  	at java.lang.Thread.run(Unknown Source)
      10:43:10  java.nio.file.DirectoryNotEmptyException: D:\Hudson\workspace\jenkins_auto_buildersBK_b24_1556271982\q@tmp\durable-215a6186
      10:43:10  	at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
      10:43:10  	at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
      10:43:10  	at java.nio.file.Files.deleteIfExists(Unknown Source)
      10:43:10  	at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:233)
      10:43:10  	at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:201)
      10:43:10  	at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:212)
      10:43:10  	at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:92)
      10:43:10  	at hudson.Util.deleteRecursive(Util.java:293)
      10:43:10  	at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1272)
      10:43:10  	at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1268)
      10:43:10  	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3050)
      10:43:10  	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
      10:43:10  	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      10:43:10  	at hudson.remoting.Request$2.run(Request.java:369)
      10:43:10  	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      10:43:10  	at java.util.concurrent.FutureTask.run(Unknown Source)
      10:43:10  	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      10:43:10  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      10:43:10  	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
      10:43:10  	at java.lang.Thread.run(Unknown Source)
      10:43:10  java.nio.file.DirectoryNotEmptyException: D:\Hudson\workspace\jenkins_auto_buildersBK_b24_1556271982\q@tmp\durable-215a6186
      10:43:10  	at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
      10:43:10  	at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
      10:43:10  	at java.nio.file.Files.deleteIfExists(Unknown Source)
      10:43:10  	at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:237)
      10:43:10  	at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:201)
      10:43:10  	at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:212)
      10:43:10  	at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:92)
      10:43:10  	at hudson.Util.deleteRecursive(Util.java:293)
      10:43:10  	at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1272)
      10:43:10  	at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1268)
      10:43:10  	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3050)
      10:43:10  	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
      10:43:10  	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      10:43:10  	at hudson.remoting.Request$2.run(Request.java:369)
      10:43:10  	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      10:43:10  	at java.util.concurrent.FutureTask.run(Unknown Source)
      10:43:10  	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      10:43:10  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      10:43:10  	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
      10:43:10  	at java.lang.Thread.run(Unknown Source)*
      

      "Proces nie może uzyskać dostępu do pliku, ponieważ jest on używany przez inny proces." means that "The process cannot access the file because it is being used by another process"

        Attachments

          Activity

          There are no comments yet on this issue.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            mdelikat Mateusz Delikat
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated: