-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Critical
-
Environment:Jenkins ver. 2.176.1
Durable Task ver 1.29
Pipeline: Groovy ver 2.70
Pipeline: Job ver 2.32
Pipeline: Nodes and Processes ver 2.31
Master: Linux
Slave: Windows 10
Just recently we've encountered an issue when we want to abort Jenkins Pipeline jobs. When it happens it prevents build to use correct workspace. It's starting to select "@2" workspace instead (which breaks p4 sync'ing for an example).
The worst thing is that the issue persists for the broken workspaces even if we manually delete "@2" versions, disconnect/restart slaves. Changing durability settings for jobs that use those workspaces doesn't help either. Only thing that appears to be helping is to restart Jenkins master.
From what I could collect by myself it seems that durable-{id}\jenkins-log.txt is still being locked by slave's process (which is spawned in groovy by using bat step) while jenkins tries to delete it. Shouldn't it wait for the process to be killed?
This happens with very high repro when we're using batch/python scripts (spawned in groovy by using bat step) to run child processes that write to stdout (or null) - jenkins-log.txt is being locked and not released in time for deletion.
Callstack:
*10:43:10 jenkins.util.io.CompositeIOException: Unable to delete 'D:\Hudson\workspace\jenkins_auto_buildersBK_b24_1556271982\q@tmp\durable-215a6186'. Tried 3 times (of a maximum of 3) waiting 0,1 sek between attempts. 10:43:10 at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:95) 10:43:10 at hudson.Util.deleteRecursive(Util.java:293) 10:43:10 at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1272) 10:43:10 at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1268) 10:43:10 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3050) 10:43:10 at hudson.remoting.UserRequest.perform(UserRequest.java:212) 10:43:10 at hudson.remoting.UserRequest.perform(UserRequest.java:54) 10:43:10 at hudson.remoting.Request$2.run(Request.java:369) 10:43:10 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) 10:43:10 at java.util.concurrent.FutureTask.run(Unknown Source) 10:43:10 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 10:43:10 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 10:43:10 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93) 10:43:10 at java.lang.Thread.run(Unknown Source) 10:43:10 Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 172.17.41.124/172.17.41.124:49896 10:43:10 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743) 10:43:10 at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357) 10:43:10 at hudson.remoting.Channel.call(Channel.java:957) 10:43:10 at hudson.FilePath.act(FilePath.java:1070) 10:43:10 at hudson.FilePath.act(FilePath.java:1059) 10:43:10 at hudson.FilePath.deleteRecursive(FilePath.java:1266) 10:43:10 at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.cleanup(FileMonitoringTask.java:349) 10:43:10 at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution$1.run(DurableTaskStep.java:417) 10:43:10 at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) 10:43:10 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 10:43:10 at java.util.concurrent.FutureTask.run(FutureTask.java:266) 10:43:10 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) 10:43:10 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) 10:43:10 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 10:43:10 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 10:43:10 at java.lang.Thread.run(Thread.java:748) 10:43:10 jenkins.util.io.CompositeIOException: Unable to remove file D:\Hudson\workspace\jenkins_auto_buildersBK_b24_1556271982\q@tmp\durable-215a6186\jenkins-log.txt 10:43:10 at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:248) 10:43:10 at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:201) 10:43:10 at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:212) 10:43:10 at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:222) 10:43:10 at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:211) 10:43:10 at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:92) 10:43:10 at hudson.Util.deleteRecursive(Util.java:293) 10:43:10 at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1272) 10:43:10 at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1268) 10:43:10 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3050) 10:43:10 at hudson.remoting.UserRequest.perform(UserRequest.java:212) 10:43:10 at hudson.remoting.UserRequest.perform(UserRequest.java:54) 10:43:10 at hudson.remoting.Request$2.run(Request.java:369) 10:43:10 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) 10:43:10 at java.util.concurrent.FutureTask.run(Unknown Source) 10:43:10 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 10:43:10 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 10:43:10 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93) 10:43:10 at java.lang.Thread.run(Unknown Source) 10:43:10 java.nio.file.FileSystemException: D:\Hudson\workspace\jenkins_auto_buildersBK_b24_1556271982\q@tmp\durable-215a6186\jenkins-log.txt: Proces nie może uzyskać dostępu do pliku, ponieważ jest on używany przez inny proces. 10:43:10 10:43:10 at sun.nio.fs.WindowsException.translateToIOException(Unknown Source) 10:43:10 at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) 10:43:10 at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) 10:43:10 at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source) 10:43:10 at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source) 10:43:10 at java.nio.file.Files.deleteIfExists(Unknown Source) 10:43:10 at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:233) 10:43:10 at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:201) 10:43:10 at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:212) 10:43:10 at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:222) 10:43:10 at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:211) 10:43:10 at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:92) 10:43:10 at hudson.Util.deleteRecursive(Util.java:293) 10:43:10 at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1272) 10:43:10 at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1268) 10:43:10 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3050) 10:43:10 at hudson.remoting.UserRequest.perform(UserRequest.java:212) 10:43:10 at hudson.remoting.UserRequest.perform(UserRequest.java:54) 10:43:10 at hudson.remoting.Request$2.run(Request.java:369) 10:43:10 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) 10:43:10 at java.util.concurrent.FutureTask.run(Unknown Source) 10:43:10 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 10:43:10 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 10:43:10 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93) 10:43:10 at java.lang.Thread.run(Unknown Source) 10:43:10 java.nio.file.FileSystemException: D:\Hudson\workspace\jenkins_auto_buildersBK_b24_1556271982\q@tmp\durable-215a6186\jenkins-log.txt: Proces nie może uzyskać dostępu do pliku, ponieważ jest on używany przez inny proces. 10:43:10 10:43:10 at sun.nio.fs.WindowsException.translateToIOException(Unknown Source) 10:43:10 at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) 10:43:10 at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) 10:43:10 at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source) 10:43:10 at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source) 10:43:10 at java.nio.file.Files.deleteIfExists(Unknown Source) 10:43:10 at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:237) 10:43:10 at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:201) 10:43:10 at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:212) 10:43:10 at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:222) 10:43:10 at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:211) 10:43:10 at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:92) 10:43:10 at hudson.Util.deleteRecursive(Util.java:293) 10:43:10 at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1272) 10:43:10 at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1268) 10:43:10 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3050) 10:43:10 at hudson.remoting.UserRequest.perform(UserRequest.java:212) 10:43:10 at hudson.remoting.UserRequest.perform(UserRequest.java:54) 10:43:10 at hudson.remoting.Request$2.run(Request.java:369) 10:43:10 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) 10:43:10 at java.util.concurrent.FutureTask.run(Unknown Source) 10:43:10 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 10:43:10 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 10:43:10 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93) 10:43:10 at java.lang.Thread.run(Unknown Source) 10:43:10 jenkins.util.io.CompositeIOException: Unable to remove directory D:\Hudson\workspace\jenkins_auto_buildersBK_b24_1556271982\q@tmp\durable-215a6186 with directory contents: [D:\Hudson\workspace\jenkins_auto_buildersBK_b24_1556271982\q@tmp\durable-215a6186\jenkins-log.txt] 10:43:10 at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:246) 10:43:10 at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:201) 10:43:10 at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:212) 10:43:10 at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:92) 10:43:10 at hudson.Util.deleteRecursive(Util.java:293) 10:43:10 at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1272) 10:43:10 at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1268) 10:43:10 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3050) 10:43:10 at hudson.remoting.UserRequest.perform(UserRequest.java:212) 10:43:10 at hudson.remoting.UserRequest.perform(UserRequest.java:54) 10:43:10 at hudson.remoting.Request$2.run(Request.java:369) 10:43:10 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) 10:43:10 at java.util.concurrent.FutureTask.run(Unknown Source) 10:43:10 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 10:43:10 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 10:43:10 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93) 10:43:10 at java.lang.Thread.run(Unknown Source) 10:43:10 java.nio.file.DirectoryNotEmptyException: D:\Hudson\workspace\jenkins_auto_buildersBK_b24_1556271982\q@tmp\durable-215a6186 10:43:10 at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source) 10:43:10 at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source) 10:43:10 at java.nio.file.Files.deleteIfExists(Unknown Source) 10:43:10 at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:233) 10:43:10 at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:201) 10:43:10 at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:212) 10:43:10 at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:92) 10:43:10 at hudson.Util.deleteRecursive(Util.java:293) 10:43:10 at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1272) 10:43:10 at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1268) 10:43:10 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3050) 10:43:10 at hudson.remoting.UserRequest.perform(UserRequest.java:212) 10:43:10 at hudson.remoting.UserRequest.perform(UserRequest.java:54) 10:43:10 at hudson.remoting.Request$2.run(Request.java:369) 10:43:10 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) 10:43:10 at java.util.concurrent.FutureTask.run(Unknown Source) 10:43:10 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 10:43:10 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 10:43:10 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93) 10:43:10 at java.lang.Thread.run(Unknown Source) 10:43:10 java.nio.file.DirectoryNotEmptyException: D:\Hudson\workspace\jenkins_auto_buildersBK_b24_1556271982\q@tmp\durable-215a6186 10:43:10 at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source) 10:43:10 at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source) 10:43:10 at java.nio.file.Files.deleteIfExists(Unknown Source) 10:43:10 at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:237) 10:43:10 at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:201) 10:43:10 at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:212) 10:43:10 at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:92) 10:43:10 at hudson.Util.deleteRecursive(Util.java:293) 10:43:10 at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1272) 10:43:10 at hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1268) 10:43:10 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3050) 10:43:10 at hudson.remoting.UserRequest.perform(UserRequest.java:212) 10:43:10 at hudson.remoting.UserRequest.perform(UserRequest.java:54) 10:43:10 at hudson.remoting.Request$2.run(Request.java:369) 10:43:10 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) 10:43:10 at java.util.concurrent.FutureTask.run(Unknown Source) 10:43:10 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 10:43:10 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 10:43:10 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93) 10:43:10 at java.lang.Thread.run(Unknown Source)*
"Proces nie może uzyskać dostępu do pliku, ponieważ jest on używany przez inny proces." means that "The process cannot access the file because it is being used by another process"
