Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-24824

Asynchronous cleanup not removing renamed workspace directories on slaves

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • ws-cleanup-plugin
    • None
    • Jenkins 1.579
      ws-cleanup 0.24

      After upgrading to ws-cleanup 0.24 in order to get the asynchronous cleanup function we noticed the workspaces on our slaves getting renamed to the form of ${WORKSPACE}ws-cleanup${TIMESTAMP}. (ie, job1 would become job1_ws-cleanup_1411197183394). The expected behavior under ws-cleanup 0.24 is that these were temporary to support asynchronous processing and would be deleted. However, these directories never get removed from the slave. Over time, the slave hard drives filled up resulting in build failures.

          [JENKINS-24824] Asynchronous cleanup not removing renamed workspace directories on slaves

          Tom Moore added a comment -

          No, we don't put the slaves offline. We didn't notice any errors in the logs, the first indication we had that this was a problem was when we ran out of disk space.

          Tom Moore added a comment - No, we don't put the slaves offline. We didn't notice any errors in the logs, the first indication we had that this was a problem was when we ran out of disk space.

          It seems that the files we have trouble with are the ones that contain Danish characters æ,ø or å in the file name.

          Katrine Skovbo added a comment - It seems that the files we have trouble with are the ones that contain Danish characters æ,ø or å in the file name.

          Daniel Beck added a comment -

          katrine: Are these characters in the workspace (project) name, a folder name in the workspace, or a file name in the workspace? Are you deleting the entire workspace? Did it work on the older Jenkins version? What are the contents of the /systemInfo URL (or /computer/slavename/systemInfo URL for workspaces on slaves)?

          Daniel Beck added a comment - katrine : Are these characters in the workspace (project) name, a folder name in the workspace, or a file name in the workspace? Are you deleting the entire workspace? Did it work on the older Jenkins version? What are the contents of the /systemInfo URL (or /computer/slavename/systemInfo URL for workspaces on slaves)?

          The characters are in file names in the workspace.
          We delete the entire workspace.
          It worked before the upgrade to version 0.24, but I tried downgrading to 0.23 again and that also produced errors now. In 0.23 it now explicitly says that it cannot delete the files:

          ERROR: Failed to clean the workspace
          09:05:02 java.io.IOException: java.lang.reflect.InvocationTargetException
          09:05:02 	at hudson.Util.isSymlinkJava7(Util.java:360)
          09:05:02 	at hudson.Util.isSymlink(Util.java:325)
          09:05:02 	at hudson.Util.deleteRecursive(Util.java:291)
          09:05:02 	at hudson.Util.deleteContentsRecursive(Util.java:203)
          09:05:02 	at hudson.Util.deleteRecursive(Util.java:292)
          09:05:02 	at hudson.Util.deleteContentsRecursive(Util.java:203)
          09:05:02 	at hudson.Util.deleteRecursive(Util.java:292)
          09:05:02 	at hudson.Util.deleteContentsRecursive(Util.java:203)
          09:05:02 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:393)
          09:05:02 	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
          09:05:02 	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
          09:05:02 	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          09:05:02 	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          09:05:02 	at hudson.remoting.Request$2.run(Request.java:328)
          09:05:02 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          09:05:02 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
          09:05:02 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          09:05:02 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          09:05:02 	at java.lang.Thread.run(Thread.java:745)
          09:05:02 Caused by: java.lang.reflect.InvocationTargetException
          09:05:02 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          09:05:02 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          09:05:02 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          09:05:02 	at java.lang.reflect.Method.invoke(Method.java:606)
          09:05:02 	at hudson.Util.isSymlinkJava7(Util.java:355)
          09:05:02 	... 18 more
          09:05:02 Caused by: java.nio.file.InvalidPathException: Malformed input or input contains unmappable chacraters: /var/lib/jenkins/workspace/My-project/sub-folder/prototype/icons/skylleanl??g.svg
          09:05:02 	at sun.nio.fs.UnixPath.encode(UnixPath.java:147)
          09:05:02 	at sun.nio.fs.UnixPath.<init>(UnixPath.java:71)
          09:05:02 	at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:281)
          09:05:02 	at java.io.File.toPath(File.java:2186)
          09:05:02 	... 23 more
          

          We are using Jenkins 1.565.3 and a Unix slave, version 2.46, java 1.7.0_72

          Katrine Skovbo added a comment - The characters are in file names in the workspace. We delete the entire workspace. It worked before the upgrade to version 0.24, but I tried downgrading to 0.23 again and that also produced errors now. In 0.23 it now explicitly says that it cannot delete the files: ERROR: Failed to clean the workspace 09:05:02 java.io.IOException: java.lang.reflect.InvocationTargetException 09:05:02 at hudson.Util.isSymlinkJava7(Util.java:360) 09:05:02 at hudson.Util.isSymlink(Util.java:325) 09:05:02 at hudson.Util.deleteRecursive(Util.java:291) 09:05:02 at hudson.Util.deleteContentsRecursive(Util.java:203) 09:05:02 at hudson.Util.deleteRecursive(Util.java:292) 09:05:02 at hudson.Util.deleteContentsRecursive(Util.java:203) 09:05:02 at hudson.Util.deleteRecursive(Util.java:292) 09:05:02 at hudson.Util.deleteContentsRecursive(Util.java:203) 09:05:02 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:393) 09:05:02 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153) 09:05:02 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146) 09:05:02 at hudson.remoting.UserRequest.perform(UserRequest.java:118) 09:05:02 at hudson.remoting.UserRequest.perform(UserRequest.java:48) 09:05:02 at hudson.remoting.Request$2.run(Request.java:328) 09:05:02 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) 09:05:02 at java.util.concurrent.FutureTask.run(FutureTask.java:262) 09:05:02 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 09:05:02 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 09:05:02 at java.lang. Thread .run( Thread .java:745) 09:05:02 Caused by: java.lang.reflect.InvocationTargetException 09:05:02 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 09:05:02 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 09:05:02 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 09:05:02 at java.lang.reflect.Method.invoke(Method.java:606) 09:05:02 at hudson.Util.isSymlinkJava7(Util.java:355) 09:05:02 ... 18 more 09:05:02 Caused by: java.nio.file.InvalidPathException: Malformed input or input contains unmappable chacraters: / var /lib/jenkins/workspace/My-project/sub-folder/prototype/icons/skylleanl??g.svg 09:05:02 at sun.nio.fs.UnixPath.encode(UnixPath.java:147) 09:05:02 at sun.nio.fs.UnixPath.<init>(UnixPath.java:71) 09:05:02 at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:281) 09:05:02 at java.io.File.toPath(File.java:2186) 09:05:02 ... 23 more We are using Jenkins 1.565.3 and a Unix slave, version 2.46, java 1.7.0_72

          vjuranek added a comment -

          katrine: this seems to be unrelated to the bug discussed in this issue. This is probably different bug and this error is just hidden by async delete. I filed a new issue for this: JENKINS-25317

          vjuranek added a comment - katrine : this seems to be unrelated to the bug discussed in this issue. This is probably different bug and this error is just hidden by async delete. I filed a new issue for this: JENKINS-25317

          Oliver Gondža added a comment - - edited

          I expect the deletion can fail for some reason even after we successfully renamed the directory. I see several ways to improve current situation:

          • Retry the deletion until it succeeds - it should not block the build but can as well run for unpredictable amount of time
          • Introduce a periodic task to clean up such dangling directories
          • Pass Future from FilePath.actAsync to some "collector" task that would retry and/or report in case some deletion fails.*

          [*] I guess this might be the only way to find out why it failed on the first place as it seems that FilePath.actAsync does not log exceptions.

          Oliver Gondža added a comment - - edited I expect the deletion can fail for some reason even after we successfully renamed the directory. I see several ways to improve current situation: Retry the deletion until it succeeds - it should not block the build but can as well run for unpredictable amount of time Introduce a periodic task to clean up such dangling directories Pass Future from FilePath.actAsync to some "collector" task that would retry and/or report in case some deletion fails.* [*] I guess this might be the only way to find out why it failed on the first place as it seems that FilePath.actAsync does not log exceptions.

          Victor Volle added a comment -

          We added some logging around the deletion (see below) and found that there was a permissions problem.

          I would suggest to apply the patch to the plugin so that others can see the root cause as well

          From 4dd5c02d5c65a30860ed4ccc2baa860f331f156f Mon Sep 17 00:00:00 2001
          From: Victor Volle <vivo@Victors-MacBook-Pro.local>
          Date: Tue, 18 Aug 2015 17:53:41 +0200
          Subject: [PATCH] JENKINS-24824: log error
          
          ---
           src/main/java/hudson/plugins/ws_cleanup/Wipeout.java | 7 ++++++-
           1 file changed, 6 insertions(+), 1 deletion(-)
          
          diff --git a/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java b/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java
          index e1e759b..a9e50a6 100644
          --- a/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java
          +++ b/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java
          @@ -69,7 +69,12 @@ import hudson.remoting.VirtualChannel;
               private final static Command COMMAND = new Command();
               private final static class Command implements FileCallable<Object> {
                   public Object invoke(File f, VirtualChannel channel) throws IOException, InterruptedException {
          -            Util.deleteRecursive(f);
          +            try {
          +                Util.deleteRecursive(f);
          +            } catch (IOException e) {
          +                LOGGER.log(Level.WARNING, "error cleaning up", e);
          +                throw e;
          +            }
                       return null;
                   }
               }
          -- 
          2.3.2 (Apple Git-55)
          

          Victor Volle added a comment - We added some logging around the deletion (see below) and found that there was a permissions problem. I would suggest to apply the patch to the plugin so that others can see the root cause as well From 4dd5c02d5c65a30860ed4ccc2baa860f331f156f Mon Sep 17 00:00:00 2001 From: Victor Volle <vivo@Victors-MacBook-Pro.local> Date: Tue, 18 Aug 2015 17:53:41 +0200 Subject: [PATCH] JENKINS-24824: log error --- src/main/java/hudson/plugins/ws_cleanup/Wipeout.java | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java b/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java index e1e759b..a9e50a6 100644 --- a/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java +++ b/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java @@ -69,7 +69,12 @@ import hudson.remoting.VirtualChannel; private final static Command COMMAND = new Command(); private final static class Command implements FileCallable< Object > { public Object invoke(File f, VirtualChannel channel) throws IOException, InterruptedException { - Util.deleteRecursive(f); + try { + Util.deleteRecursive(f); + } catch (IOException e) { + LOGGER.log(Level.WARNING, "error cleaning up" , e); + throw e; + } return null ; } } -- 2.3.2 (Apple Git-55)

          Code changed in jenkins
          User: Oliver Gondža
          Path:
          src/main/java/hudson/plugins/ws_cleanup/Wipeout.java
          http://jenkins-ci.org/commit/ws-cleanup-plugin/e151b89222f93b8b125f3f5e1dffb51e6ad8be9d
          Log:
          JENKINS-24824 Log deletion failure

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: src/main/java/hudson/plugins/ws_cleanup/Wipeout.java http://jenkins-ci.org/commit/ws-cleanup-plugin/e151b89222f93b8b125f3f5e1dffb51e6ad8be9d Log: JENKINS-24824 Log deletion failure

          Code changed in jenkins
          User: Oliver Gondža
          Path:
          src/main/java/hudson/plugins/ws_cleanup/Wipeout.java
          http://jenkins-ci.org/commit/ws-cleanup-plugin/32766a78cf81399fcd0b46f68600863c0369a869
          Log:
          Merge pull request #22 from olivergondza/JENKINS-24824

          JENKINS-24824 Log deletion failure

          Compare: https://github.com/jenkinsci/ws-cleanup-plugin/compare/6bc314398d0f...32766a78cf81

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: src/main/java/hudson/plugins/ws_cleanup/Wipeout.java http://jenkins-ci.org/commit/ws-cleanup-plugin/32766a78cf81399fcd0b46f68600863c0369a869 Log: Merge pull request #22 from olivergondza/ JENKINS-24824 JENKINS-24824 Log deletion failure Compare: https://github.com/jenkinsci/ws-cleanup-plugin/compare/6bc314398d0f...32766a78cf81

          Same issue here.

          We have one Linux master running multiple slaves, some windows, some Linux.

          Linux slaves (VMWare VM) get a revert to snapshot using vmware plugin, which is not possible for windows slaves in our current environment.

          To handle hanging processes and locked files, we disconnect each windows slave after 1 single build, shutdown and power on the vm.

          Our problem are ever growing workspace folders, allocating far more than 80gig of disk space within about two weeks, as the async cleanup can not run and remove old workspaces due to premature disconnect and termination of the java vm.

          In our case the workspace cleanup plugin can rename old workspaces successfully (* -> *ws-cleanup[1234567890]+) , but does not delete those old folders after successfully reconnecting the windows slaves.

          We get no errors logged using ws-cleanup plugin version 0.28 in our build logs.

          Tim-Christian Bloss added a comment - Same issue here. We have one Linux master running multiple slaves, some windows, some Linux. Linux slaves (VMWare VM) get a revert to snapshot using vmware plugin, which is not possible for windows slaves in our current environment. To handle hanging processes and locked files, we disconnect each windows slave after 1 single build, shutdown and power on the vm. Our problem are ever growing workspace folders, allocating far more than 80gig of disk space within about two weeks, as the async cleanup can not run and remove old workspaces due to premature disconnect and termination of the java vm. In our case the workspace cleanup plugin can rename old workspaces successfully (* -> * ws-cleanup [1234567890] +) , but does not delete those old folders after successfully reconnecting the windows slaves. We get no errors logged using ws-cleanup plugin version 0.28 in our build logs.

          Gergely Brautigam added a comment - - edited

          Hi.

          Same issue here. Looks like the async delete isn't happening fast enough or isn't happening at all. Most of the time, when our git ran, but the job failed early, the async cleanup did not have time to finish deleting a folder.

          Or the other problem is usually an access denied error. Note, not a permission, but access. Meaning, the directory was still being used by some loose thread of a process.

          For now, I'm inclined to create a script on the slaves and put it into cron job which runs and cleans up every folder with ws-cleanup in it. But that is not a really nice solution.

          EDIT: And yes, same here, usually there is no error involved. Just a [WS-CLEANUP] Deleting project workspace... and no 'Done' in the logs anywhere.

          Gergely Brautigam added a comment - - edited Hi. Same issue here. Looks like the async delete isn't happening fast enough or isn't happening at all. Most of the time, when our git ran, but the job failed early, the async cleanup did not have time to finish deleting a folder. Or the other problem is usually an access denied error. Note, not a permission, but access. Meaning, the directory was still being used by some loose thread of a process. For now, I'm inclined to create a script on the slaves and put it into cron job which runs and cleans up every folder with ws-cleanup in it. But that is not a really nice solution. EDIT : And yes, same here, usually there is no error involved. Just a [WS-CLEANUP] Deleting project workspace... and no 'Done' in the logs anywhere.

          Oliver Gondža added a comment - - edited

          tcb_xy Your use-case is quite special and was not taken into account originally.

          skarlso

          Looks like the async delete isn't happening fast enough or isn't happening at all. Most of the time, when our git ran, but the job failed early, the async cleanup did not have time to finish deleting a folder.

          I do not understand this. How does the problem manifests?

          Note the async deletion failure cuses should not go into a build log (as is runs in parallel with the build and can even run for a longer time than the build itself) but a java logger. Check Jenkins master log or configure custom logged in UI for hudson.plugins.ws_cleanup to see why it fails.

          In the meantime, JENKINS-27648 was rejected in jenkins core so I guess we have to implement a periodic task to clean dangling *ws-cleanup* dirs. That should resolve the problems with slaves disconnected unexpectedly and directories not deleted in time because of opened file descriptors.

          Oliver Gondža added a comment - - edited tcb_xy Your use-case is quite special and was not taken into account originally. skarlso Looks like the async delete isn't happening fast enough or isn't happening at all. Most of the time, when our git ran, but the job failed early, the async cleanup did not have time to finish deleting a folder. I do not understand this. How does the problem manifests? Note the async deletion failure cuses should not go into a build log (as is runs in parallel with the build and can even run for a longer time than the build itself) but a java logger. Check Jenkins master log or configure custom logged in UI for hudson.plugins.ws_cleanup to see why it fails. In the meantime, JENKINS-27648 was rejected in jenkins core so I guess we have to implement a periodic task to clean dangling *ws-cleanup* dirs. That should resolve the problems with slaves disconnected unexpectedly and directories not deleted in time because of opened file descriptors.

          The symptom is a lingering ws-cleanup folder with the timestamp which didn't get deleted. Btw, most of the times it manifests on windows only. Rarely happens on a linux slave interestingly enough.

          Yes, sorry, the access denied error message was in the Jenkins Log.

          However, the Done, part is in the job's log. And that one was obviously missing because of the exception in the main log.

          Gergely Brautigam added a comment - The symptom is a lingering ws-cleanup folder with the timestamp which didn't get deleted. Btw, most of the times it manifests on windows only. Rarely happens on a linux slave interestingly enough. Yes, sorry, the access denied error message was in the Jenkins Log. However, the Done, part is in the job's log. And that one was obviously missing because of the exception in the main log.

          Daniel Beck added a comment -

          interestingly enough

          Windows file locking?

          Daniel Beck added a comment - interestingly enough Windows file locking?

          Windows file locking?

          Lacking enough evidence to say yay, or nay. It is 'a' possibility.

          Or even some other process locking a certain file. I even saw git lingering and the .git folder could not be deleted in a RENAMED folder. So much crazy, such wow.

          Gergely Brautigam added a comment - Windows file locking? Lacking enough evidence to say yay, or nay. It is 'a' possibility. Or even some other process locking a certain file. I even saw git lingering and the .git folder could not be deleted in a RENAMED folder. So much crazy, such wow.

          Daniel Beck added a comment -

          some other process locking a certain file. I even saw git lingering

          Sorry, that's what I meant. Too much locking going on and Windows doesn't allow deleting folders with open handles.

          Daniel Beck added a comment - some other process locking a certain file. I even saw git lingering Sorry, that's what I meant. Too much locking going on and Windows doesn't allow deleting folders with open handles.

          Affirmative. When it happens, it's usually because of windows file locking.

          Gergely Brautigam added a comment - Affirmative. When it happens, it's usually because of windows file locking.

          Frank Vissing added a comment - - edited

          I have a reproduce that can trigger this.. not the same as the classic slave setup but still.
          Provision a docker slave that will be provisioned with the 'One retention strategy'
          have a job that executes in the docker container (my job has less than 100 mb data)
          perform ws clean-up on success, fail etc.
          now once the job completes the docker container is killed and the wc-cleanup_timestamp folder remains
          this job runs on linux, so no relation to windows here
          one option could be a checkbox to disable assync delete
          the /home/jenkins/workspace folder is mounted in the docker container using data volumes, therfore the data remains on the filesystem until we destroy the data volumen

          The workaround for this is not to delete the workspace after building but prior to building.

          Frank Vissing added a comment - - edited I have a reproduce that can trigger this.. not the same as the classic slave setup but still. Provision a docker slave that will be provisioned with the 'One retention strategy' have a job that executes in the docker container (my job has less than 100 mb data) perform ws clean-up on success, fail etc. now once the job completes the docker container is killed and the wc-cleanup_timestamp folder remains this job runs on linux, so no relation to windows here one option could be a checkbox to disable assync delete the /home/jenkins/workspace folder is mounted in the docker container using data volumes, therfore the data remains on the filesystem until we destroy the data volumen The workaround for this is not to delete the workspace after building but prior to building.

          Jarkko Rantavuori added a comment - - edited

          We get this all the time - for out test jobs, there are hundreds of cleanup folders appearing on our disk. This is Ubuntu 14.04.1 64-bit slave, so at least for us it is not related to windows file locking. Also, we have had "Delete workspace before build starts" selected, but it has not fixed the issue.

          Our Jenkins version is 1.632, cleanup plugin 0.26.

          Update: reverting to cleanup plugin 0.23, which didn't have asynch deletion, allowed us to see the error: we had a shell script section in the jobs which created a folder having root owner instead of jenkins, so the following runs of a job would fail since there weren't able to delete the previous workspace.

          I think what needs to be done for asynch plugin is to fail the build somehow if the deletion fails and report the error. Also, it would be nice if user could select between asynch and synch delete instead of having to downgrade all the way to 0.23.

          Jarkko Rantavuori added a comment - - edited We get this all the time - for out test jobs, there are hundreds of cleanup folders appearing on our disk. This is Ubuntu 14.04.1 64-bit slave, so at least for us it is not related to windows file locking. Also, we have had "Delete workspace before build starts" selected, but it has not fixed the issue. Our Jenkins version is 1.632, cleanup plugin 0.26. Update: reverting to cleanup plugin 0.23, which didn't have asynch deletion, allowed us to see the error: we had a shell script section in the jobs which created a folder having root owner instead of jenkins, so the following runs of a job would fail since there weren't able to delete the previous workspace. I think what needs to be done for asynch plugin is to fail the build somehow if the deletion fails and report the error. Also, it would be nice if user could select between asynch and synch delete instead of having to downgrade all the way to 0.23.

          Jens Doose added a comment - - edited

          I have the same behaviour in a non-slave environment, I don't know if this is the same or a new issue.
          A lot of directories like workspace_ws-cleanup_1447277714463 get created and never deleted.

          In the log of the jenkins job there is the output that everything is ok:
          [WS-CLEANUP] Deleting project workspace...
          [WS-CLEANUP] Done

          The job is building node projects and loading npm componentes as well as bower components which result in quite huge directory structures, might that be a problem?

          Jens Doose added a comment - - edited I have the same behaviour in a non-slave environment, I don't know if this is the same or a new issue. A lot of directories like workspace_ws-cleanup_1447277714463 get created and never deleted. In the log of the jenkins job there is the output that everything is ok: [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done The job is building node projects and loading npm componentes as well as bower components which result in quite huge directory structures, might that be a problem?

          In fact if you want to switch "asynch/synch" mode, you can add Patterns, because this will affect cleanup method :

          • Add Patterns : synch mode
          • No Patterns : asynch mode

          This is just a trick to see the cleanup problem without downgrade to 0.23.

          Thomas Collignon added a comment - In fact if you want to switch "asynch/synch" mode, you can add Patterns, because this will affect cleanup method : Add Patterns : synch mode No Patterns : asynch mode This is just a trick to see the cleanup problem without downgrade to 0.23.

          And I have submit a pull request to add a new option to switch between synch and asynch mode without downgrading : https://github.com/jenkinsci/ws-cleanup-plugin/pull/26

          Thomas Collignon added a comment - And I have submit a pull request to add a new option to switch between synch and asynch mode without downgrading : https://github.com/jenkinsci/ws-cleanup-plugin/pull/26

          tcollignon, thanks for the workaround!

          @All, please check the logged (hudson.plugins.ws_cleanup.Wipeout) cause in slave/master log and attach it to this issue.

          Oliver Gondža added a comment - tcollignon , thanks for the workaround! @All, please check the logged ( hudson.plugins.ws_cleanup.Wipeout ) cause in slave/master log and attach it to this issue.

          It seems that people generally prefer to fail the build in case the cleanup fails rather that clutter the slave's workspace in the long run. This does not get well together with asynchronous deletion requirement.

          One way I can think of is always use sync approach from post-build step and async only in pre-build step. If the original (renamed) workspace will not be gone by the end of the build, sych deletion will be reattempted and will have a chance to fail the build in case of problems. The advantage is that build can start right away and most of the post build actions will do its job (publish junit, determine result, etc.) before the final cleanup kicks in.

          WDYT?

          Oliver Gondža added a comment - It seems that people generally prefer to fail the build in case the cleanup fails rather that clutter the slave's workspace in the long run. This does not get well together with asynchronous deletion requirement. One way I can think of is always use sync approach from post-build step and async only in pre-build step. If the original (renamed) workspace will not be gone by the end of the build, sych deletion will be reattempted and will have a chance to fail the build in case of problems. The advantage is that build can start right away and most of the post build actions will do its job (publish junit, determine result, etc.) before the final cleanup kicks in. WDYT?

          This may be a good thing, why not
          I ask myself if it's possible when asynch mode is on post-build, to get error et bring back to jenkins job trace. In this case the boolean "asynchronously" is necessary I Think to let people chosing the strategy.

          Thomas Collignon added a comment - This may be a good thing, why not I ask myself if it's possible when asynch mode is on post-build, to get error et bring back to jenkins job trace. In this case the boolean "asynchronously" is necessary I Think to let people chosing the strategy.

          At a certain point Jenkins considers the build log to be complete and never amended again. The problem is that in theory, the cleanup can take longer than the build itself. We can either not care about the result at all (current implementation), wait for the result at the end of the build (potentially postponing the build completion) or try a compromise - check the status at the end of the build and report failure if there is any but do not wait for the completion (this can not guarantee there will be no dangling workspace directories).

          When I think about this further, once we rename the workspace (as we do for async cleanup) it will always require manual cleanup in case of failure as it will not be any build's workspace any longer. When the cleanup fail with sync approach, it leaves the workspace half deleted to be reused by future builds - which is far from optimal too.

          I am working on implementing plugin specific periodic task to cleanup all the temporary directories (since more general solution was rejected from core).

          Oliver Gondža added a comment - At a certain point Jenkins considers the build log to be complete and never amended again. The problem is that in theory, the cleanup can take longer than the build itself. We can either not care about the result at all (current implementation), wait for the result at the end of the build (potentially postponing the build completion) or try a compromise - check the status at the end of the build and report failure if there is any but do not wait for the completion (this can not guarantee there will be no dangling workspace directories). When I think about this further, once we rename the workspace (as we do for async cleanup) it will always require manual cleanup in case of failure as it will not be any build's workspace any longer. When the cleanup fail with sync approach, it leaves the workspace half deleted to be reused by future builds - which is far from optimal too. I am working on implementing plugin specific periodic task to cleanup all the temporary directories (since more general solution was rejected from core).

          ok I see
          you work on another plugin to clean periodically or you think added this features in this one? you need some help?

          right now you think add option "asynchronously" is not necessary ?

          Thomas Collignon added a comment - ok I see you work on another plugin to clean periodically or you think added this features in this one? you need some help? right now you think add option "asynchronously" is not necessary ?

          Oliver Gondža added a comment - - edited

          My original idea was the async cleanup will be performance optimization that will be transparent to the user (whoever run the builds). Which it is only partially.

          I am extending this plugin with the periodic task. I understand the plugin becomes lot more sophisticated that we hoped it to be so I will implement a Jenkins-wide, property-based killswitch to get back to sync cleanup should it cause further problems.

          I still do not think that slave workspace getting full should be a concern of a user (as opposed to instance administrator), so I prefer this over a per-job configuration option.

          Oliver Gondža added a comment - - edited My original idea was the async cleanup will be performance optimization that will be transparent to the user (whoever run the builds). Which it is only partially. I am extending this plugin with the periodic task. I understand the plugin becomes lot more sophisticated that we hoped it to be so I will implement a Jenkins-wide, property-based killswitch to get back to sync cleanup should it cause further problems. I still do not think that slave workspace getting full should be a concern of a user (as opposed to instance administrator), so I prefer this over a per-job configuration option.

          Ok I agree with you. So I wait for your new implementation

          Thank for your answer

          Thomas Collignon added a comment - Ok I agree with you. So I wait for your new implementation Thank for your answer

          Bim Paras added a comment -

          Has this issue been resolved? I am still seeing this issue in version 0.25 of plugin. Thanks.

          Bim Paras added a comment - Has this issue been resolved? I am still seeing this issue in version 0.25 of plugin. Thanks.

          Sven Schott added a comment -

          Hi, just wanting to know if there is a timeframe on resolution of this issue. Would like to know just in case it's not in the near future so that I can set up a semi-temporary workaround for the problem (most likely a scheduled cleanup on our windows machines).

          Sven Schott added a comment - Hi, just wanting to know if there is a timeframe on resolution of this issue. Would like to know just in case it's not in the near future so that I can set up a semi-temporary workaround for the problem (most likely a scheduled cleanup on our windows machines).

          Jon Whitcraft added a comment -

          +1 to a timeframe for fixing this

          I'm thinking of rolling back to 0.23 becuase of this problem.

          Jon Whitcraft added a comment - +1 to a timeframe for fixing this I'm thinking of rolling back to 0.23 becuase of this problem.

          Mike Dimmick added a comment -

          I have been experiencing this problem on a Windows 7 Enterprise installation. During manual clean-up I noticed that even using the command line rmdir /s I was getting a lot of The directory is not empty errors, meaning I had to run the command twice to complete removal.

          I also noticed that a large amount of disk space was being consumed by Windows Search content indexing data. I disabled the Windows Search service and deleted the content indexes. Having done this, I'm no longer getting the errors from rmdir. I can also delete workspaces from the Jenkins 'Wipe Out Current Workspace' feature without problems, which was typically reporting an error previously.

          On this system, JENKINS_HOME is at C:\Users\jenkins\.jenkins. We also have an installation of Jenkins on Windows Server 2012 R2, which doesn't suffer from the problem - here, JENKINS_HOME is C:\Jenkins, and the Windows Search feature is not installed.

          Mike Dimmick added a comment - I have been experiencing this problem on a Windows 7 Enterprise installation. During manual clean-up I noticed that even using the command line rmdir /s I was getting a lot of The directory is not empty errors, meaning I had to run the command twice to complete removal. I also noticed that a large amount of disk space was being consumed by Windows Search content indexing data. I disabled the Windows Search service and deleted the content indexes. Having done this, I'm no longer getting the errors from rmdir . I can also delete workspaces from the Jenkins 'Wipe Out Current Workspace' feature without problems, which was typically reporting an error previously. On this system, JENKINS_HOME is at C:\Users\jenkins\.jenkins. We also have an installation of Jenkins on Windows Server 2012 R2, which doesn't suffer from the problem - here, JENKINS_HOME is C:\Jenkins, and the Windows Search feature is not installed.

          Daniel Beck added a comment -

          mdimmick_mnetics

          Windows Search content indexing data. I disabled the Windows Search service

          Don't run the Windows Search on JENKINS_HOME. Don't run antivirus on JENKINS_HOME. No exceptions.

          Daniel Beck added a comment - mdimmick_mnetics Windows Search content indexing data. I disabled the Windows Search service Don't run the Windows Search on JENKINS_HOME. Don't run antivirus on JENKINS_HOME. No exceptions.

          I am having a second look at this: https://github.com/jenkinsci/ws-cleanup-plugin/pull/28

          It implements a administrative monitor that retries the deletion on report failures in UI.

          Oliver Gondža added a comment - I am having a second look at this: https://github.com/jenkinsci/ws-cleanup-plugin/pull/28 It implements a administrative monitor that retries the deletion on report failures in UI.

          Code changed in jenkins
          User: Oliver Gondža
          Path:
          pom.xml
          src/main/java/hudson/plugins/ws_cleanup/Wipeout.java
          src/test/java/hudson/plugins/ws_cleanup/CleanupPowermockTest.java
          src/test/java/hudson/plugins/ws_cleanup/CleanupTest.java
          http://jenkins-ci.org/commit/ws-cleanup-plugin/ccd907188e0489e76ca21a7513843864bab48c90
          Log:
          [FIXED JENKINS-24824] Collect all asynchronously deleted directories

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: pom.xml src/main/java/hudson/plugins/ws_cleanup/Wipeout.java src/test/java/hudson/plugins/ws_cleanup/CleanupPowermockTest.java src/test/java/hudson/plugins/ws_cleanup/CleanupTest.java http://jenkins-ci.org/commit/ws-cleanup-plugin/ccd907188e0489e76ca21a7513843864bab48c90 Log: [FIXED JENKINS-24824] Collect all asynchronously deleted directories

          This took lot longer than I expected. The fix will be in 0.32.

          Oliver Gondža added a comment - This took lot longer than I expected. The fix will be in 0.32.

          aflat added a comment -

          I'm still hitting this on a solaris x86 machine. Seems to work on other OS's

          java.io.IOException: Unable to delete '/opt/jenkins/workspace/myjob-Solaris_ws-cleanup_1478635926754/local/.git/refs/remotes/origin'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
          at hudson.Util.deleteFile(Util.java:248)
          at hudson.FilePath.deleteRecursive(FilePath.java:1209)
          at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
          at hudson.FilePath.deleteRecursive(FilePath.java:1200)
          at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
          at hudson.FilePath.deleteRecursive(FilePath.java:1200)
          at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
          at hudson.FilePath.deleteRecursive(FilePath.java:1200)
          at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
          at hudson.FilePath.deleteRecursive(FilePath.java:1200)
          at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
          at hudson.FilePath.deleteRecursive(FilePath.java:1200)
          at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
          at hudson.FilePath.deleteRecursive(FilePath.java:1200)
          at hudson.FilePath.access$1000(FilePath.java:195)
          at hudson.FilePath$14.invoke(FilePath.java:1179)
          at hudson.FilePath$14.invoke(FilePath.java:1176)
          at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2731)
          at hudson.remoting.UserRequest.perform(UserRequest.java:153)
          at hudson.remoting.UserRequest.perform(UserRequest.java:50)
          at hudson.remoting.Request$2.run(Request.java:332)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
          at java.util.concurrent.FutureTask.run(FutureTask.java:262)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)
          at ......remote call to sun04(Native Method)
          at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1435)
          at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
          at hudson.remoting.Channel.call(Channel.java:795)
          at hudson.FilePath.act(FilePath.java:985)
          at hudson.FilePath.act(FilePath.java:974)
          at hudson.FilePath.deleteRecursive(FilePath.java:1176)
          at hudson.plugins.ws_cleanup.Wipeout$DisposableImpl.dispose(Wipeout.java:110)
          at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer$WorkItem.run(AsyncResourceDisposer.java:254)
          at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
          Caused by: java.nio.file.DirectoryNotEmptyException: /opt/jenkins/workspace/myjob-Solaris_ws-cleanup_1478635926754/local/.git/refs/remotes/origin
          at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
          at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
          at java.nio.file.Files.deleteIfExists(Files.java:1118)
          at hudson.Util.tryOnceDeleteFile(Util.java:287)
          at hudson.Util.deleteFile(Util.java:243)
          at hudson.FilePath.deleteRecursive(FilePath.java:1209)
          at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
          at hudson.FilePath.deleteRecursive(FilePath.java:1200)
          at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
          at hudson.FilePath.deleteRecursive(FilePath.java:1200)
          at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
          at hudson.FilePath.deleteRecursive(FilePath.java:1200)
          at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
          at hudson.FilePath.deleteRecursive(FilePath.java:1200)
          at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
          at hudson.FilePath.deleteRecursive(FilePath.java:1200)
          at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
          at hudson.FilePath.deleteRecursive(FilePath.java:1200)
          at hudson.FilePath.access$1000(FilePath.java:195)
          at hudson.FilePath$14.invoke(FilePath.java:1179)
          at hudson.FilePath$14.invoke(FilePath.java:1176)
          at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2731)
          at hudson.remoting.UserRequest.perform(UserRequest.java:153)
          at hudson.remoting.UserRequest.perform(UserRequest.java:50)
          at hudson.remoting.Request$2.run(Request.java:332)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
          at java.util.concurrent.FutureTask.run(FutureTask.java:262)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          ... 1 more
          java.io.IOException: Unable to delete '/opt/jenkins/workspace/myjob-Solaris_ws-cleanup_1478635926754/local/.git/refs/remotes/origin'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.

          aflat added a comment - I'm still hitting this on a solaris x86 machine. Seems to work on other OS's java.io.IOException: Unable to delete '/opt/jenkins/workspace/myjob-Solaris_ws-cleanup_1478635926754/local/.git/refs/remotes/origin'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. at hudson.Util.deleteFile(Util.java:248) at hudson.FilePath.deleteRecursive(FilePath.java:1209) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.access$1000(FilePath.java:195) at hudson.FilePath$14.invoke(FilePath.java:1179) at hudson.FilePath$14.invoke(FilePath.java:1176) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2731) at hudson.remoting.UserRequest.perform(UserRequest.java:153) at hudson.remoting.UserRequest.perform(UserRequest.java:50) at hudson.remoting.Request$2.run(Request.java:332) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) at ......remote call to sun04(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1435) at hudson.remoting.UserResponse.retrieve(UserRequest.java:253) at hudson.remoting.Channel.call(Channel.java:795) at hudson.FilePath.act(FilePath.java:985) at hudson.FilePath.act(FilePath.java:974) at hudson.FilePath.deleteRecursive(FilePath.java:1176) at hudson.plugins.ws_cleanup.Wipeout$DisposableImpl.dispose(Wipeout.java:110) at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer$WorkItem.run(AsyncResourceDisposer.java:254) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.nio.file.DirectoryNotEmptyException: /opt/jenkins/workspace/myjob-Solaris_ws-cleanup_1478635926754/local/.git/refs/remotes/origin at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242) at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108) at java.nio.file.Files.deleteIfExists(Files.java:1118) at hudson.Util.tryOnceDeleteFile(Util.java:287) at hudson.Util.deleteFile(Util.java:243) at hudson.FilePath.deleteRecursive(FilePath.java:1209) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.access$1000(FilePath.java:195) at hudson.FilePath$14.invoke(FilePath.java:1179) at hudson.FilePath$14.invoke(FilePath.java:1176) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2731) at hudson.remoting.UserRequest.perform(UserRequest.java:153) at hudson.remoting.UserRequest.perform(UserRequest.java:50) at hudson.remoting.Request$2.run(Request.java:332) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ... 1 more java.io.IOException: Unable to delete '/opt/jenkins/workspace/myjob-Solaris_ws-cleanup_1478635926754/local/.git/refs/remotes/origin'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.

          Juan Facorro added a comment - - edited

          We are having the same issue on CentOS Linux release 7.2.1511 (Core). The plugin version is 0.32.

          Directories seem to be marked for cleanup but their owner instead of being jenkins is another user.

          Juan Facorro added a comment - - edited We are having the same issue on CentOS Linux release 7.2.1511 (Core). The plugin version is 0.32. Directories seem to be marked for cleanup but their owner instead of being jenkins is another user.

          Gregor Philp added a comment - - edited

          Hi, we have the same issue still on CentOS Linux 6 and 7 platforms. We'd like these to just be removed and not reported so that we have to manually remove them. We end up with 100s, 1000s of these.

          We have latest plugin versions:
          Workspace Cleanup Plugin - 0.32
          Resource Disposer Plugin - 0.3
          and running jenkins master - 2.19.4

          It seems this might be related to the problem we're having of the java heap being used up and then our master fails. I then have to restart master.

          thanks
          Gregor

          Gregor Philp added a comment - - edited Hi, we have the same issue still on CentOS Linux 6 and 7 platforms. We'd like these to just be removed and not reported so that we have to manually remove them. We end up with 100s, 1000s of these. We have latest plugin versions: Workspace Cleanup Plugin - 0.32 Resource Disposer Plugin - 0.3 and running jenkins master - 2.19.4 It seems this might be related to the problem we're having of the java heap being used up and then our master fails. I then have to restart master. thanks Gregor

          gjphilp, the screenshot demonstrates how it is supposed to work. Jenkins tries to delete it but as it fail repeatedly so item is tracked in resource disposer until the directory get fried. It seems to be never in your case. It is reported for you attention as there is something that prevents the directory to be deleted for a long time. The cause needs to be found and eliminated in your case. Hover over the exception message to see the full exception that might contain the clue.

          Oliver Gondža added a comment - gjphilp , the screenshot demonstrates how it is supposed to work. Jenkins tries to delete it but as it fail repeatedly so item is tracked in resource disposer until the directory get fried. It seems to be never in your case. It is reported for you attention as there is something that prevents the directory to be deleted for a long time. The cause needs to be found and eliminated in your case. Hover over the exception message to see the full exception that might contain the clue.

          I am still experiencing the problem, and in my case I noticed that the build process is creating soft links that are owned by root. The cleanup process is not able to delete these, so it tracks them in the resource disposer. I am working with the developer to find how those links are being created, but this is something that should be addressed in the plugin as well.

          Stephan Fenton added a comment - I am still experiencing the problem, and in my case I noticed that the build process is creating soft links that are owned by root. The cleanup process is not able to delete these, so it tracks them in the resource disposer. I am working with the developer to find how those links are being created, but this is something that should be addressed in the plugin as well.

          stephan_fenton, note that this has nothing to do with the plugin or async deletion. The build just creates stuff Jenkins agent has no permission to clean.

          The only thing I can think of to improve this is turn the async deletion off in case the project has a bad record on workspace deletion so such persistent problems will be presented to jobs owners eventually and not the admins.

          Oliver Gondža added a comment - stephan_fenton , note that this has nothing to do with the plugin or async deletion. The build just creates stuff Jenkins agent has no permission to clean. The only thing I can think of to improve this is turn the async deletion off in case the project has a bad record on workspace deletion so such persistent problems will be presented to jobs owners eventually and not the admins.

          Albert V added a comment -

          I'm experiencing the same issue within an updated Jenkins (2.53) and ws-cleanup plugin (0.32).
          I wanted to note that when I'm using the ws-cleanup into a pipeline, is working as expected. But when I'm using it into a regular Maven job it is leaving the famous directory with a timestamp (one per build!) and is filling the workspace of my slaves.

          I think that this issue should be re-opened because there are more cases like this one.

          Albert V added a comment - I'm experiencing the same issue within an updated Jenkins (2.53) and ws-cleanup plugin (0.32). I wanted to note that when I'm using the ws-cleanup into a pipeline, is working as expected. But when I'm using it into a regular Maven job it is leaving the famous directory with a timestamp (one per build!) and is filling the workspace of my slaves. I think that this issue should be re-opened because there are more cases like this one.

          I could replicate the issue if you disconnect the agent when the prebuild cleanup it is make, if you do that the temporal folder will never be deleted

          Ivan Fernandez Calvo added a comment - I could replicate the issue if you disconnect the agent when the prebuild cleanup it is make, if you do that the temporal folder will never be deleted

          brian hewson added a comment -

          gjphilp I had the same problem of having to manually remove hundreds of reports.

          curl -s http://${JENKINS_URL}/administrativeMonitor/AsyncResourceDisposer/ -u${UTILITY_USER}:${UTILITY_PW} | tr '"' '\n' | grep 'stop-tracking' | cut -d '-' -f 3 | sort -n | while read ASYNC_THREAD; do curl http://${JENKINS_URL}/administrativeMonitor/AsyncResourceDisposer/stopTracking -u${UTILITY_USER}:${UTILITY_PW} -X POST --data "id=${ASYNC_THREAD}"; done

          Heap usage dropped significantly immediately after cleanup of 337 threads. 

          brian hewson added a comment - gjphilp I had the same problem of having to manually remove hundreds of reports. curl -s http: //${JENKINS_URL}/administrativeMonitor/AsyncResourceDisposer/ -u${UTILITY_USER}:${UTILITY_PW} | tr ' "' '\n' | grep 'stop-tracking' | cut -d '-' -f 3 | sort -n | while read ASYNC_THREAD; do curl http://${JENKINS_URL}/administrativeMonitor/AsyncResourceDisposer/stopTracking -u${UTILITY_USER}:${UTILITY_PW} -X POST --data " id=${ASYNC_THREAD}"; done Heap usage dropped significantly immediately after cleanup of 337 threads. 

          Cesar Ibarra added a comment -

          Can anyone explain me how they resolve this?
          I am using Jenkins 2.107.2 with ws-cleanup 0.34 and I am also experience the issue of a lingering ws-cleanup folder with the timestamp which didn't get deleted.
          I am running Jenkins on a OS Ubuntu 16.04.

          Any help on how to solve this?

          Cesar Ibarra added a comment - Can anyone explain me how they resolve this? I am using Jenkins 2.107.2 with ws-cleanup 0.34 and I am also experience the issue of a lingering ws-cleanup folder with the timestamp which didn't get deleted. I am running Jenkins on a OS Ubuntu 16.04. Any help on how to solve this?

          Guys, this ticket was resolved year and a half ago. Please, file a separate issue.

          Oliver Gondža added a comment - Guys, this ticket was resolved year and a half ago. Please, file a separate issue.

          But the problem still persists (here too, BTW.). So why file a new issue for the VERY SAME problem? Reopening a ticket is common practice in such cases.

          Dirk Heinrichs added a comment - But the problem still persists (here too, BTW.). So why file a new issue for the VERY SAME problem? Reopening a ticket is common practice in such cases.

          We also have this issue but only on Windows slaves (connected through Cygwin SSH).

          Adam Brousseau added a comment - We also have this issue but only on Windows slaves (connected through Cygwin SSH).

          Have the same issue. Jenkins 2.121.1 in docker container(workspace on volume) , ws-cleanup 0.34

          Denis Zakharov added a comment - Have the same issue. Jenkins 2.121.1 in docker container(workspace on volume) , ws-cleanup 0.34

          Guys, this ticket was resolved year and a half ago. Please, file a separate issue.

          Oliver Gondža added a comment - Guys, this ticket was resolved year and a half ago. Please, file a separate issue.

          Opened https://issues.jenkins-ci.org/browse/JENKINS-53579 now can we get this addressed?

          Jakub Bochenski added a comment - Opened https://issues.jenkins-ci.org/browse/JENKINS-53579 now can we get this addressed?

            olivergondza Oliver Gondža
            tmoore Tom Moore
            Votes:
            18 Vote for this issue
            Watchers:
            40 Start watching this issue

              Created:
              Updated:
              Resolved: