Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-24824

Asynchronous cleanup not removing renamed workspace directories on slaves

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • ws-cleanup-plugin
    • None
    • Jenkins 1.579
      ws-cleanup 0.24

    Description

      After upgrading to ws-cleanup 0.24 in order to get the asynchronous cleanup function we noticed the workspaces on our slaves getting renamed to the form of ${WORKSPACE}ws-cleanup${TIMESTAMP}. (ie, job1 would become job1_ws-cleanup_1411197183394). The expected behavior under ws-cleanup 0.24 is that these were temporary to support asynchronous processing and would be deleted. However, these directories never get removed from the slave. Over time, the slave hard drives filled up resulting in build failures.

      Attachments

        Issue Links

          Activity

            vjuranek vjuranek added a comment -

            Hi, any idea how to reproduce it? Everything works fine on my machine. Do you use any "Advanced" options? If so, which one? What is the OS of the slave where ws is not deleted?
            Thanks

            vjuranek vjuranek added a comment - Hi, any idea how to reproduce it? Everything works fine on my machine. Do you use any "Advanced" options? If so, which one? What is the OS of the slave where ws is not deleted? Thanks
            tmoore Tom Moore added a comment -

            The slaves are running on a VM that is running Windows 7 Enterprise, Service Pack 1. The slaves start their Jenkins connection via slave command line. Running as a Windows service is not an option as the build process must be subordinate to an active login session due to compiler restrictions. No advanced options were set.

            tmoore Tom Moore added a comment - The slaves are running on a VM that is running Windows 7 Enterprise, Service Pack 1. The slaves start their Jenkins connection via slave command line. Running as a Windows service is not an option as the build process must be subordinate to an active login session due to compiler restrictions. No advanced options were set.

            We are experiencing the same problem. In our case the renamed workspaces are not removed when we use Multiple SCMs to checkout more than one repository from git. The unremoved repositories are placed in a sub-directory.

            katrine Katrine Skovbo added a comment - We are experiencing the same problem. In our case the renamed workspaces are not removed when we use Multiple SCMs to checkout more than one repository from git. The unremoved repositories are placed in a sub-directory.
            vjuranek vjuranek added a comment -

            Unfortunately I hasn't been able reproduce it yet Any reproducer would definitely help. Will try with MultipleSCM and several git repos.

            vjuranek vjuranek added a comment - Unfortunately I hasn't been able reproduce it yet Any reproducer would definitely help. Will try with MultipleSCM and several git repos.
            sebok Marton Sebok added a comment -

            Hi, I have the same issue. I think one key point is that my slave is configured to go offline after being idle for a while, and I want to delete pretty much files from a complex directory tree. I can see that some folders have been deleted from the workspace, but heavier ones are left there.

            sebok Marton Sebok added a comment - Hi, I have the same issue. I think one key point is that my slave is configured to go offline after being idle for a while, and I want to delete pretty much files from a complex directory tree. I can see that some folders have been deleted from the workspace, but heavier ones are left there.
            tmoore Tom Moore added a comment -

            perhaps workspace size is a factor? The workspaces we saw this on come in at just over 6 GB. (Which is why we wanted to be able to use this feature in the first place)

            tmoore Tom Moore added a comment - perhaps workspace size is a factor? The workspaces we saw this on come in at just over 6 GB. (Which is why we wanted to be able to use this feature in the first place)
            vjuranek vjuranek added a comment -

            Sorry, cannot reproduce even with MutilpeSCM plugin (Jenkins 1.574, MutipleSCM 0.3, Git 2.2.7, ws clenaup 0.24).
            @Marton Sebok haven't tried it yet, but this seem to be a valid concern. Will investigate it deeper. Thanks for a pointer!

            vjuranek vjuranek added a comment - Sorry, cannot reproduce even with MutilpeSCM plugin (Jenkins 1.574, MutipleSCM 0.3, Git 2.2.7, ws clenaup 0.24). @Marton Sebok haven't tried it yet, but this seem to be a valid concern. Will investigate it deeper. Thanks for a pointer!
            vjuranek vjuranek added a comment -

            @Tom Moore and do you also put slaves offline as pointer out by Marton Sebok? In this case it does make sense for me, but otherwise size of the workspace shouldn't IMHO matter. Do you see any errors in the logs?

            vjuranek vjuranek added a comment - @Tom Moore and do you also put slaves offline as pointer out by Marton Sebok? In this case it does make sense for me, but otherwise size of the workspace shouldn't IMHO matter. Do you see any errors in the logs?
            tmoore Tom Moore added a comment -

            No, we don't put the slaves offline. We didn't notice any errors in the logs, the first indication we had that this was a problem was when we ran out of disk space.

            tmoore Tom Moore added a comment - No, we don't put the slaves offline. We didn't notice any errors in the logs, the first indication we had that this was a problem was when we ran out of disk space.

            It seems that the files we have trouble with are the ones that contain Danish characters æ,ø or å in the file name.

            katrine Katrine Skovbo added a comment - It seems that the files we have trouble with are the ones that contain Danish characters æ,ø or å in the file name.
            danielbeck Daniel Beck added a comment -

            katrine: Are these characters in the workspace (project) name, a folder name in the workspace, or a file name in the workspace? Are you deleting the entire workspace? Did it work on the older Jenkins version? What are the contents of the /systemInfo URL (or /computer/slavename/systemInfo URL for workspaces on slaves)?

            danielbeck Daniel Beck added a comment - katrine : Are these characters in the workspace (project) name, a folder name in the workspace, or a file name in the workspace? Are you deleting the entire workspace? Did it work on the older Jenkins version? What are the contents of the /systemInfo URL (or /computer/slavename/systemInfo URL for workspaces on slaves)?

            The characters are in file names in the workspace.
            We delete the entire workspace.
            It worked before the upgrade to version 0.24, but I tried downgrading to 0.23 again and that also produced errors now. In 0.23 it now explicitly says that it cannot delete the files:

            ERROR: Failed to clean the workspace
            09:05:02 java.io.IOException: java.lang.reflect.InvocationTargetException
            09:05:02 	at hudson.Util.isSymlinkJava7(Util.java:360)
            09:05:02 	at hudson.Util.isSymlink(Util.java:325)
            09:05:02 	at hudson.Util.deleteRecursive(Util.java:291)
            09:05:02 	at hudson.Util.deleteContentsRecursive(Util.java:203)
            09:05:02 	at hudson.Util.deleteRecursive(Util.java:292)
            09:05:02 	at hudson.Util.deleteContentsRecursive(Util.java:203)
            09:05:02 	at hudson.Util.deleteRecursive(Util.java:292)
            09:05:02 	at hudson.Util.deleteContentsRecursive(Util.java:203)
            09:05:02 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:393)
            09:05:02 	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
            09:05:02 	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
            09:05:02 	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
            09:05:02 	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            09:05:02 	at hudson.remoting.Request$2.run(Request.java:328)
            09:05:02 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
            09:05:02 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            09:05:02 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            09:05:02 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            09:05:02 	at java.lang.Thread.run(Thread.java:745)
            09:05:02 Caused by: java.lang.reflect.InvocationTargetException
            09:05:02 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            09:05:02 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            09:05:02 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            09:05:02 	at java.lang.reflect.Method.invoke(Method.java:606)
            09:05:02 	at hudson.Util.isSymlinkJava7(Util.java:355)
            09:05:02 	... 18 more
            09:05:02 Caused by: java.nio.file.InvalidPathException: Malformed input or input contains unmappable chacraters: /var/lib/jenkins/workspace/My-project/sub-folder/prototype/icons/skylleanl??g.svg
            09:05:02 	at sun.nio.fs.UnixPath.encode(UnixPath.java:147)
            09:05:02 	at sun.nio.fs.UnixPath.<init>(UnixPath.java:71)
            09:05:02 	at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:281)
            09:05:02 	at java.io.File.toPath(File.java:2186)
            09:05:02 	... 23 more
            

            We are using Jenkins 1.565.3 and a Unix slave, version 2.46, java 1.7.0_72

            katrine Katrine Skovbo added a comment - The characters are in file names in the workspace. We delete the entire workspace. It worked before the upgrade to version 0.24, but I tried downgrading to 0.23 again and that also produced errors now. In 0.23 it now explicitly says that it cannot delete the files: ERROR: Failed to clean the workspace 09:05:02 java.io.IOException: java.lang.reflect.InvocationTargetException 09:05:02 at hudson.Util.isSymlinkJava7(Util.java:360) 09:05:02 at hudson.Util.isSymlink(Util.java:325) 09:05:02 at hudson.Util.deleteRecursive(Util.java:291) 09:05:02 at hudson.Util.deleteContentsRecursive(Util.java:203) 09:05:02 at hudson.Util.deleteRecursive(Util.java:292) 09:05:02 at hudson.Util.deleteContentsRecursive(Util.java:203) 09:05:02 at hudson.Util.deleteRecursive(Util.java:292) 09:05:02 at hudson.Util.deleteContentsRecursive(Util.java:203) 09:05:02 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:393) 09:05:02 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153) 09:05:02 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146) 09:05:02 at hudson.remoting.UserRequest.perform(UserRequest.java:118) 09:05:02 at hudson.remoting.UserRequest.perform(UserRequest.java:48) 09:05:02 at hudson.remoting.Request$2.run(Request.java:328) 09:05:02 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) 09:05:02 at java.util.concurrent.FutureTask.run(FutureTask.java:262) 09:05:02 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 09:05:02 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 09:05:02 at java.lang. Thread .run( Thread .java:745) 09:05:02 Caused by: java.lang.reflect.InvocationTargetException 09:05:02 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 09:05:02 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 09:05:02 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 09:05:02 at java.lang.reflect.Method.invoke(Method.java:606) 09:05:02 at hudson.Util.isSymlinkJava7(Util.java:355) 09:05:02 ... 18 more 09:05:02 Caused by: java.nio.file.InvalidPathException: Malformed input or input contains unmappable chacraters: / var /lib/jenkins/workspace/My-project/sub-folder/prototype/icons/skylleanl??g.svg 09:05:02 at sun.nio.fs.UnixPath.encode(UnixPath.java:147) 09:05:02 at sun.nio.fs.UnixPath.<init>(UnixPath.java:71) 09:05:02 at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:281) 09:05:02 at java.io.File.toPath(File.java:2186) 09:05:02 ... 23 more We are using Jenkins 1.565.3 and a Unix slave, version 2.46, java 1.7.0_72
            vjuranek vjuranek added a comment -

            katrine: this seems to be unrelated to the bug discussed in this issue. This is probably different bug and this error is just hidden by async delete. I filed a new issue for this: JENKINS-25317

            vjuranek vjuranek added a comment - katrine : this seems to be unrelated to the bug discussed in this issue. This is probably different bug and this error is just hidden by async delete. I filed a new issue for this: JENKINS-25317
            olivergondza Oliver Gondža added a comment - - edited

            I expect the deletion can fail for some reason even after we successfully renamed the directory. I see several ways to improve current situation:

            • Retry the deletion until it succeeds - it should not block the build but can as well run for unpredictable amount of time
            • Introduce a periodic task to clean up such dangling directories
            • Pass Future from FilePath.actAsync to some "collector" task that would retry and/or report in case some deletion fails.*

            [*] I guess this might be the only way to find out why it failed on the first place as it seems that FilePath.actAsync does not log exceptions.

            olivergondza Oliver Gondža added a comment - - edited I expect the deletion can fail for some reason even after we successfully renamed the directory. I see several ways to improve current situation: Retry the deletion until it succeeds - it should not block the build but can as well run for unpredictable amount of time Introduce a periodic task to clean up such dangling directories Pass Future from FilePath.actAsync to some "collector" task that would retry and/or report in case some deletion fails.* [*] I guess this might be the only way to find out why it failed on the first place as it seems that FilePath.actAsync does not log exceptions.
            vivo Victor Volle added a comment -

            We added some logging around the deletion (see below) and found that there was a permissions problem.

            I would suggest to apply the patch to the plugin so that others can see the root cause as well

            From 4dd5c02d5c65a30860ed4ccc2baa860f331f156f Mon Sep 17 00:00:00 2001
            From: Victor Volle <vivo@Victors-MacBook-Pro.local>
            Date: Tue, 18 Aug 2015 17:53:41 +0200
            Subject: [PATCH] JENKINS-24824: log error
            
            ---
             src/main/java/hudson/plugins/ws_cleanup/Wipeout.java | 7 ++++++-
             1 file changed, 6 insertions(+), 1 deletion(-)
            
            diff --git a/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java b/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java
            index e1e759b..a9e50a6 100644
            --- a/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java
            +++ b/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java
            @@ -69,7 +69,12 @@ import hudson.remoting.VirtualChannel;
                 private final static Command COMMAND = new Command();
                 private final static class Command implements FileCallable<Object> {
                     public Object invoke(File f, VirtualChannel channel) throws IOException, InterruptedException {
            -            Util.deleteRecursive(f);
            +            try {
            +                Util.deleteRecursive(f);
            +            } catch (IOException e) {
            +                LOGGER.log(Level.WARNING, "error cleaning up", e);
            +                throw e;
            +            }
                         return null;
                     }
                 }
            -- 
            2.3.2 (Apple Git-55)
            
            vivo Victor Volle added a comment - We added some logging around the deletion (see below) and found that there was a permissions problem. I would suggest to apply the patch to the plugin so that others can see the root cause as well From 4dd5c02d5c65a30860ed4ccc2baa860f331f156f Mon Sep 17 00:00:00 2001 From: Victor Volle <vivo@Victors-MacBook-Pro.local> Date: Tue, 18 Aug 2015 17:53:41 +0200 Subject: [PATCH] JENKINS-24824: log error --- src/main/java/hudson/plugins/ws_cleanup/Wipeout.java | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java b/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java index e1e759b..a9e50a6 100644 --- a/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java +++ b/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java @@ -69,7 +69,12 @@ import hudson.remoting.VirtualChannel; private final static Command COMMAND = new Command(); private final static class Command implements FileCallable< Object > { public Object invoke(File f, VirtualChannel channel) throws IOException, InterruptedException { - Util.deleteRecursive(f); + try { + Util.deleteRecursive(f); + } catch (IOException e) { + LOGGER.log(Level.WARNING, "error cleaning up" , e); + throw e; + } return null ; } } -- 2.3.2 (Apple Git-55)

            Code changed in jenkins
            User: Oliver Gondža
            Path:
            src/main/java/hudson/plugins/ws_cleanup/Wipeout.java
            http://jenkins-ci.org/commit/ws-cleanup-plugin/e151b89222f93b8b125f3f5e1dffb51e6ad8be9d
            Log:
            JENKINS-24824 Log deletion failure

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: src/main/java/hudson/plugins/ws_cleanup/Wipeout.java http://jenkins-ci.org/commit/ws-cleanup-plugin/e151b89222f93b8b125f3f5e1dffb51e6ad8be9d Log: JENKINS-24824 Log deletion failure

            Code changed in jenkins
            User: Oliver Gondža
            Path:
            src/main/java/hudson/plugins/ws_cleanup/Wipeout.java
            http://jenkins-ci.org/commit/ws-cleanup-plugin/32766a78cf81399fcd0b46f68600863c0369a869
            Log:
            Merge pull request #22 from olivergondza/JENKINS-24824

            JENKINS-24824 Log deletion failure

            Compare: https://github.com/jenkinsci/ws-cleanup-plugin/compare/6bc314398d0f...32766a78cf81

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: src/main/java/hudson/plugins/ws_cleanup/Wipeout.java http://jenkins-ci.org/commit/ws-cleanup-plugin/32766a78cf81399fcd0b46f68600863c0369a869 Log: Merge pull request #22 from olivergondza/ JENKINS-24824 JENKINS-24824 Log deletion failure Compare: https://github.com/jenkinsci/ws-cleanup-plugin/compare/6bc314398d0f...32766a78cf81

            Same issue here.

            We have one Linux master running multiple slaves, some windows, some Linux.

            Linux slaves (VMWare VM) get a revert to snapshot using vmware plugin, which is not possible for windows slaves in our current environment.

            To handle hanging processes and locked files, we disconnect each windows slave after 1 single build, shutdown and power on the vm.

            Our problem are ever growing workspace folders, allocating far more than 80gig of disk space within about two weeks, as the async cleanup can not run and remove old workspaces due to premature disconnect and termination of the java vm.

            In our case the workspace cleanup plugin can rename old workspaces successfully (* -> *ws-cleanup[1234567890]+) , but does not delete those old folders after successfully reconnecting the windows slaves.

            We get no errors logged using ws-cleanup plugin version 0.28 in our build logs.

            tcb_xy Tim-Christian Bloss added a comment - Same issue here. We have one Linux master running multiple slaves, some windows, some Linux. Linux slaves (VMWare VM) get a revert to snapshot using vmware plugin, which is not possible for windows slaves in our current environment. To handle hanging processes and locked files, we disconnect each windows slave after 1 single build, shutdown and power on the vm. Our problem are ever growing workspace folders, allocating far more than 80gig of disk space within about two weeks, as the async cleanup can not run and remove old workspaces due to premature disconnect and termination of the java vm. In our case the workspace cleanup plugin can rename old workspaces successfully (* -> * ws-cleanup [1234567890] +) , but does not delete those old folders after successfully reconnecting the windows slaves. We get no errors logged using ws-cleanup plugin version 0.28 in our build logs.
            skarlso Gergely Brautigam added a comment - - edited

            Hi.

            Same issue here. Looks like the async delete isn't happening fast enough or isn't happening at all. Most of the time, when our git ran, but the job failed early, the async cleanup did not have time to finish deleting a folder.

            Or the other problem is usually an access denied error. Note, not a permission, but access. Meaning, the directory was still being used by some loose thread of a process.

            For now, I'm inclined to create a script on the slaves and put it into cron job which runs and cleans up every folder with ws-cleanup in it. But that is not a really nice solution.

            EDIT: And yes, same here, usually there is no error involved. Just a [WS-CLEANUP] Deleting project workspace... and no 'Done' in the logs anywhere.

            skarlso Gergely Brautigam added a comment - - edited Hi. Same issue here. Looks like the async delete isn't happening fast enough or isn't happening at all. Most of the time, when our git ran, but the job failed early, the async cleanup did not have time to finish deleting a folder. Or the other problem is usually an access denied error. Note, not a permission, but access. Meaning, the directory was still being used by some loose thread of a process. For now, I'm inclined to create a script on the slaves and put it into cron job which runs and cleans up every folder with ws-cleanup in it. But that is not a really nice solution. EDIT : And yes, same here, usually there is no error involved. Just a [WS-CLEANUP] Deleting project workspace... and no 'Done' in the logs anywhere.
            olivergondza Oliver Gondža added a comment - - edited

            tcb_xy Your use-case is quite special and was not taken into account originally.

            skarlso

            Looks like the async delete isn't happening fast enough or isn't happening at all. Most of the time, when our git ran, but the job failed early, the async cleanup did not have time to finish deleting a folder.

            I do not understand this. How does the problem manifests?

            Note the async deletion failure cuses should not go into a build log (as is runs in parallel with the build and can even run for a longer time than the build itself) but a java logger. Check Jenkins master log or configure custom logged in UI for hudson.plugins.ws_cleanup to see why it fails.

            In the meantime, JENKINS-27648 was rejected in jenkins core so I guess we have to implement a periodic task to clean dangling *ws-cleanup* dirs. That should resolve the problems with slaves disconnected unexpectedly and directories not deleted in time because of opened file descriptors.

            olivergondza Oliver Gondža added a comment - - edited tcb_xy Your use-case is quite special and was not taken into account originally. skarlso Looks like the async delete isn't happening fast enough or isn't happening at all. Most of the time, when our git ran, but the job failed early, the async cleanup did not have time to finish deleting a folder. I do not understand this. How does the problem manifests? Note the async deletion failure cuses should not go into a build log (as is runs in parallel with the build and can even run for a longer time than the build itself) but a java logger. Check Jenkins master log or configure custom logged in UI for hudson.plugins.ws_cleanup to see why it fails. In the meantime, JENKINS-27648 was rejected in jenkins core so I guess we have to implement a periodic task to clean dangling *ws-cleanup* dirs. That should resolve the problems with slaves disconnected unexpectedly and directories not deleted in time because of opened file descriptors.

            The symptom is a lingering ws-cleanup folder with the timestamp which didn't get deleted. Btw, most of the times it manifests on windows only. Rarely happens on a linux slave interestingly enough.

            Yes, sorry, the access denied error message was in the Jenkins Log.

            However, the Done, part is in the job's log. And that one was obviously missing because of the exception in the main log.

            skarlso Gergely Brautigam added a comment - The symptom is a lingering ws-cleanup folder with the timestamp which didn't get deleted. Btw, most of the times it manifests on windows only. Rarely happens on a linux slave interestingly enough. Yes, sorry, the access denied error message was in the Jenkins Log. However, the Done, part is in the job's log. And that one was obviously missing because of the exception in the main log.
            danielbeck Daniel Beck added a comment -

            interestingly enough

            Windows file locking?

            danielbeck Daniel Beck added a comment - interestingly enough Windows file locking?

            Windows file locking?

            Lacking enough evidence to say yay, or nay. It is 'a' possibility.

            Or even some other process locking a certain file. I even saw git lingering and the .git folder could not be deleted in a RENAMED folder. So much crazy, such wow.

            skarlso Gergely Brautigam added a comment - Windows file locking? Lacking enough evidence to say yay, or nay. It is 'a' possibility. Or even some other process locking a certain file. I even saw git lingering and the .git folder could not be deleted in a RENAMED folder. So much crazy, such wow.
            danielbeck Daniel Beck added a comment -

            some other process locking a certain file. I even saw git lingering

            Sorry, that's what I meant. Too much locking going on and Windows doesn't allow deleting folders with open handles.

            danielbeck Daniel Beck added a comment - some other process locking a certain file. I even saw git lingering Sorry, that's what I meant. Too much locking going on and Windows doesn't allow deleting folders with open handles.

            Affirmative. When it happens, it's usually because of windows file locking.

            skarlso Gergely Brautigam added a comment - Affirmative. When it happens, it's usually because of windows file locking.
            lunarfs Frank Vissing added a comment - - edited

            I have a reproduce that can trigger this.. not the same as the classic slave setup but still.
            Provision a docker slave that will be provisioned with the 'One retention strategy'
            have a job that executes in the docker container (my job has less than 100 mb data)
            perform ws clean-up on success, fail etc.
            now once the job completes the docker container is killed and the wc-cleanup_timestamp folder remains
            this job runs on linux, so no relation to windows here
            one option could be a checkbox to disable assync delete
            the /home/jenkins/workspace folder is mounted in the docker container using data volumes, therfore the data remains on the filesystem until we destroy the data volumen

            The workaround for this is not to delete the workspace after building but prior to building.

            lunarfs Frank Vissing added a comment - - edited I have a reproduce that can trigger this.. not the same as the classic slave setup but still. Provision a docker slave that will be provisioned with the 'One retention strategy' have a job that executes in the docker container (my job has less than 100 mb data) perform ws clean-up on success, fail etc. now once the job completes the docker container is killed and the wc-cleanup_timestamp folder remains this job runs on linux, so no relation to windows here one option could be a checkbox to disable assync delete the /home/jenkins/workspace folder is mounted in the docker container using data volumes, therfore the data remains on the filesystem until we destroy the data volumen The workaround for this is not to delete the workspace after building but prior to building.
            slsh Jarkko Rantavuori added a comment - - edited

            We get this all the time - for out test jobs, there are hundreds of cleanup folders appearing on our disk. This is Ubuntu 14.04.1 64-bit slave, so at least for us it is not related to windows file locking. Also, we have had "Delete workspace before build starts" selected, but it has not fixed the issue.

            Our Jenkins version is 1.632, cleanup plugin 0.26.

            Update: reverting to cleanup plugin 0.23, which didn't have asynch deletion, allowed us to see the error: we had a shell script section in the jobs which created a folder having root owner instead of jenkins, so the following runs of a job would fail since there weren't able to delete the previous workspace.

            I think what needs to be done for asynch plugin is to fail the build somehow if the deletion fails and report the error. Also, it would be nice if user could select between asynch and synch delete instead of having to downgrade all the way to 0.23.

            slsh Jarkko Rantavuori added a comment - - edited We get this all the time - for out test jobs, there are hundreds of cleanup folders appearing on our disk. This is Ubuntu 14.04.1 64-bit slave, so at least for us it is not related to windows file locking. Also, we have had "Delete workspace before build starts" selected, but it has not fixed the issue. Our Jenkins version is 1.632, cleanup plugin 0.26. Update: reverting to cleanup plugin 0.23, which didn't have asynch deletion, allowed us to see the error: we had a shell script section in the jobs which created a folder having root owner instead of jenkins, so the following runs of a job would fail since there weren't able to delete the previous workspace. I think what needs to be done for asynch plugin is to fail the build somehow if the deletion fails and report the error. Also, it would be nice if user could select between asynch and synch delete instead of having to downgrade all the way to 0.23.
            jdoose Jens Doose added a comment - - edited

            I have the same behaviour in a non-slave environment, I don't know if this is the same or a new issue.
            A lot of directories like workspace_ws-cleanup_1447277714463 get created and never deleted.

            In the log of the jenkins job there is the output that everything is ok:
            [WS-CLEANUP] Deleting project workspace...
            [WS-CLEANUP] Done

            The job is building node projects and loading npm componentes as well as bower components which result in quite huge directory structures, might that be a problem?

            jdoose Jens Doose added a comment - - edited I have the same behaviour in a non-slave environment, I don't know if this is the same or a new issue. A lot of directories like workspace_ws-cleanup_1447277714463 get created and never deleted. In the log of the jenkins job there is the output that everything is ok: [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done The job is building node projects and loading npm componentes as well as bower components which result in quite huge directory structures, might that be a problem?

            In fact if you want to switch "asynch/synch" mode, you can add Patterns, because this will affect cleanup method :

            • Add Patterns : synch mode
            • No Patterns : asynch mode

            This is just a trick to see the cleanup problem without downgrade to 0.23.

            tcollignon Thomas Collignon added a comment - In fact if you want to switch "asynch/synch" mode, you can add Patterns, because this will affect cleanup method : Add Patterns : synch mode No Patterns : asynch mode This is just a trick to see the cleanup problem without downgrade to 0.23.

            And I have submit a pull request to add a new option to switch between synch and asynch mode without downgrading : https://github.com/jenkinsci/ws-cleanup-plugin/pull/26

            tcollignon Thomas Collignon added a comment - And I have submit a pull request to add a new option to switch between synch and asynch mode without downgrading : https://github.com/jenkinsci/ws-cleanup-plugin/pull/26

            tcollignon, thanks for the workaround!

            @All, please check the logged (hudson.plugins.ws_cleanup.Wipeout) cause in slave/master log and attach it to this issue.

            olivergondza Oliver Gondža added a comment - tcollignon , thanks for the workaround! @All, please check the logged ( hudson.plugins.ws_cleanup.Wipeout ) cause in slave/master log and attach it to this issue.

            It seems that people generally prefer to fail the build in case the cleanup fails rather that clutter the slave's workspace in the long run. This does not get well together with asynchronous deletion requirement.

            One way I can think of is always use sync approach from post-build step and async only in pre-build step. If the original (renamed) workspace will not be gone by the end of the build, sych deletion will be reattempted and will have a chance to fail the build in case of problems. The advantage is that build can start right away and most of the post build actions will do its job (publish junit, determine result, etc.) before the final cleanup kicks in.

            WDYT?

            olivergondza Oliver Gondža added a comment - It seems that people generally prefer to fail the build in case the cleanup fails rather that clutter the slave's workspace in the long run. This does not get well together with asynchronous deletion requirement. One way I can think of is always use sync approach from post-build step and async only in pre-build step. If the original (renamed) workspace will not be gone by the end of the build, sych deletion will be reattempted and will have a chance to fail the build in case of problems. The advantage is that build can start right away and most of the post build actions will do its job (publish junit, determine result, etc.) before the final cleanup kicks in. WDYT?

            This may be a good thing, why not
            I ask myself if it's possible when asynch mode is on post-build, to get error et bring back to jenkins job trace. In this case the boolean "asynchronously" is necessary I Think to let people chosing the strategy.

            tcollignon Thomas Collignon added a comment - This may be a good thing, why not I ask myself if it's possible when asynch mode is on post-build, to get error et bring back to jenkins job trace. In this case the boolean "asynchronously" is necessary I Think to let people chosing the strategy.

            At a certain point Jenkins considers the build log to be complete and never amended again. The problem is that in theory, the cleanup can take longer than the build itself. We can either not care about the result at all (current implementation), wait for the result at the end of the build (potentially postponing the build completion) or try a compromise - check the status at the end of the build and report failure if there is any but do not wait for the completion (this can not guarantee there will be no dangling workspace directories).

            When I think about this further, once we rename the workspace (as we do for async cleanup) it will always require manual cleanup in case of failure as it will not be any build's workspace any longer. When the cleanup fail with sync approach, it leaves the workspace half deleted to be reused by future builds - which is far from optimal too.

            I am working on implementing plugin specific periodic task to cleanup all the temporary directories (since more general solution was rejected from core).

            olivergondza Oliver Gondža added a comment - At a certain point Jenkins considers the build log to be complete and never amended again. The problem is that in theory, the cleanup can take longer than the build itself. We can either not care about the result at all (current implementation), wait for the result at the end of the build (potentially postponing the build completion) or try a compromise - check the status at the end of the build and report failure if there is any but do not wait for the completion (this can not guarantee there will be no dangling workspace directories). When I think about this further, once we rename the workspace (as we do for async cleanup) it will always require manual cleanup in case of failure as it will not be any build's workspace any longer. When the cleanup fail with sync approach, it leaves the workspace half deleted to be reused by future builds - which is far from optimal too. I am working on implementing plugin specific periodic task to cleanup all the temporary directories (since more general solution was rejected from core).

            ok I see
            you work on another plugin to clean periodically or you think added this features in this one? you need some help?

            right now you think add option "asynchronously" is not necessary ?

            tcollignon Thomas Collignon added a comment - ok I see you work on another plugin to clean periodically or you think added this features in this one? you need some help? right now you think add option "asynchronously" is not necessary ?
            olivergondza Oliver Gondža added a comment - - edited

            My original idea was the async cleanup will be performance optimization that will be transparent to the user (whoever run the builds). Which it is only partially.

            I am extending this plugin with the periodic task. I understand the plugin becomes lot more sophisticated that we hoped it to be so I will implement a Jenkins-wide, property-based killswitch to get back to sync cleanup should it cause further problems.

            I still do not think that slave workspace getting full should be a concern of a user (as opposed to instance administrator), so I prefer this over a per-job configuration option.

            olivergondza Oliver Gondža added a comment - - edited My original idea was the async cleanup will be performance optimization that will be transparent to the user (whoever run the builds). Which it is only partially. I am extending this plugin with the periodic task. I understand the plugin becomes lot more sophisticated that we hoped it to be so I will implement a Jenkins-wide, property-based killswitch to get back to sync cleanup should it cause further problems. I still do not think that slave workspace getting full should be a concern of a user (as opposed to instance administrator), so I prefer this over a per-job configuration option.

            Ok I agree with you. So I wait for your new implementation

            Thank for your answer

            tcollignon Thomas Collignon added a comment - Ok I agree with you. So I wait for your new implementation Thank for your answer
            bimp Bim Paras added a comment -

            Has this issue been resolved? I am still seeing this issue in version 0.25 of plugin. Thanks.

            bimp Bim Paras added a comment - Has this issue been resolved? I am still seeing this issue in version 0.25 of plugin. Thanks.
            genericpenguin Sven Schott added a comment -

            Hi, just wanting to know if there is a timeframe on resolution of this issue. Would like to know just in case it's not in the near future so that I can set up a semi-temporary workaround for the problem (most likely a scheduled cleanup on our windows machines).

            genericpenguin Sven Schott added a comment - Hi, just wanting to know if there is a timeframe on resolution of this issue. Would like to know just in case it's not in the near future so that I can set up a semi-temporary workaround for the problem (most likely a scheduled cleanup on our windows machines).
            jwhitcraft Jon Whitcraft added a comment -

            +1 to a timeframe for fixing this

            I'm thinking of rolling back to 0.23 becuase of this problem.

            jwhitcraft Jon Whitcraft added a comment - +1 to a timeframe for fixing this I'm thinking of rolling back to 0.23 becuase of this problem.
            mdimmick_mnetics Mike Dimmick added a comment -

            I have been experiencing this problem on a Windows 7 Enterprise installation. During manual clean-up I noticed that even using the command line rmdir /s I was getting a lot of The directory is not empty errors, meaning I had to run the command twice to complete removal.

            I also noticed that a large amount of disk space was being consumed by Windows Search content indexing data. I disabled the Windows Search service and deleted the content indexes. Having done this, I'm no longer getting the errors from rmdir. I can also delete workspaces from the Jenkins 'Wipe Out Current Workspace' feature without problems, which was typically reporting an error previously.

            On this system, JENKINS_HOME is at C:\Users\jenkins\.jenkins. We also have an installation of Jenkins on Windows Server 2012 R2, which doesn't suffer from the problem - here, JENKINS_HOME is C:\Jenkins, and the Windows Search feature is not installed.

            mdimmick_mnetics Mike Dimmick added a comment - I have been experiencing this problem on a Windows 7 Enterprise installation. During manual clean-up I noticed that even using the command line rmdir /s I was getting a lot of The directory is not empty errors, meaning I had to run the command twice to complete removal. I also noticed that a large amount of disk space was being consumed by Windows Search content indexing data. I disabled the Windows Search service and deleted the content indexes. Having done this, I'm no longer getting the errors from rmdir . I can also delete workspaces from the Jenkins 'Wipe Out Current Workspace' feature without problems, which was typically reporting an error previously. On this system, JENKINS_HOME is at C:\Users\jenkins\.jenkins. We also have an installation of Jenkins on Windows Server 2012 R2, which doesn't suffer from the problem - here, JENKINS_HOME is C:\Jenkins, and the Windows Search feature is not installed.
            danielbeck Daniel Beck added a comment -

            mdimmick_mnetics

            Windows Search content indexing data. I disabled the Windows Search service

            Don't run the Windows Search on JENKINS_HOME. Don't run antivirus on JENKINS_HOME. No exceptions.

            danielbeck Daniel Beck added a comment - mdimmick_mnetics Windows Search content indexing data. I disabled the Windows Search service Don't run the Windows Search on JENKINS_HOME. Don't run antivirus on JENKINS_HOME. No exceptions.

            I am having a second look at this: https://github.com/jenkinsci/ws-cleanup-plugin/pull/28

            It implements a administrative monitor that retries the deletion on report failures in UI.

            olivergondza Oliver Gondža added a comment - I am having a second look at this: https://github.com/jenkinsci/ws-cleanup-plugin/pull/28 It implements a administrative monitor that retries the deletion on report failures in UI.

            Code changed in jenkins
            User: Oliver Gondža
            Path:
            pom.xml
            src/main/java/hudson/plugins/ws_cleanup/Wipeout.java
            src/test/java/hudson/plugins/ws_cleanup/CleanupPowermockTest.java
            src/test/java/hudson/plugins/ws_cleanup/CleanupTest.java
            http://jenkins-ci.org/commit/ws-cleanup-plugin/ccd907188e0489e76ca21a7513843864bab48c90
            Log:
            [FIXED JENKINS-24824] Collect all asynchronously deleted directories

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: pom.xml src/main/java/hudson/plugins/ws_cleanup/Wipeout.java src/test/java/hudson/plugins/ws_cleanup/CleanupPowermockTest.java src/test/java/hudson/plugins/ws_cleanup/CleanupTest.java http://jenkins-ci.org/commit/ws-cleanup-plugin/ccd907188e0489e76ca21a7513843864bab48c90 Log: [FIXED JENKINS-24824] Collect all asynchronously deleted directories

            This took lot longer than I expected. The fix will be in 0.32.

            olivergondza Oliver Gondža added a comment - This took lot longer than I expected. The fix will be in 0.32.
            aflat aflat added a comment -

            I'm still hitting this on a solaris x86 machine. Seems to work on other OS's

            java.io.IOException: Unable to delete '/opt/jenkins/workspace/myjob-Solaris_ws-cleanup_1478635926754/local/.git/refs/remotes/origin'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
            at hudson.Util.deleteFile(Util.java:248)
            at hudson.FilePath.deleteRecursive(FilePath.java:1209)
            at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
            at hudson.FilePath.deleteRecursive(FilePath.java:1200)
            at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
            at hudson.FilePath.deleteRecursive(FilePath.java:1200)
            at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
            at hudson.FilePath.deleteRecursive(FilePath.java:1200)
            at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
            at hudson.FilePath.deleteRecursive(FilePath.java:1200)
            at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
            at hudson.FilePath.deleteRecursive(FilePath.java:1200)
            at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
            at hudson.FilePath.deleteRecursive(FilePath.java:1200)
            at hudson.FilePath.access$1000(FilePath.java:195)
            at hudson.FilePath$14.invoke(FilePath.java:1179)
            at hudson.FilePath$14.invoke(FilePath.java:1176)
            at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2731)
            at hudson.remoting.UserRequest.perform(UserRequest.java:153)
            at hudson.remoting.UserRequest.perform(UserRequest.java:50)
            at hudson.remoting.Request$2.run(Request.java:332)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)
            at ......remote call to sun04(Native Method)
            at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1435)
            at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
            at hudson.remoting.Channel.call(Channel.java:795)
            at hudson.FilePath.act(FilePath.java:985)
            at hudson.FilePath.act(FilePath.java:974)
            at hudson.FilePath.deleteRecursive(FilePath.java:1176)
            at hudson.plugins.ws_cleanup.Wipeout$DisposableImpl.dispose(Wipeout.java:110)
            at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer$WorkItem.run(AsyncResourceDisposer.java:254)
            at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            at java.lang.Thread.run(Thread.java:745)
            Caused by: java.nio.file.DirectoryNotEmptyException: /opt/jenkins/workspace/myjob-Solaris_ws-cleanup_1478635926754/local/.git/refs/remotes/origin
            at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
            at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
            at java.nio.file.Files.deleteIfExists(Files.java:1118)
            at hudson.Util.tryOnceDeleteFile(Util.java:287)
            at hudson.Util.deleteFile(Util.java:243)
            at hudson.FilePath.deleteRecursive(FilePath.java:1209)
            at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
            at hudson.FilePath.deleteRecursive(FilePath.java:1200)
            at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
            at hudson.FilePath.deleteRecursive(FilePath.java:1200)
            at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
            at hudson.FilePath.deleteRecursive(FilePath.java:1200)
            at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
            at hudson.FilePath.deleteRecursive(FilePath.java:1200)
            at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
            at hudson.FilePath.deleteRecursive(FilePath.java:1200)
            at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
            at hudson.FilePath.deleteRecursive(FilePath.java:1200)
            at hudson.FilePath.access$1000(FilePath.java:195)
            at hudson.FilePath$14.invoke(FilePath.java:1179)
            at hudson.FilePath$14.invoke(FilePath.java:1176)
            at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2731)
            at hudson.remoting.UserRequest.perform(UserRequest.java:153)
            at hudson.remoting.UserRequest.perform(UserRequest.java:50)
            at hudson.remoting.Request$2.run(Request.java:332)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            ... 1 more
            java.io.IOException: Unable to delete '/opt/jenkins/workspace/myjob-Solaris_ws-cleanup_1478635926754/local/.git/refs/remotes/origin'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.

            aflat aflat added a comment - I'm still hitting this on a solaris x86 machine. Seems to work on other OS's java.io.IOException: Unable to delete '/opt/jenkins/workspace/myjob-Solaris_ws-cleanup_1478635926754/local/.git/refs/remotes/origin'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. at hudson.Util.deleteFile(Util.java:248) at hudson.FilePath.deleteRecursive(FilePath.java:1209) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.access$1000(FilePath.java:195) at hudson.FilePath$14.invoke(FilePath.java:1179) at hudson.FilePath$14.invoke(FilePath.java:1176) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2731) at hudson.remoting.UserRequest.perform(UserRequest.java:153) at hudson.remoting.UserRequest.perform(UserRequest.java:50) at hudson.remoting.Request$2.run(Request.java:332) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) at ......remote call to sun04(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1435) at hudson.remoting.UserResponse.retrieve(UserRequest.java:253) at hudson.remoting.Channel.call(Channel.java:795) at hudson.FilePath.act(FilePath.java:985) at hudson.FilePath.act(FilePath.java:974) at hudson.FilePath.deleteRecursive(FilePath.java:1176) at hudson.plugins.ws_cleanup.Wipeout$DisposableImpl.dispose(Wipeout.java:110) at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer$WorkItem.run(AsyncResourceDisposer.java:254) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.nio.file.DirectoryNotEmptyException: /opt/jenkins/workspace/myjob-Solaris_ws-cleanup_1478635926754/local/.git/refs/remotes/origin at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242) at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108) at java.nio.file.Files.deleteIfExists(Files.java:1118) at hudson.Util.tryOnceDeleteFile(Util.java:287) at hudson.Util.deleteFile(Util.java:243) at hudson.FilePath.deleteRecursive(FilePath.java:1209) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.access$1000(FilePath.java:195) at hudson.FilePath$14.invoke(FilePath.java:1179) at hudson.FilePath$14.invoke(FilePath.java:1176) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2731) at hudson.remoting.UserRequest.perform(UserRequest.java:153) at hudson.remoting.UserRequest.perform(UserRequest.java:50) at hudson.remoting.Request$2.run(Request.java:332) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ... 1 more java.io.IOException: Unable to delete '/opt/jenkins/workspace/myjob-Solaris_ws-cleanup_1478635926754/local/.git/refs/remotes/origin'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
            jfacorro Juan Facorro added a comment - - edited

            We are having the same issue on CentOS Linux release 7.2.1511 (Core). The plugin version is 0.32.

            Directories seem to be marked for cleanup but their owner instead of being jenkins is another user.

            jfacorro Juan Facorro added a comment - - edited We are having the same issue on CentOS Linux release 7.2.1511 (Core). The plugin version is 0.32. Directories seem to be marked for cleanup but their owner instead of being jenkins is another user.
            gjphilp Gregor Philp added a comment - - edited

            Hi, we have the same issue still on CentOS Linux 6 and 7 platforms. We'd like these to just be removed and not reported so that we have to manually remove them. We end up with 100s, 1000s of these.

            We have latest plugin versions:
            Workspace Cleanup Plugin - 0.32
            Resource Disposer Plugin - 0.3
            and running jenkins master - 2.19.4

            It seems this might be related to the problem we're having of the java heap being used up and then our master fails. I then have to restart master.

            thanks
            Gregor

            gjphilp Gregor Philp added a comment - - edited Hi, we have the same issue still on CentOS Linux 6 and 7 platforms. We'd like these to just be removed and not reported so that we have to manually remove them. We end up with 100s, 1000s of these. We have latest plugin versions: Workspace Cleanup Plugin - 0.32 Resource Disposer Plugin - 0.3 and running jenkins master - 2.19.4 It seems this might be related to the problem we're having of the java heap being used up and then our master fails. I then have to restart master. thanks Gregor

            gjphilp, the screenshot demonstrates how it is supposed to work. Jenkins tries to delete it but as it fail repeatedly so item is tracked in resource disposer until the directory get fried. It seems to be never in your case. It is reported for you attention as there is something that prevents the directory to be deleted for a long time. The cause needs to be found and eliminated in your case. Hover over the exception message to see the full exception that might contain the clue.

            olivergondza Oliver Gondža added a comment - gjphilp , the screenshot demonstrates how it is supposed to work. Jenkins tries to delete it but as it fail repeatedly so item is tracked in resource disposer until the directory get fried. It seems to be never in your case. It is reported for you attention as there is something that prevents the directory to be deleted for a long time. The cause needs to be found and eliminated in your case. Hover over the exception message to see the full exception that might contain the clue.

            I am still experiencing the problem, and in my case I noticed that the build process is creating soft links that are owned by root. The cleanup process is not able to delete these, so it tracks them in the resource disposer. I am working with the developer to find how those links are being created, but this is something that should be addressed in the plugin as well.

            stephan_fenton Stephan Fenton added a comment - I am still experiencing the problem, and in my case I noticed that the build process is creating soft links that are owned by root. The cleanup process is not able to delete these, so it tracks them in the resource disposer. I am working with the developer to find how those links are being created, but this is something that should be addressed in the plugin as well.

            stephan_fenton, note that this has nothing to do with the plugin or async deletion. The build just creates stuff Jenkins agent has no permission to clean.

            The only thing I can think of to improve this is turn the async deletion off in case the project has a bad record on workspace deletion so such persistent problems will be presented to jobs owners eventually and not the admins.

            olivergondza Oliver Gondža added a comment - stephan_fenton , note that this has nothing to do with the plugin or async deletion. The build just creates stuff Jenkins agent has no permission to clean. The only thing I can think of to improve this is turn the async deletion off in case the project has a bad record on workspace deletion so such persistent problems will be presented to jobs owners eventually and not the admins.
            m4x1m0v3r Albert V added a comment -

            I'm experiencing the same issue within an updated Jenkins (2.53) and ws-cleanup plugin (0.32).
            I wanted to note that when I'm using the ws-cleanup into a pipeline, is working as expected. But when I'm using it into a regular Maven job it is leaving the famous directory with a timestamp (one per build!) and is filling the workspace of my slaves.

            I think that this issue should be re-opened because there are more cases like this one.

            m4x1m0v3r Albert V added a comment - I'm experiencing the same issue within an updated Jenkins (2.53) and ws-cleanup plugin (0.32). I wanted to note that when I'm using the ws-cleanup into a pipeline, is working as expected. But when I'm using it into a regular Maven job it is leaving the famous directory with a timestamp (one per build!) and is filling the workspace of my slaves. I think that this issue should be re-opened because there are more cases like this one.

            I could replicate the issue if you disconnect the agent when the prebuild cleanup it is make, if you do that the temporal folder will never be deleted

            ifernandezcalvo Ivan Fernandez Calvo added a comment - I could replicate the issue if you disconnect the agent when the prebuild cleanup it is make, if you do that the temporal folder will never be deleted
            brian_hewson brian hewson added a comment -

            gjphilp I had the same problem of having to manually remove hundreds of reports.

            curl -s http://${JENKINS_URL}/administrativeMonitor/AsyncResourceDisposer/ -u${UTILITY_USER}:${UTILITY_PW} | tr '"' '\n' | grep 'stop-tracking' | cut -d '-' -f 3 | sort -n | while read ASYNC_THREAD; do curl http://${JENKINS_URL}/administrativeMonitor/AsyncResourceDisposer/stopTracking -u${UTILITY_USER}:${UTILITY_PW} -X POST --data "id=${ASYNC_THREAD}"; done

            Heap usage dropped significantly immediately after cleanup of 337 threads. 

            brian_hewson brian hewson added a comment - gjphilp I had the same problem of having to manually remove hundreds of reports. curl -s http: //${JENKINS_URL}/administrativeMonitor/AsyncResourceDisposer/ -u${UTILITY_USER}:${UTILITY_PW} | tr ' "' '\n' | grep 'stop-tracking' | cut -d '-' -f 3 | sort -n | while read ASYNC_THREAD; do curl http://${JENKINS_URL}/administrativeMonitor/AsyncResourceDisposer/stopTracking -u${UTILITY_USER}:${UTILITY_PW} -X POST --data " id=${ASYNC_THREAD}"; done Heap usage dropped significantly immediately after cleanup of 337 threads. 
            zuboram Cesar Ibarra added a comment -

            Can anyone explain me how they resolve this?
            I am using Jenkins 2.107.2 with ws-cleanup 0.34 and I am also experience the issue of a lingering ws-cleanup folder with the timestamp which didn't get deleted.
            I am running Jenkins on a OS Ubuntu 16.04.

            Any help on how to solve this?

            zuboram Cesar Ibarra added a comment - Can anyone explain me how they resolve this? I am using Jenkins 2.107.2 with ws-cleanup 0.34 and I am also experience the issue of a lingering ws-cleanup folder with the timestamp which didn't get deleted. I am running Jenkins on a OS Ubuntu 16.04. Any help on how to solve this?

            Guys, this ticket was resolved year and a half ago. Please, file a separate issue.

            olivergondza Oliver Gondža added a comment - Guys, this ticket was resolved year and a half ago. Please, file a separate issue.

            But the problem still persists (here too, BTW.). So why file a new issue for the VERY SAME problem? Reopening a ticket is common practice in such cases.

            dhs Dirk Heinrichs added a comment - But the problem still persists (here too, BTW.). So why file a new issue for the VERY SAME problem? Reopening a ticket is common practice in such cases.

            We also have this issue but only on Windows slaves (connected through Cygwin SSH).

            broussar Adam Brousseau added a comment - We also have this issue but only on Windows slaves (connected through Cygwin SSH).

            Have the same issue. Jenkins 2.121.1 in docker container(workspace on volume) , ws-cleanup 0.34

            zakharovdi Denis Zakharov added a comment - Have the same issue. Jenkins 2.121.1 in docker container(workspace on volume) , ws-cleanup 0.34

            Guys, this ticket was resolved year and a half ago. Please, file a separate issue.

            olivergondza Oliver Gondža added a comment - Guys, this ticket was resolved year and a half ago. Please, file a separate issue.

            Opened https://issues.jenkins-ci.org/browse/JENKINS-53579 now can we get this addressed?

            jbochenski Jakub Bochenski added a comment - Opened https://issues.jenkins-ci.org/browse/JENKINS-53579 now can we get this addressed?

            People

              olivergondza Oliver Gondža
              tmoore Tom Moore
              Votes:
              18 Vote for this issue
              Watchers:
              40 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: