Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-24824

Asynchronous cleanup not removing renamed workspace directories on slaves

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: ws-cleanup-plugin
    • Labels:
      None
    • Environment:
      Jenkins 1.579
      ws-cleanup 0.24
    • Similar Issues:

      Description

      After upgrading to ws-cleanup 0.24 in order to get the asynchronous cleanup function we noticed the workspaces on our slaves getting renamed to the form of ${WORKSPACE}ws-cleanup${TIMESTAMP}. (ie, job1 would become job1_ws-cleanup_1411197183394). The expected behavior under ws-cleanup 0.24 is that these were temporary to support asynchronous processing and would be deleted. However, these directories never get removed from the slave. Over time, the slave hard drives filled up resulting in build failures.

        Attachments

          Issue Links

            Activity

            Hide
            vjuranek vjuranek added a comment -

            Hi, any idea how to reproduce it? Everything works fine on my machine. Do you use any "Advanced" options? If so, which one? What is the OS of the slave where ws is not deleted?
            Thanks

            Show
            vjuranek vjuranek added a comment - Hi, any idea how to reproduce it? Everything works fine on my machine. Do you use any "Advanced" options? If so, which one? What is the OS of the slave where ws is not deleted? Thanks
            Hide
            tmoore Tom Moore added a comment -

            The slaves are running on a VM that is running Windows 7 Enterprise, Service Pack 1. The slaves start their Jenkins connection via slave command line. Running as a Windows service is not an option as the build process must be subordinate to an active login session due to compiler restrictions. No advanced options were set.

            Show
            tmoore Tom Moore added a comment - The slaves are running on a VM that is running Windows 7 Enterprise, Service Pack 1. The slaves start their Jenkins connection via slave command line. Running as a Windows service is not an option as the build process must be subordinate to an active login session due to compiler restrictions. No advanced options were set.
            Hide
            katrine Katrine Skovbo added a comment -

            We are experiencing the same problem. In our case the renamed workspaces are not removed when we use Multiple SCMs to checkout more than one repository from git. The unremoved repositories are placed in a sub-directory.

            Show
            katrine Katrine Skovbo added a comment - We are experiencing the same problem. In our case the renamed workspaces are not removed when we use Multiple SCMs to checkout more than one repository from git. The unremoved repositories are placed in a sub-directory.
            Hide
            vjuranek vjuranek added a comment -

            Unfortunately I hasn't been able reproduce it yet Any reproducer would definitely help. Will try with MultipleSCM and several git repos.

            Show
            vjuranek vjuranek added a comment - Unfortunately I hasn't been able reproduce it yet Any reproducer would definitely help. Will try with MultipleSCM and several git repos.
            Hide
            sebok Marton Sebok added a comment -

            Hi, I have the same issue. I think one key point is that my slave is configured to go offline after being idle for a while, and I want to delete pretty much files from a complex directory tree. I can see that some folders have been deleted from the workspace, but heavier ones are left there.

            Show
            sebok Marton Sebok added a comment - Hi, I have the same issue. I think one key point is that my slave is configured to go offline after being idle for a while, and I want to delete pretty much files from a complex directory tree. I can see that some folders have been deleted from the workspace, but heavier ones are left there.
            Hide
            tmoore Tom Moore added a comment -

            perhaps workspace size is a factor? The workspaces we saw this on come in at just over 6 GB. (Which is why we wanted to be able to use this feature in the first place)

            Show
            tmoore Tom Moore added a comment - perhaps workspace size is a factor? The workspaces we saw this on come in at just over 6 GB. (Which is why we wanted to be able to use this feature in the first place)
            Hide
            vjuranek vjuranek added a comment -

            Sorry, cannot reproduce even with MutilpeSCM plugin (Jenkins 1.574, MutipleSCM 0.3, Git 2.2.7, ws clenaup 0.24).
            @Marton Sebok haven't tried it yet, but this seem to be a valid concern. Will investigate it deeper. Thanks for a pointer!

            Show
            vjuranek vjuranek added a comment - Sorry, cannot reproduce even with MutilpeSCM plugin (Jenkins 1.574, MutipleSCM 0.3, Git 2.2.7, ws clenaup 0.24). @Marton Sebok haven't tried it yet, but this seem to be a valid concern. Will investigate it deeper. Thanks for a pointer!
            Hide
            vjuranek vjuranek added a comment -

            @Tom Moore and do you also put slaves offline as pointer out by Marton Sebok? In this case it does make sense for me, but otherwise size of the workspace shouldn't IMHO matter. Do you see any errors in the logs?

            Show
            vjuranek vjuranek added a comment - @Tom Moore and do you also put slaves offline as pointer out by Marton Sebok? In this case it does make sense for me, but otherwise size of the workspace shouldn't IMHO matter. Do you see any errors in the logs?
            Hide
            tmoore Tom Moore added a comment -

            No, we don't put the slaves offline. We didn't notice any errors in the logs, the first indication we had that this was a problem was when we ran out of disk space.

            Show
            tmoore Tom Moore added a comment - No, we don't put the slaves offline. We didn't notice any errors in the logs, the first indication we had that this was a problem was when we ran out of disk space.
            Hide
            katrine Katrine Skovbo added a comment -

            It seems that the files we have trouble with are the ones that contain Danish characters æ,ø or å in the file name.

            Show
            katrine Katrine Skovbo added a comment - It seems that the files we have trouble with are the ones that contain Danish characters æ,ø or å in the file name.
            Hide
            danielbeck Daniel Beck added a comment -

            Katrine Skovbo: Are these characters in the workspace (project) name, a folder name in the workspace, or a file name in the workspace? Are you deleting the entire workspace? Did it work on the older Jenkins version? What are the contents of the /systemInfo URL (or /computer/slavename/systemInfo URL for workspaces on slaves)?

            Show
            danielbeck Daniel Beck added a comment - Katrine Skovbo : Are these characters in the workspace (project) name, a folder name in the workspace, or a file name in the workspace? Are you deleting the entire workspace? Did it work on the older Jenkins version? What are the contents of the /systemInfo URL (or /computer/slavename/systemInfo URL for workspaces on slaves)?
            Hide
            katrine Katrine Skovbo added a comment -

            The characters are in file names in the workspace.
            We delete the entire workspace.
            It worked before the upgrade to version 0.24, but I tried downgrading to 0.23 again and that also produced errors now. In 0.23 it now explicitly says that it cannot delete the files:

            ERROR: Failed to clean the workspace
            09:05:02 java.io.IOException: java.lang.reflect.InvocationTargetException
            09:05:02 	at hudson.Util.isSymlinkJava7(Util.java:360)
            09:05:02 	at hudson.Util.isSymlink(Util.java:325)
            09:05:02 	at hudson.Util.deleteRecursive(Util.java:291)
            09:05:02 	at hudson.Util.deleteContentsRecursive(Util.java:203)
            09:05:02 	at hudson.Util.deleteRecursive(Util.java:292)
            09:05:02 	at hudson.Util.deleteContentsRecursive(Util.java:203)
            09:05:02 	at hudson.Util.deleteRecursive(Util.java:292)
            09:05:02 	at hudson.Util.deleteContentsRecursive(Util.java:203)
            09:05:02 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:393)
            09:05:02 	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
            09:05:02 	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
            09:05:02 	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
            09:05:02 	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            09:05:02 	at hudson.remoting.Request$2.run(Request.java:328)
            09:05:02 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
            09:05:02 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            09:05:02 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            09:05:02 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            09:05:02 	at java.lang.Thread.run(Thread.java:745)
            09:05:02 Caused by: java.lang.reflect.InvocationTargetException
            09:05:02 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            09:05:02 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            09:05:02 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            09:05:02 	at java.lang.reflect.Method.invoke(Method.java:606)
            09:05:02 	at hudson.Util.isSymlinkJava7(Util.java:355)
            09:05:02 	... 18 more
            09:05:02 Caused by: java.nio.file.InvalidPathException: Malformed input or input contains unmappable chacraters: /var/lib/jenkins/workspace/My-project/sub-folder/prototype/icons/skylleanl??g.svg
            09:05:02 	at sun.nio.fs.UnixPath.encode(UnixPath.java:147)
            09:05:02 	at sun.nio.fs.UnixPath.<init>(UnixPath.java:71)
            09:05:02 	at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:281)
            09:05:02 	at java.io.File.toPath(File.java:2186)
            09:05:02 	... 23 more
            

            We are using Jenkins 1.565.3 and a Unix slave, version 2.46, java 1.7.0_72

            Show
            katrine Katrine Skovbo added a comment - The characters are in file names in the workspace. We delete the entire workspace. It worked before the upgrade to version 0.24, but I tried downgrading to 0.23 again and that also produced errors now. In 0.23 it now explicitly says that it cannot delete the files: ERROR: Failed to clean the workspace 09:05:02 java.io.IOException: java.lang.reflect.InvocationTargetException 09:05:02 at hudson.Util.isSymlinkJava7(Util.java:360) 09:05:02 at hudson.Util.isSymlink(Util.java:325) 09:05:02 at hudson.Util.deleteRecursive(Util.java:291) 09:05:02 at hudson.Util.deleteContentsRecursive(Util.java:203) 09:05:02 at hudson.Util.deleteRecursive(Util.java:292) 09:05:02 at hudson.Util.deleteContentsRecursive(Util.java:203) 09:05:02 at hudson.Util.deleteRecursive(Util.java:292) 09:05:02 at hudson.Util.deleteContentsRecursive(Util.java:203) 09:05:02 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:393) 09:05:02 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153) 09:05:02 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146) 09:05:02 at hudson.remoting.UserRequest.perform(UserRequest.java:118) 09:05:02 at hudson.remoting.UserRequest.perform(UserRequest.java:48) 09:05:02 at hudson.remoting.Request$2.run(Request.java:328) 09:05:02 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) 09:05:02 at java.util.concurrent.FutureTask.run(FutureTask.java:262) 09:05:02 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 09:05:02 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 09:05:02 at java.lang. Thread .run( Thread .java:745) 09:05:02 Caused by: java.lang.reflect.InvocationTargetException 09:05:02 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 09:05:02 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 09:05:02 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 09:05:02 at java.lang.reflect.Method.invoke(Method.java:606) 09:05:02 at hudson.Util.isSymlinkJava7(Util.java:355) 09:05:02 ... 18 more 09:05:02 Caused by: java.nio.file.InvalidPathException: Malformed input or input contains unmappable chacraters: / var /lib/jenkins/workspace/My-project/sub-folder/prototype/icons/skylleanl??g.svg 09:05:02 at sun.nio.fs.UnixPath.encode(UnixPath.java:147) 09:05:02 at sun.nio.fs.UnixPath.<init>(UnixPath.java:71) 09:05:02 at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:281) 09:05:02 at java.io.File.toPath(File.java:2186) 09:05:02 ... 23 more We are using Jenkins 1.565.3 and a Unix slave, version 2.46, java 1.7.0_72
            Hide
            vjuranek vjuranek added a comment -

            Katrine Skovbo: this seems to be unrelated to the bug discussed in this issue. This is probably different bug and this error is just hidden by async delete. I filed a new issue for this: JENKINS-25317

            Show
            vjuranek vjuranek added a comment - Katrine Skovbo : this seems to be unrelated to the bug discussed in this issue. This is probably different bug and this error is just hidden by async delete. I filed a new issue for this: JENKINS-25317
            Hide
            olivergondza Oliver Gondža added a comment - - edited

            I expect the deletion can fail for some reason even after we successfully renamed the directory. I see several ways to improve current situation:

            • Retry the deletion until it succeeds - it should not block the build but can as well run for unpredictable amount of time
            • Introduce a periodic task to clean up such dangling directories
            • Pass Future from FilePath.actAsync to some "collector" task that would retry and/or report in case some deletion fails.*

            [*] I guess this might be the only way to find out why it failed on the first place as it seems that FilePath.actAsync does not log exceptions.

            Show
            olivergondza Oliver Gondža added a comment - - edited I expect the deletion can fail for some reason even after we successfully renamed the directory. I see several ways to improve current situation: Retry the deletion until it succeeds - it should not block the build but can as well run for unpredictable amount of time Introduce a periodic task to clean up such dangling directories Pass Future from FilePath.actAsync to some "collector" task that would retry and/or report in case some deletion fails.* [*] I guess this might be the only way to find out why it failed on the first place as it seems that FilePath.actAsync does not log exceptions.
            Hide
            vivo Victor Volle added a comment -

            We added some logging around the deletion (see below) and found that there was a permissions problem.

            I would suggest to apply the patch to the plugin so that others can see the root cause as well

            From 4dd5c02d5c65a30860ed4ccc2baa860f331f156f Mon Sep 17 00:00:00 2001
            From: Victor Volle <vivo@Victors-MacBook-Pro.local>
            Date: Tue, 18 Aug 2015 17:53:41 +0200
            Subject: [PATCH] JENKINS-24824: log error
            
            ---
             src/main/java/hudson/plugins/ws_cleanup/Wipeout.java | 7 ++++++-
             1 file changed, 6 insertions(+), 1 deletion(-)
            
            diff --git a/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java b/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java
            index e1e759b..a9e50a6 100644
            --- a/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java
            +++ b/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java
            @@ -69,7 +69,12 @@ import hudson.remoting.VirtualChannel;
                 private final static Command COMMAND = new Command();
                 private final static class Command implements FileCallable<Object> {
                     public Object invoke(File f, VirtualChannel channel) throws IOException, InterruptedException {
            -            Util.deleteRecursive(f);
            +            try {
            +                Util.deleteRecursive(f);
            +            } catch (IOException e) {
            +                LOGGER.log(Level.WARNING, "error cleaning up", e);
            +                throw e;
            +            }
                         return null;
                     }
                 }
            -- 
            2.3.2 (Apple Git-55)
            
            Show
            vivo Victor Volle added a comment - We added some logging around the deletion (see below) and found that there was a permissions problem. I would suggest to apply the patch to the plugin so that others can see the root cause as well From 4dd5c02d5c65a30860ed4ccc2baa860f331f156f Mon Sep 17 00:00:00 2001 From: Victor Volle <vivo@Victors-MacBook-Pro.local> Date: Tue, 18 Aug 2015 17:53:41 +0200 Subject: [PATCH] JENKINS-24824: log error --- src/main/java/hudson/plugins/ws_cleanup/Wipeout.java | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java b/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java index e1e759b..a9e50a6 100644 --- a/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java +++ b/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java @@ -69,7 +69,12 @@ import hudson.remoting.VirtualChannel; private final static Command COMMAND = new Command(); private final static class Command implements FileCallable< Object > { public Object invoke(File f, VirtualChannel channel) throws IOException, InterruptedException { - Util.deleteRecursive(f); + try { + Util.deleteRecursive(f); + } catch (IOException e) { + LOGGER.log(Level.WARNING, "error cleaning up" , e); + throw e; + } return null ; } } -- 2.3.2 (Apple Git-55)
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oliver Gondža
            Path:
            src/main/java/hudson/plugins/ws_cleanup/Wipeout.java
            http://jenkins-ci.org/commit/ws-cleanup-plugin/e151b89222f93b8b125f3f5e1dffb51e6ad8be9d
            Log:
            JENKINS-24824 Log deletion failure

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: src/main/java/hudson/plugins/ws_cleanup/Wipeout.java http://jenkins-ci.org/commit/ws-cleanup-plugin/e151b89222f93b8b125f3f5e1dffb51e6ad8be9d Log: JENKINS-24824 Log deletion failure
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oliver Gondža
            Path:
            src/main/java/hudson/plugins/ws_cleanup/Wipeout.java
            http://jenkins-ci.org/commit/ws-cleanup-plugin/32766a78cf81399fcd0b46f68600863c0369a869
            Log:
            Merge pull request #22 from olivergondza/JENKINS-24824

            JENKINS-24824 Log deletion failure

            Compare: https://github.com/jenkinsci/ws-cleanup-plugin/compare/6bc314398d0f...32766a78cf81

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: src/main/java/hudson/plugins/ws_cleanup/Wipeout.java http://jenkins-ci.org/commit/ws-cleanup-plugin/32766a78cf81399fcd0b46f68600863c0369a869 Log: Merge pull request #22 from olivergondza/ JENKINS-24824 JENKINS-24824 Log deletion failure Compare: https://github.com/jenkinsci/ws-cleanup-plugin/compare/6bc314398d0f...32766a78cf81
            Hide
            tcb_xy Tim-Christian Bloss added a comment -

            Same issue here.

            We have one Linux master running multiple slaves, some windows, some Linux.

            Linux slaves (VMWare VM) get a revert to snapshot using vmware plugin, which is not possible for windows slaves in our current environment.

            To handle hanging processes and locked files, we disconnect each windows slave after 1 single build, shutdown and power on the vm.

            Our problem are ever growing workspace folders, allocating far more than 80gig of disk space within about two weeks, as the async cleanup can not run and remove old workspaces due to premature disconnect and termination of the java vm.

            In our case the workspace cleanup plugin can rename old workspaces successfully (* -> *ws-cleanup[1234567890]+) , but does not delete those old folders after successfully reconnecting the windows slaves.

            We get no errors logged using ws-cleanup plugin version 0.28 in our build logs.

            Show
            tcb_xy Tim-Christian Bloss added a comment - Same issue here. We have one Linux master running multiple slaves, some windows, some Linux. Linux slaves (VMWare VM) get a revert to snapshot using vmware plugin, which is not possible for windows slaves in our current environment. To handle hanging processes and locked files, we disconnect each windows slave after 1 single build, shutdown and power on the vm. Our problem are ever growing workspace folders, allocating far more than 80gig of disk space within about two weeks, as the async cleanup can not run and remove old workspaces due to premature disconnect and termination of the java vm. In our case the workspace cleanup plugin can rename old workspaces successfully (* -> * ws-cleanup [1234567890] +) , but does not delete those old folders after successfully reconnecting the windows slaves. We get no errors logged using ws-cleanup plugin version 0.28 in our build logs.
            Hide
            skarlso Gergely Brautigam added a comment - - edited

            Hi.

            Same issue here. Looks like the async delete isn't happening fast enough or isn't happening at all. Most of the time, when our git ran, but the job failed early, the async cleanup did not have time to finish deleting a folder.

            Or the other problem is usually an access denied error. Note, not a permission, but access. Meaning, the directory was still being used by some loose thread of a process.

            For now, I'm inclined to create a script on the slaves and put it into cron job which runs and cleans up every folder with ws-cleanup in it. But that is not a really nice solution.

            EDIT: And yes, same here, usually there is no error involved. Just a [WS-CLEANUP] Deleting project workspace... and no 'Done' in the logs anywhere.

            Show
            skarlso Gergely Brautigam added a comment - - edited Hi. Same issue here. Looks like the async delete isn't happening fast enough or isn't happening at all. Most of the time, when our git ran, but the job failed early, the async cleanup did not have time to finish deleting a folder. Or the other problem is usually an access denied error. Note, not a permission, but access. Meaning, the directory was still being used by some loose thread of a process. For now, I'm inclined to create a script on the slaves and put it into cron job which runs and cleans up every folder with ws-cleanup in it. But that is not a really nice solution. EDIT : And yes, same here, usually there is no error involved. Just a [WS-CLEANUP] Deleting project workspace... and no 'Done' in the logs anywhere.
            Hide
            olivergondza Oliver Gondža added a comment - - edited

            Tim-Christian Bloss Your use-case is quite special and was not taken into account originally.

            Gergely Brautigam

            Looks like the async delete isn't happening fast enough or isn't happening at all. Most of the time, when our git ran, but the job failed early, the async cleanup did not have time to finish deleting a folder.

            I do not understand this. How does the problem manifests?

            Note the async deletion failure cuses should not go into a build log (as is runs in parallel with the build and can even run for a longer time than the build itself) but a java logger. Check Jenkins master log or configure custom logged in UI for hudson.plugins.ws_cleanup to see why it fails.

            In the meantime, JENKINS-27648 was rejected in jenkins core so I guess we have to implement a periodic task to clean dangling *ws-cleanup* dirs. That should resolve the problems with slaves disconnected unexpectedly and directories not deleted in time because of opened file descriptors.

            Show
            olivergondza Oliver Gondža added a comment - - edited Tim-Christian Bloss Your use-case is quite special and was not taken into account originally. Gergely Brautigam Looks like the async delete isn't happening fast enough or isn't happening at all. Most of the time, when our git ran, but the job failed early, the async cleanup did not have time to finish deleting a folder. I do not understand this. How does the problem manifests? Note the async deletion failure cuses should not go into a build log (as is runs in parallel with the build and can even run for a longer time than the build itself) but a java logger. Check Jenkins master log or configure custom logged in UI for hudson.plugins.ws_cleanup to see why it fails. In the meantime, JENKINS-27648 was rejected in jenkins core so I guess we have to implement a periodic task to clean dangling *ws-cleanup* dirs. That should resolve the problems with slaves disconnected unexpectedly and directories not deleted in time because of opened file descriptors.
            Hide
            skarlso Gergely Brautigam added a comment -

            The symptom is a lingering ws-cleanup folder with the timestamp which didn't get deleted. Btw, most of the times it manifests on windows only. Rarely happens on a linux slave interestingly enough.

            Yes, sorry, the access denied error message was in the Jenkins Log.

            However, the Done, part is in the job's log. And that one was obviously missing because of the exception in the main log.

            Show
            skarlso Gergely Brautigam added a comment - The symptom is a lingering ws-cleanup folder with the timestamp which didn't get deleted. Btw, most of the times it manifests on windows only. Rarely happens on a linux slave interestingly enough. Yes, sorry, the access denied error message was in the Jenkins Log. However, the Done, part is in the job's log. And that one was obviously missing because of the exception in the main log.
            Hide
            danielbeck Daniel Beck added a comment -

            interestingly enough

            Windows file locking?

            Show
            danielbeck Daniel Beck added a comment - interestingly enough Windows file locking?
            Hide
            skarlso Gergely Brautigam added a comment -

            Windows file locking?

            Lacking enough evidence to say yay, or nay. It is 'a' possibility.

            Or even some other process locking a certain file. I even saw git lingering and the .git folder could not be deleted in a RENAMED folder. So much crazy, such wow.

            Show
            skarlso Gergely Brautigam added a comment - Windows file locking? Lacking enough evidence to say yay, or nay. It is 'a' possibility. Or even some other process locking a certain file. I even saw git lingering and the .git folder could not be deleted in a RENAMED folder. So much crazy, such wow.
            Hide
            danielbeck Daniel Beck added a comment -

            some other process locking a certain file. I even saw git lingering

            Sorry, that's what I meant. Too much locking going on and Windows doesn't allow deleting folders with open handles.

            Show
            danielbeck Daniel Beck added a comment - some other process locking a certain file. I even saw git lingering Sorry, that's what I meant. Too much locking going on and Windows doesn't allow deleting folders with open handles.
            Hide
            skarlso Gergely Brautigam added a comment -

            Affirmative. When it happens, it's usually because of windows file locking.

            Show
            skarlso Gergely Brautigam added a comment - Affirmative. When it happens, it's usually because of windows file locking.
            Hide
            lunarfs Frank Vissing added a comment - - edited

            I have a reproduce that can trigger this.. not the same as the classic slave setup but still.
            Provision a docker slave that will be provisioned with the 'One retention strategy'
            have a job that executes in the docker container (my job has less than 100 mb data)
            perform ws clean-up on success, fail etc.
            now once the job completes the docker container is killed and the wc-cleanup_timestamp folder remains
            this job runs on linux, so no relation to windows here
            one option could be a checkbox to disable assync delete
            the /home/jenkins/workspace folder is mounted in the docker container using data volumes, therfore the data remains on the filesystem until we destroy the data volumen

            The workaround for this is not to delete the workspace after building but prior to building.

            Show
            lunarfs Frank Vissing added a comment - - edited I have a reproduce that can trigger this.. not the same as the classic slave setup but still. Provision a docker slave that will be provisioned with the 'One retention strategy' have a job that executes in the docker container (my job has less than 100 mb data) perform ws clean-up on success, fail etc. now once the job completes the docker container is killed and the wc-cleanup_timestamp folder remains this job runs on linux, so no relation to windows here one option could be a checkbox to disable assync delete the /home/jenkins/workspace folder is mounted in the docker container using data volumes, therfore the data remains on the filesystem until we destroy the data volumen The workaround for this is not to delete the workspace after building but prior to building.
            Hide
            slsh Jarkko Rantavuori added a comment - - edited

            We get this all the time - for out test jobs, there are hundreds of cleanup folders appearing on our disk. This is Ubuntu 14.04.1 64-bit slave, so at least for us it is not related to windows file locking. Also, we have had "Delete workspace before build starts" selected, but it has not fixed the issue.

            Our Jenkins version is 1.632, cleanup plugin 0.26.

            Update: reverting to cleanup plugin 0.23, which didn't have asynch deletion, allowed us to see the error: we had a shell script section in the jobs which created a folder having root owner instead of jenkins, so the following runs of a job would fail since there weren't able to delete the previous workspace.

            I think what needs to be done for asynch plugin is to fail the build somehow if the deletion fails and report the error. Also, it would be nice if user could select between asynch and synch delete instead of having to downgrade all the way to 0.23.

            Show
            slsh Jarkko Rantavuori added a comment - - edited We get this all the time - for out test jobs, there are hundreds of cleanup folders appearing on our disk. This is Ubuntu 14.04.1 64-bit slave, so at least for us it is not related to windows file locking. Also, we have had "Delete workspace before build starts" selected, but it has not fixed the issue. Our Jenkins version is 1.632, cleanup plugin 0.26. Update: reverting to cleanup plugin 0.23, which didn't have asynch deletion, allowed us to see the error: we had a shell script section in the jobs which created a folder having root owner instead of jenkins, so the following runs of a job would fail since there weren't able to delete the previous workspace. I think what needs to be done for asynch plugin is to fail the build somehow if the deletion fails and report the error. Also, it would be nice if user could select between asynch and synch delete instead of having to downgrade all the way to 0.23.
            Hide
            jdoose Jens Doose added a comment - - edited

            I have the same behaviour in a non-slave environment, I don't know if this is the same or a new issue.
            A lot of directories like workspace_ws-cleanup_1447277714463 get created and never deleted.

            In the log of the jenkins job there is the output that everything is ok:
            [WS-CLEANUP] Deleting project workspace...
            [WS-CLEANUP] Done

            The job is building node projects and loading npm componentes as well as bower components which result in quite huge directory structures, might that be a problem?

            Show
            jdoose Jens Doose added a comment - - edited I have the same behaviour in a non-slave environment, I don't know if this is the same or a new issue. A lot of directories like workspace_ws-cleanup_1447277714463 get created and never deleted. In the log of the jenkins job there is the output that everything is ok: [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done The job is building node projects and loading npm componentes as well as bower components which result in quite huge directory structures, might that be a problem?
            Hide
            tcollignon Thomas Collignon added a comment -

            In fact if you want to switch "asynch/synch" mode, you can add Patterns, because this will affect cleanup method :

            • Add Patterns : synch mode
            • No Patterns : asynch mode

            This is just a trick to see the cleanup problem without downgrade to 0.23.

            Show
            tcollignon Thomas Collignon added a comment - In fact if you want to switch "asynch/synch" mode, you can add Patterns, because this will affect cleanup method : Add Patterns : synch mode No Patterns : asynch mode This is just a trick to see the cleanup problem without downgrade to 0.23.
            Hide
            tcollignon Thomas Collignon added a comment -

            And I have submit a pull request to add a new option to switch between synch and asynch mode without downgrading : https://github.com/jenkinsci/ws-cleanup-plugin/pull/26

            Show
            tcollignon Thomas Collignon added a comment - And I have submit a pull request to add a new option to switch between synch and asynch mode without downgrading : https://github.com/jenkinsci/ws-cleanup-plugin/pull/26
            Hide
            olivergondza Oliver Gondža added a comment -

            Thomas Collignon, thanks for the workaround!

            @All, please check the logged (hudson.plugins.ws_cleanup.Wipeout) cause in slave/master log and attach it to this issue.

            Show
            olivergondza Oliver Gondža added a comment - Thomas Collignon , thanks for the workaround! @All, please check the logged ( hudson.plugins.ws_cleanup.Wipeout ) cause in slave/master log and attach it to this issue.
            Hide
            olivergondza Oliver Gondža added a comment -

            It seems that people generally prefer to fail the build in case the cleanup fails rather that clutter the slave's workspace in the long run. This does not get well together with asynchronous deletion requirement.

            One way I can think of is always use sync approach from post-build step and async only in pre-build step. If the original (renamed) workspace will not be gone by the end of the build, sych deletion will be reattempted and will have a chance to fail the build in case of problems. The advantage is that build can start right away and most of the post build actions will do its job (publish junit, determine result, etc.) before the final cleanup kicks in.

            WDYT?

            Show
            olivergondza Oliver Gondža added a comment - It seems that people generally prefer to fail the build in case the cleanup fails rather that clutter the slave's workspace in the long run. This does not get well together with asynchronous deletion requirement. One way I can think of is always use sync approach from post-build step and async only in pre-build step. If the original (renamed) workspace will not be gone by the end of the build, sych deletion will be reattempted and will have a chance to fail the build in case of problems. The advantage is that build can start right away and most of the post build actions will do its job (publish junit, determine result, etc.) before the final cleanup kicks in. WDYT?
            Hide
            tcollignon Thomas Collignon added a comment -

            This may be a good thing, why not
            I ask myself if it's possible when asynch mode is on post-build, to get error et bring back to jenkins job trace. In this case the boolean "asynchronously" is necessary I Think to let people chosing the strategy.

            Show
            tcollignon Thomas Collignon added a comment - This may be a good thing, why not I ask myself if it's possible when asynch mode is on post-build, to get error et bring back to jenkins job trace. In this case the boolean "asynchronously" is necessary I Think to let people chosing the strategy.
            Hide
            olivergondza Oliver Gondža added a comment -

            At a certain point Jenkins considers the build log to be complete and never amended again. The problem is that in theory, the cleanup can take longer than the build itself. We can either not care about the result at all (current implementation), wait for the result at the end of the build (potentially postponing the build completion) or try a compromise - check the status at the end of the build and report failure if there is any but do not wait for the completion (this can not guarantee there will be no dangling workspace directories).

            When I think about this further, once we rename the workspace (as we do for async cleanup) it will always require manual cleanup in case of failure as it will not be any build's workspace any longer. When the cleanup fail with sync approach, it leaves the workspace half deleted to be reused by future builds - which is far from optimal too.

            I am working on implementing plugin specific periodic task to cleanup all the temporary directories (since more general solution was rejected from core).

            Show
            olivergondza Oliver Gondža added a comment - At a certain point Jenkins considers the build log to be complete and never amended again. The problem is that in theory, the cleanup can take longer than the build itself. We can either not care about the result at all (current implementation), wait for the result at the end of the build (potentially postponing the build completion) or try a compromise - check the status at the end of the build and report failure if there is any but do not wait for the completion (this can not guarantee there will be no dangling workspace directories). When I think about this further, once we rename the workspace (as we do for async cleanup) it will always require manual cleanup in case of failure as it will not be any build's workspace any longer. When the cleanup fail with sync approach, it leaves the workspace half deleted to be reused by future builds - which is far from optimal too. I am working on implementing plugin specific periodic task to cleanup all the temporary directories (since more general solution was rejected from core).
            Hide
            tcollignon Thomas Collignon added a comment -

            ok I see
            you work on another plugin to clean periodically or you think added this features in this one? you need some help?

            right now you think add option "asynchronously" is not necessary ?

            Show
            tcollignon Thomas Collignon added a comment - ok I see you work on another plugin to clean periodically or you think added this features in this one? you need some help? right now you think add option "asynchronously" is not necessary ?
            Hide
            olivergondza Oliver Gondža added a comment - - edited

            My original idea was the async cleanup will be performance optimization that will be transparent to the user (whoever run the builds). Which it is only partially.

            I am extending this plugin with the periodic task. I understand the plugin becomes lot more sophisticated that we hoped it to be so I will implement a Jenkins-wide, property-based killswitch to get back to sync cleanup should it cause further problems.

            I still do not think that slave workspace getting full should be a concern of a user (as opposed to instance administrator), so I prefer this over a per-job configuration option.

            Show
            olivergondza Oliver Gondža added a comment - - edited My original idea was the async cleanup will be performance optimization that will be transparent to the user (whoever run the builds). Which it is only partially. I am extending this plugin with the periodic task. I understand the plugin becomes lot more sophisticated that we hoped it to be so I will implement a Jenkins-wide, property-based killswitch to get back to sync cleanup should it cause further problems. I still do not think that slave workspace getting full should be a concern of a user (as opposed to instance administrator), so I prefer this over a per-job configuration option.
            Hide
            tcollignon Thomas Collignon added a comment -

            Ok I agree with you. So I wait for your new implementation

            Thank for your answer

            Show
            tcollignon Thomas Collignon added a comment - Ok I agree with you. So I wait for your new implementation Thank for your answer
            Hide
            bimp Bim Paras added a comment -

            Has this issue been resolved? I am still seeing this issue in version 0.25 of plugin. Thanks.

            Show
            bimp Bim Paras added a comment - Has this issue been resolved? I am still seeing this issue in version 0.25 of plugin. Thanks.
            Hide
            genericpenguin Sven Schott added a comment -

            Hi, just wanting to know if there is a timeframe on resolution of this issue. Would like to know just in case it's not in the near future so that I can set up a semi-temporary workaround for the problem (most likely a scheduled cleanup on our windows machines).

            Show
            genericpenguin Sven Schott added a comment - Hi, just wanting to know if there is a timeframe on resolution of this issue. Would like to know just in case it's not in the near future so that I can set up a semi-temporary workaround for the problem (most likely a scheduled cleanup on our windows machines).
            Hide
            jwhitcraft Jon Whitcraft added a comment -

            +1 to a timeframe for fixing this

            I'm thinking of rolling back to 0.23 becuase of this problem.

            Show
            jwhitcraft Jon Whitcraft added a comment - +1 to a timeframe for fixing this I'm thinking of rolling back to 0.23 becuase of this problem.
            Hide
            mdimmick_mnetics Mike Dimmick added a comment -

            I have been experiencing this problem on a Windows 7 Enterprise installation. During manual clean-up I noticed that even using the command line rmdir /s I was getting a lot of The directory is not empty errors, meaning I had to run the command twice to complete removal.

            I also noticed that a large amount of disk space was being consumed by Windows Search content indexing data. I disabled the Windows Search service and deleted the content indexes. Having done this, I'm no longer getting the errors from rmdir. I can also delete workspaces from the Jenkins 'Wipe Out Current Workspace' feature without problems, which was typically reporting an error previously.

            On this system, JENKINS_HOME is at C:\Users\jenkins\.jenkins. We also have an installation of Jenkins on Windows Server 2012 R2, which doesn't suffer from the problem - here, JENKINS_HOME is C:\Jenkins, and the Windows Search feature is not installed.

            Show
            mdimmick_mnetics Mike Dimmick added a comment - I have been experiencing this problem on a Windows 7 Enterprise installation. During manual clean-up I noticed that even using the command line rmdir /s I was getting a lot of The directory is not empty errors, meaning I had to run the command twice to complete removal. I also noticed that a large amount of disk space was being consumed by Windows Search content indexing data. I disabled the Windows Search service and deleted the content indexes. Having done this, I'm no longer getting the errors from rmdir . I can also delete workspaces from the Jenkins 'Wipe Out Current Workspace' feature without problems, which was typically reporting an error previously. On this system, JENKINS_HOME is at C:\Users\jenkins\.jenkins. We also have an installation of Jenkins on Windows Server 2012 R2, which doesn't suffer from the problem - here, JENKINS_HOME is C:\Jenkins, and the Windows Search feature is not installed.
            Hide
            danielbeck Daniel Beck added a comment -

            Mike Dimmick

            Windows Search content indexing data. I disabled the Windows Search service

            Don't run the Windows Search on JENKINS_HOME. Don't run antivirus on JENKINS_HOME. No exceptions.

            Show
            danielbeck Daniel Beck added a comment - Mike Dimmick Windows Search content indexing data. I disabled the Windows Search service Don't run the Windows Search on JENKINS_HOME. Don't run antivirus on JENKINS_HOME. No exceptions.
            Hide
            olivergondza Oliver Gondža added a comment -

            I am having a second look at this: https://github.com/jenkinsci/ws-cleanup-plugin/pull/28

            It implements a administrative monitor that retries the deletion on report failures in UI.

            Show
            olivergondza Oliver Gondža added a comment - I am having a second look at this: https://github.com/jenkinsci/ws-cleanup-plugin/pull/28 It implements a administrative monitor that retries the deletion on report failures in UI.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oliver Gondža
            Path:
            pom.xml
            src/main/java/hudson/plugins/ws_cleanup/Wipeout.java
            src/test/java/hudson/plugins/ws_cleanup/CleanupPowermockTest.java
            src/test/java/hudson/plugins/ws_cleanup/CleanupTest.java
            http://jenkins-ci.org/commit/ws-cleanup-plugin/ccd907188e0489e76ca21a7513843864bab48c90
            Log:
            [FIXED JENKINS-24824] Collect all asynchronously deleted directories

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: pom.xml src/main/java/hudson/plugins/ws_cleanup/Wipeout.java src/test/java/hudson/plugins/ws_cleanup/CleanupPowermockTest.java src/test/java/hudson/plugins/ws_cleanup/CleanupTest.java http://jenkins-ci.org/commit/ws-cleanup-plugin/ccd907188e0489e76ca21a7513843864bab48c90 Log: [FIXED JENKINS-24824] Collect all asynchronously deleted directories
            Hide
            olivergondza Oliver Gondža added a comment -

            This took lot longer than I expected. The fix will be in 0.32.

            Show
            olivergondza Oliver Gondža added a comment - This took lot longer than I expected. The fix will be in 0.32.
            Hide
            aflat aflat added a comment -

            I'm still hitting this on a solaris x86 machine. Seems to work on other OS's

            java.io.IOException: Unable to delete '/opt/jenkins/workspace/myjob-Solaris_ws-cleanup_1478635926754/local/.git/refs/remotes/origin'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
            at hudson.Util.deleteFile(Util.java:248)
            at hudson.FilePath.deleteRecursive(FilePath.java:1209)
            at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
            at hudson.FilePath.deleteRecursive(FilePath.java:1200)
            at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
            at hudson.FilePath.deleteRecursive(FilePath.java:1200)
            at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
            at hudson.FilePath.deleteRecursive(FilePath.java:1200)
            at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
            at hudson.FilePath.deleteRecursive(FilePath.java:1200)
            at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
            at hudson.FilePath.deleteRecursive(FilePath.java:1200)
            at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
            at hudson.FilePath.deleteRecursive(FilePath.java:1200)
            at hudson.FilePath.access$1000(FilePath.java:195)
            at hudson.FilePath$14.invoke(FilePath.java:1179)
            at hudson.FilePath$14.invoke(FilePath.java:1176)
            at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2731)
            at hudson.remoting.UserRequest.perform(UserRequest.java:153)
            at hudson.remoting.UserRequest.perform(UserRequest.java:50)
            at hudson.remoting.Request$2.run(Request.java:332)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)
            at ......remote call to sun04(Native Method)
            at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1435)
            at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
            at hudson.remoting.Channel.call(Channel.java:795)
            at hudson.FilePath.act(FilePath.java:985)
            at hudson.FilePath.act(FilePath.java:974)
            at hudson.FilePath.deleteRecursive(FilePath.java:1176)
            at hudson.plugins.ws_cleanup.Wipeout$DisposableImpl.dispose(Wipeout.java:110)
            at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer$WorkItem.run(AsyncResourceDisposer.java:254)
            at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            at java.lang.Thread.run(Thread.java:745)
            Caused by: java.nio.file.DirectoryNotEmptyException: /opt/jenkins/workspace/myjob-Solaris_ws-cleanup_1478635926754/local/.git/refs/remotes/origin
            at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
            at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
            at java.nio.file.Files.deleteIfExists(Files.java:1118)
            at hudson.Util.tryOnceDeleteFile(Util.java:287)
            at hudson.Util.deleteFile(Util.java:243)
            at hudson.FilePath.deleteRecursive(FilePath.java:1209)
            at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
            at hudson.FilePath.deleteRecursive(FilePath.java:1200)
            at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
            at hudson.FilePath.deleteRecursive(FilePath.java:1200)
            at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
            at hudson.FilePath.deleteRecursive(FilePath.java:1200)
            at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
            at hudson.FilePath.deleteRecursive(FilePath.java:1200)
            at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
            at hudson.FilePath.deleteRecursive(FilePath.java:1200)
            at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
            at hudson.FilePath.deleteRecursive(FilePath.java:1200)
            at hudson.FilePath.access$1000(FilePath.java:195)
            at hudson.FilePath$14.invoke(FilePath.java:1179)
            at hudson.FilePath$14.invoke(FilePath.java:1176)
            at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2731)
            at hudson.remoting.UserRequest.perform(UserRequest.java:153)
            at hudson.remoting.UserRequest.perform(UserRequest.java:50)
            at hudson.remoting.Request$2.run(Request.java:332)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            ... 1 more
            java.io.IOException: Unable to delete '/opt/jenkins/workspace/myjob-Solaris_ws-cleanup_1478635926754/local/.git/refs/remotes/origin'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.

            Show
            aflat aflat added a comment - I'm still hitting this on a solaris x86 machine. Seems to work on other OS's java.io.IOException: Unable to delete '/opt/jenkins/workspace/myjob-Solaris_ws-cleanup_1478635926754/local/.git/refs/remotes/origin'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. at hudson.Util.deleteFile(Util.java:248) at hudson.FilePath.deleteRecursive(FilePath.java:1209) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.access$1000(FilePath.java:195) at hudson.FilePath$14.invoke(FilePath.java:1179) at hudson.FilePath$14.invoke(FilePath.java:1176) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2731) at hudson.remoting.UserRequest.perform(UserRequest.java:153) at hudson.remoting.UserRequest.perform(UserRequest.java:50) at hudson.remoting.Request$2.run(Request.java:332) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) at ......remote call to sun04(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1435) at hudson.remoting.UserResponse.retrieve(UserRequest.java:253) at hudson.remoting.Channel.call(Channel.java:795) at hudson.FilePath.act(FilePath.java:985) at hudson.FilePath.act(FilePath.java:974) at hudson.FilePath.deleteRecursive(FilePath.java:1176) at hudson.plugins.ws_cleanup.Wipeout$DisposableImpl.dispose(Wipeout.java:110) at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer$WorkItem.run(AsyncResourceDisposer.java:254) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.nio.file.DirectoryNotEmptyException: /opt/jenkins/workspace/myjob-Solaris_ws-cleanup_1478635926754/local/.git/refs/remotes/origin at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242) at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108) at java.nio.file.Files.deleteIfExists(Files.java:1118) at hudson.Util.tryOnceDeleteFile(Util.java:287) at hudson.Util.deleteFile(Util.java:243) at hudson.FilePath.deleteRecursive(FilePath.java:1209) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218) at hudson.FilePath.deleteRecursive(FilePath.java:1200) at hudson.FilePath.access$1000(FilePath.java:195) at hudson.FilePath$14.invoke(FilePath.java:1179) at hudson.FilePath$14.invoke(FilePath.java:1176) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2731) at hudson.remoting.UserRequest.perform(UserRequest.java:153) at hudson.remoting.UserRequest.perform(UserRequest.java:50) at hudson.remoting.Request$2.run(Request.java:332) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ... 1 more java.io.IOException: Unable to delete '/opt/jenkins/workspace/myjob-Solaris_ws-cleanup_1478635926754/local/.git/refs/remotes/origin'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
            Hide
            jfacorro Juan Facorro added a comment - - edited

            We are having the same issue on CentOS Linux release 7.2.1511 (Core). The plugin version is 0.32.

            Directories seem to be marked for cleanup but their owner instead of being jenkins is another user.

            Show
            jfacorro Juan Facorro added a comment - - edited We are having the same issue on CentOS Linux release 7.2.1511 (Core). The plugin version is 0.32. Directories seem to be marked for cleanup but their owner instead of being jenkins is another user.
            Hide
            gjphilp Gregor Philp added a comment - - edited

            Hi, we have the same issue still on CentOS Linux 6 and 7 platforms. We'd like these to just be removed and not reported so that we have to manually remove them. We end up with 100s, 1000s of these.

            We have latest plugin versions:
            Workspace Cleanup Plugin - 0.32
            Resource Disposer Plugin - 0.3
            and running jenkins master - 2.19.4

            It seems this might be related to the problem we're having of the java heap being used up and then our master fails. I then have to restart master.

            thanks
            Gregor

            Show
            gjphilp Gregor Philp added a comment - - edited Hi, we have the same issue still on CentOS Linux 6 and 7 platforms. We'd like these to just be removed and not reported so that we have to manually remove them. We end up with 100s, 1000s of these. We have latest plugin versions: Workspace Cleanup Plugin - 0.32 Resource Disposer Plugin - 0.3 and running jenkins master - 2.19.4 It seems this might be related to the problem we're having of the java heap being used up and then our master fails. I then have to restart master. thanks Gregor
            Hide
            olivergondza Oliver Gondža added a comment -

            Gregor Philp, the screenshot demonstrates how it is supposed to work. Jenkins tries to delete it but as it fail repeatedly so item is tracked in resource disposer until the directory get fried. It seems to be never in your case. It is reported for you attention as there is something that prevents the directory to be deleted for a long time. The cause needs to be found and eliminated in your case. Hover over the exception message to see the full exception that might contain the clue.

            Show
            olivergondza Oliver Gondža added a comment - Gregor Philp , the screenshot demonstrates how it is supposed to work. Jenkins tries to delete it but as it fail repeatedly so item is tracked in resource disposer until the directory get fried. It seems to be never in your case. It is reported for you attention as there is something that prevents the directory to be deleted for a long time. The cause needs to be found and eliminated in your case. Hover over the exception message to see the full exception that might contain the clue.
            Hide
            stephan_fenton Stephan Fenton added a comment -

            I am still experiencing the problem, and in my case I noticed that the build process is creating soft links that are owned by root. The cleanup process is not able to delete these, so it tracks them in the resource disposer. I am working with the developer to find how those links are being created, but this is something that should be addressed in the plugin as well.

            Show
            stephan_fenton Stephan Fenton added a comment - I am still experiencing the problem, and in my case I noticed that the build process is creating soft links that are owned by root. The cleanup process is not able to delete these, so it tracks them in the resource disposer. I am working with the developer to find how those links are being created, but this is something that should be addressed in the plugin as well.
            Hide
            olivergondza Oliver Gondža added a comment -

            Stephan Fenton, note that this has nothing to do with the plugin or async deletion. The build just creates stuff Jenkins agent has no permission to clean.

            The only thing I can think of to improve this is turn the async deletion off in case the project has a bad record on workspace deletion so such persistent problems will be presented to jobs owners eventually and not the admins.

            Show
            olivergondza Oliver Gondža added a comment - Stephan Fenton , note that this has nothing to do with the plugin or async deletion. The build just creates stuff Jenkins agent has no permission to clean. The only thing I can think of to improve this is turn the async deletion off in case the project has a bad record on workspace deletion so such persistent problems will be presented to jobs owners eventually and not the admins.
            Hide
            m4x1m0v3r Albert V added a comment -

            I'm experiencing the same issue within an updated Jenkins (2.53) and ws-cleanup plugin (0.32).
            I wanted to note that when I'm using the ws-cleanup into a pipeline, is working as expected. But when I'm using it into a regular Maven job it is leaving the famous directory with a timestamp (one per build!) and is filling the workspace of my slaves.

            I think that this issue should be re-opened because there are more cases like this one.

            Show
            m4x1m0v3r Albert V added a comment - I'm experiencing the same issue within an updated Jenkins (2.53) and ws-cleanup plugin (0.32). I wanted to note that when I'm using the ws-cleanup into a pipeline, is working as expected. But when I'm using it into a regular Maven job it is leaving the famous directory with a timestamp (one per build!) and is filling the workspace of my slaves. I think that this issue should be re-opened because there are more cases like this one.
            Hide
            ifernandezcalvo Ivan Fernandez Calvo added a comment -

            I could replicate the issue if you disconnect the agent when the prebuild cleanup it is make, if you do that the temporal folder will never be deleted

            Show
            ifernandezcalvo Ivan Fernandez Calvo added a comment - I could replicate the issue if you disconnect the agent when the prebuild cleanup it is make, if you do that the temporal folder will never be deleted
            Hide
            brian_hewson brian hewson added a comment -

            Gregor Philp I had the same problem of having to manually remove hundreds of reports.

            curl -s http://${JENKINS_URL}/administrativeMonitor/AsyncResourceDisposer/ -u${UTILITY_USER}:${UTILITY_PW} | tr '"' '\n' | grep 'stop-tracking' | cut -d '-' -f 3 | sort -n | while read ASYNC_THREAD; do curl http://${JENKINS_URL}/administrativeMonitor/AsyncResourceDisposer/stopTracking -u${UTILITY_USER}:${UTILITY_PW} -X POST --data "id=${ASYNC_THREAD}"; done

            Heap usage dropped significantly immediately after cleanup of 337 threads. 

            Show
            brian_hewson brian hewson added a comment - Gregor Philp I had the same problem of having to manually remove hundreds of reports. curl -s http: //${JENKINS_URL}/administrativeMonitor/AsyncResourceDisposer/ -u${UTILITY_USER}:${UTILITY_PW} | tr ' "' '\n' | grep 'stop-tracking' | cut -d '-' -f 3 | sort -n | while read ASYNC_THREAD; do curl http://${JENKINS_URL}/administrativeMonitor/AsyncResourceDisposer/stopTracking -u${UTILITY_USER}:${UTILITY_PW} -X POST --data " id=${ASYNC_THREAD}"; done Heap usage dropped significantly immediately after cleanup of 337 threads. 
            Hide
            zuboram Cesar Ibarra added a comment -

            Can anyone explain me how they resolve this?
            I am using Jenkins 2.107.2 with ws-cleanup 0.34 and I am also experience the issue of a lingering ws-cleanup folder with the timestamp which didn't get deleted.
            I am running Jenkins on a OS Ubuntu 16.04.

            Any help on how to solve this?

            Show
            zuboram Cesar Ibarra added a comment - Can anyone explain me how they resolve this? I am using Jenkins 2.107.2 with ws-cleanup 0.34 and I am also experience the issue of a lingering ws-cleanup folder with the timestamp which didn't get deleted. I am running Jenkins on a OS Ubuntu 16.04. Any help on how to solve this?
            Hide
            olivergondza Oliver Gondža added a comment -

            Guys, this ticket was resolved year and a half ago. Please, file a separate issue.

            Show
            olivergondza Oliver Gondža added a comment - Guys, this ticket was resolved year and a half ago. Please, file a separate issue.
            Hide
            dhs Dirk Heinrichs added a comment -

            But the problem still persists (here too, BTW.). So why file a new issue for the VERY SAME problem? Reopening a ticket is common practice in such cases.

            Show
            dhs Dirk Heinrichs added a comment - But the problem still persists (here too, BTW.). So why file a new issue for the VERY SAME problem? Reopening a ticket is common practice in such cases.
            Hide
            broussar Adam Brousseau added a comment -

            We also have this issue but only on Windows slaves (connected through Cygwin SSH).

            Show
            broussar Adam Brousseau added a comment - We also have this issue but only on Windows slaves (connected through Cygwin SSH).
            Hide
            zakharovdi Denis Zakharov added a comment -

            Have the same issue. Jenkins 2.121.1 in docker container(workspace on volume) , ws-cleanup 0.34

            Show
            zakharovdi Denis Zakharov added a comment - Have the same issue. Jenkins 2.121.1 in docker container(workspace on volume) , ws-cleanup 0.34
            Hide
            olivergondza Oliver Gondža added a comment -

            Guys, this ticket was resolved year and a half ago. Please, file a separate issue.

            Show
            olivergondza Oliver Gondža added a comment - Guys, this ticket was resolved year and a half ago. Please, file a separate issue.
            Hide
            jbochenski Jakub Bochenski added a comment -

            Opened https://issues.jenkins-ci.org/browse/JENKINS-53579 now can we get this addressed?

            Show
            jbochenski Jakub Bochenski added a comment - Opened https://issues.jenkins-ci.org/browse/JENKINS-53579 now can we get this addressed?

              People

              Assignee:
              olivergondza Oliver Gondža
              Reporter:
              tmoore Tom Moore
              Votes:
              18 Vote for this issue
              Watchers:
              40 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: