Hi, I have the same issue. I think one key point is that my slave is configured to go offline after being idle for a while, and I want to delete pretty much files from a complex directory tree. I can see that some folders have been deleted from the workspace, but heavier ones are left there.

perhaps workspace size is a factor? The workspaces we saw this on come in at just over 6 GB. (Which is why we wanted to be able to use this feature in the first place)

Sorry, cannot reproduce even with MutilpeSCM plugin (Jenkins 1.574, MutipleSCM 0.3, Git 2.2.7, ws clenaup 0.24).
@Marton Sebok haven't tried it yet, but this seem to be a valid concern. Will investigate it deeper. Thanks for a pointer!

@Tom Moore and do you also put slaves offline as pointer out by Marton Sebok? In this case it does make sense for me, but otherwise size of the workspace shouldn't IMHO matter. Do you see any errors in the logs?

No, we don't put the slaves offline. We didn't notice any errors in the logs, the first indication we had that this was a problem was when we ran out of disk space.

It seems that the files we have trouble with are the ones that contain Danish characters æ,ø or å in the file name.

katrine: Are these characters in the workspace (project) name, a folder name in the workspace, or a file name in the workspace? Are you deleting the entire workspace? Did it work on the older Jenkins version? What are the contents of the /systemInfo URL (or /computer/slavename/systemInfo URL for workspaces on slaves)?

The characters are in file names in the workspace.
We delete the entire workspace.
It worked before the upgrade to version 0.24, but I tried downgrading to 0.23 again and that also produced errors now. In 0.23 it now explicitly says that it cannot delete the files:

ERROR: Failed to clean the workspace
09:05:02 java.io.IOException: java.lang.reflect.InvocationTargetException
09:05:02 	at hudson.Util.isSymlinkJava7(Util.java:360)
09:05:02 	at hudson.Util.isSymlink(Util.java:325)
09:05:02 	at hudson.Util.deleteRecursive(Util.java:291)
09:05:02 	at hudson.Util.deleteContentsRecursive(Util.java:203)
09:05:02 	at hudson.Util.deleteRecursive(Util.java:292)
09:05:02 	at hudson.Util.deleteContentsRecursive(Util.java:203)
09:05:02 	at hudson.Util.deleteRecursive(Util.java:292)
09:05:02 	at hudson.Util.deleteContentsRecursive(Util.java:203)
09:05:02 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:393)
09:05:02 	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
09:05:02 	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
09:05:02 	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
09:05:02 	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
09:05:02 	at hudson.remoting.Request$2.run(Request.java:328)
09:05:02 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
09:05:02 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
09:05:02 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
09:05:02 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
09:05:02 	at java.lang.Thread.run(Thread.java:745)
09:05:02 Caused by: java.lang.reflect.InvocationTargetException
09:05:02 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
09:05:02 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
09:05:02 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
09:05:02 	at java.lang.reflect.Method.invoke(Method.java:606)
09:05:02 	at hudson.Util.isSymlinkJava7(Util.java:355)
09:05:02 	... 18 more
09:05:02 Caused by: java.nio.file.InvalidPathException: Malformed input or input contains unmappable chacraters: /var/lib/jenkins/workspace/My-project/sub-folder/prototype/icons/skylleanl??g.svg
09:05:02 	at sun.nio.fs.UnixPath.encode(UnixPath.java:147)
09:05:02 	at sun.nio.fs.UnixPath.<init>(UnixPath.java:71)
09:05:02 	at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:281)
09:05:02 	at java.io.File.toPath(File.java:2186)
09:05:02 	... 23 more

We are using Jenkins 1.565.3 and a Unix slave, version 2.46, java 1.7.0_72

katrine: this seems to be unrelated to the bug discussed in this issue. This is probably different bug and this error is just hidden by async delete. I filed a new issue for this: JENKINS-25317

I expect the deletion can fail for some reason even after we successfully renamed the directory. I see several ways to improve current situation:

• Retry the deletion until it succeeds - it should not block the build but can as well run for unpredictable amount of time
• Introduce a periodic task to clean up such dangling directories
• Pass Future from FilePath.actAsync to some "collector" task that would retry and/or report in case some deletion fails.*

[*] I guess this might be the only way to find out why it failed on the first place as it seems that FilePath.actAsync does not log exceptions.

We added some logging around the deletion (see below) and found that there was a permissions problem.

I would suggest to apply the patch to the plugin so that others can see the root cause as well

From 4dd5c02d5c65a30860ed4ccc2baa860f331f156f Mon Sep 17 00:00:00 2001
From: Victor Volle <vivo@Victors-MacBook-Pro.local>
Date: Tue, 18 Aug 2015 17:53:41 +0200
Subject: [PATCH] JENKINS-24824: log error

---
 src/main/java/hudson/plugins/ws_cleanup/Wipeout.java | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java b/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java
index e1e759b..a9e50a6 100644
--- a/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java
+++ b/src/main/java/hudson/plugins/ws_cleanup/Wipeout.java
@@ -69,7 +69,12 @@ import hudson.remoting.VirtualChannel;
     private final static Command COMMAND = new Command();
     private final static class Command implements FileCallable<Object> {
         public Object invoke(File f, VirtualChannel channel) throws IOException, InterruptedException {
-            Util.deleteRecursive(f);
+            try {
+                Util.deleteRecursive(f);
+            } catch (IOException e) {
+                LOGGER.log(Level.WARNING, "error cleaning up", e);
+                throw e;
+            }
             return null;
         }
     }
-- 
2.3.2 (Apple Git-55)

Code changed in jenkins
User: Oliver Gondža
Path:
src/main/java/hudson/plugins/ws_cleanup/Wipeout.java
http://jenkins-ci.org/commit/ws-cleanup-plugin/e151b89222f93b8b125f3f5e1dffb51e6ad8be9d
Log:
JENKINS-24824 Log deletion failure

Code changed in jenkins
User: Oliver Gondža
Path:
src/main/java/hudson/plugins/ws_cleanup/Wipeout.java
http://jenkins-ci.org/commit/ws-cleanup-plugin/32766a78cf81399fcd0b46f68600863c0369a869
Log:
Merge pull request #22 from olivergondza/JENKINS-24824

JENKINS-24824 Log deletion failure

Compare: https://github.com/jenkinsci/ws-cleanup-plugin/compare/6bc314398d0f...32766a78cf81

Same issue here.

We have one Linux master running multiple slaves, some windows, some Linux.

Linux slaves (VMWare VM) get a revert to snapshot using vmware plugin, which is not possible for windows slaves in our current environment.

To handle hanging processes and locked files, we disconnect each windows slave after 1 single build, shutdown and power on the vm.

Our problem are ever growing workspace folders, allocating far more than 80gig of disk space within about two weeks, as the async cleanup can not run and remove old workspaces due to premature disconnect and termination of the java vm.

In our case the workspace cleanup plugin can rename old workspaces successfully (* -> *ws-cleanup[1234567890]+) , but does not delete those old folders after successfully reconnecting the windows slaves.

We get no errors logged using ws-cleanup plugin version 0.28 in our build logs.

Hi.

Same issue here. Looks like the async delete isn't happening fast enough or isn't happening at all. Most of the time, when our git ran, but the job failed early, the async cleanup did not have time to finish deleting a folder.

Or the other problem is usually an access denied error. Note, not a permission, but access. Meaning, the directory was still being used by some loose thread of a process.

For now, I'm inclined to create a script on the slaves and put it into cron job which runs and cleans up every folder with ws-cleanup in it. But that is not a really nice solution.

EDIT: And yes, same here, usually there is no error involved. Just a [WS-CLEANUP] Deleting project workspace... and no 'Done' in the logs anywhere.

tcb_xy Your use-case is quite special and was not taken into account originally.

skarlso

Looks like the async delete isn't happening fast enough or isn't happening at all. Most of the time, when our git ran, but the job failed early, the async cleanup did not have time to finish deleting a folder.

I do not understand this. How does the problem manifests?

Note the async deletion failure cuses should not go into a build log (as is runs in parallel with the build and can even run for a longer time than the build itself) but a java logger. Check Jenkins master log or configure custom logged in UI for hudson.plugins.ws_cleanup to see why it fails.

In the meantime, JENKINS-27648 was rejected in jenkins core so I guess we have to implement a periodic task to clean dangling *ws-cleanup* dirs. That should resolve the problems with slaves disconnected unexpectedly and directories not deleted in time because of opened file descriptors.

The symptom is a lingering ws-cleanup folder with the timestamp which didn't get deleted. Btw, most of the times it manifests on windows only. Rarely happens on a linux slave interestingly enough.

Yes, sorry, the access denied error message was in the Jenkins Log.

However, the Done, part is in the job's log. And that one was obviously missing because of the exception in the main log.

interestingly enough

Windows file locking?

Windows file locking?

Lacking enough evidence to say yay, or nay. It is 'a' possibility.

Or even some other process locking a certain file. I even saw git lingering and the .git folder could not be deleted in a RENAMED folder. So much crazy, such wow.

some other process locking a certain file. I even saw git lingering

Sorry, that's what I meant. Too much locking going on and Windows doesn't allow deleting folders with open handles.

Affirmative. When it happens, it's usually because of windows file locking.

I have a reproduce that can trigger this.. not the same as the classic slave setup but still.
Provision a docker slave that will be provisioned with the 'One retention strategy'
have a job that executes in the docker container (my job has less than 100 mb data)
perform ws clean-up on success, fail etc.
now once the job completes the docker container is killed and the wc-cleanup_timestamp folder remains
this job runs on linux, so no relation to windows here
one option could be a checkbox to disable assync delete
the /home/jenkins/workspace folder is mounted in the docker container using data volumes, therfore the data remains on the filesystem until we destroy the data volumen

The workaround for this is not to delete the workspace after building but prior to building.

We get this all the time - for out test jobs, there are hundreds of cleanup folders appearing on our disk. This is Ubuntu 14.04.1 64-bit slave, so at least for us it is not related to windows file locking. Also, we have had "Delete workspace before build starts" selected, but it has not fixed the issue.

Our Jenkins version is 1.632, cleanup plugin 0.26.

Update: reverting to cleanup plugin 0.23, which didn't have asynch deletion, allowed us to see the error: we had a shell script section in the jobs which created a folder having root owner instead of jenkins, so the following runs of a job would fail since there weren't able to delete the previous workspace.

I think what needs to be done for asynch plugin is to fail the build somehow if the deletion fails and report the error. Also, it would be nice if user could select between asynch and synch delete instead of having to downgrade all the way to 0.23.

I have the same behaviour in a non-slave environment, I don't know if this is the same or a new issue.
A lot of directories like workspace_ws-cleanup_1447277714463 get created and never deleted.

In the log of the jenkins job there is the output that everything is ok:
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Done

The job is building node projects and loading npm componentes as well as bower components which result in quite huge directory structures, might that be a problem?

In fact if you want to switch "asynch/synch" mode, you can add Patterns, because this will affect cleanup method :

• Add Patterns : synch mode
• No Patterns : asynch mode

This is just a trick to see the cleanup problem without downgrade to 0.23.

And I have submit a pull request to add a new option to switch between synch and asynch mode without downgrading : https://github.com/jenkinsci/ws-cleanup-plugin/pull/26

tcollignon, thanks for the workaround!

@All, please check the logged (hudson.plugins.ws_cleanup.Wipeout) cause in slave/master log and attach it to this issue.

It seems that people generally prefer to fail the build in case the cleanup fails rather that clutter the slave's workspace in the long run. This does not get well together with asynchronous deletion requirement.

One way I can think of is always use sync approach from post-build step and async only in pre-build step. If the original (renamed) workspace will not be gone by the end of the build, sych deletion will be reattempted and will have a chance to fail the build in case of problems. The advantage is that build can start right away and most of the post build actions will do its job (publish junit, determine result, etc.) before the final cleanup kicks in.

WDYT?

This may be a good thing, why not
I ask myself if it's possible when asynch mode is on post-build, to get error et bring back to jenkins job trace. In this case the boolean "asynchronously" is necessary I Think to let people chosing the strategy.

At a certain point Jenkins considers the build log to be complete and never amended again. The problem is that in theory, the cleanup can take longer than the build itself. We can either not care about the result at all (current implementation), wait for the result at the end of the build (potentially postponing the build completion) or try a compromise - check the status at the end of the build and report failure if there is any but do not wait for the completion (this can not guarantee there will be no dangling workspace directories).

When I think about this further, once we rename the workspace (as we do for async cleanup) it will always require manual cleanup in case of failure as it will not be any build's workspace any longer. When the cleanup fail with sync approach, it leaves the workspace half deleted to be reused by future builds - which is far from optimal too.

I am working on implementing plugin specific periodic task to cleanup all the temporary directories (since more general solution was rejected from core).

ok I see
you work on another plugin to clean periodically or you think added this features in this one? you need some help?

right now you think add option "asynchronously" is not necessary ?

My original idea was the async cleanup will be performance optimization that will be transparent to the user (whoever run the builds). Which it is only partially.

I am extending this plugin with the periodic task. I understand the plugin becomes lot more sophisticated that we hoped it to be so I will implement a Jenkins-wide, property-based killswitch to get back to sync cleanup should it cause further problems.

I still do not think that slave workspace getting full should be a concern of a user (as opposed to instance administrator), so I prefer this over a per-job configuration option.

Ok I agree with you. So I wait for your new implementation

Thank for your answer

Has this issue been resolved? I am still seeing this issue in version 0.25 of plugin. Thanks.

Hi, just wanting to know if there is a timeframe on resolution of this issue. Would like to know just in case it's not in the near future so that I can set up a semi-temporary workaround for the problem (most likely a scheduled cleanup on our windows machines).

+1 to a timeframe for fixing this

I'm thinking of rolling back to 0.23 becuase of this problem.

I have been experiencing this problem on a Windows 7 Enterprise installation. During manual clean-up I noticed that even using the command line rmdir /s I was getting a lot of The directory is not empty errors, meaning I had to run the command twice to complete removal.

I also noticed that a large amount of disk space was being consumed by Windows Search content indexing data. I disabled the Windows Search service and deleted the content indexes. Having done this, I'm no longer getting the errors from rmdir. I can also delete workspaces from the Jenkins 'Wipe Out Current Workspace' feature without problems, which was typically reporting an error previously.

On this system, JENKINS_HOME is at C:\Users\jenkins\.jenkins. We also have an installation of Jenkins on Windows Server 2012 R2, which doesn't suffer from the problem - here, JENKINS_HOME is C:\Jenkins, and the Windows Search feature is not installed.

mdimmick_mnetics

Windows Search content indexing data. I disabled the Windows Search service

Don't run the Windows Search on JENKINS_HOME. Don't run antivirus on JENKINS_HOME. No exceptions.

I am having a second look at this: https://github.com/jenkinsci/ws-cleanup-plugin/pull/28

It implements a administrative monitor that retries the deletion on report failures in UI.

Code changed in jenkins
User: Oliver Gondža
Path:
pom.xml
src/main/java/hudson/plugins/ws_cleanup/Wipeout.java
src/test/java/hudson/plugins/ws_cleanup/CleanupPowermockTest.java
src/test/java/hudson/plugins/ws_cleanup/CleanupTest.java
http://jenkins-ci.org/commit/ws-cleanup-plugin/ccd907188e0489e76ca21a7513843864bab48c90
Log:
[FIXED JENKINS-24824] Collect all asynchronously deleted directories

This took lot longer than I expected. The fix will be in 0.32.

I'm still hitting this on a solaris x86 machine. Seems to work on other OS's

java.io.IOException: Unable to delete '/opt/jenkins/workspace/myjob-Solaris_ws-cleanup_1478635926754/local/.git/refs/remotes/origin'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
at hudson.Util.deleteFile(Util.java:248)
at hudson.FilePath.deleteRecursive(FilePath.java:1209)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
at hudson.FilePath.deleteRecursive(FilePath.java:1200)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
at hudson.FilePath.deleteRecursive(FilePath.java:1200)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
at hudson.FilePath.deleteRecursive(FilePath.java:1200)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
at hudson.FilePath.deleteRecursive(FilePath.java:1200)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
at hudson.FilePath.deleteRecursive(FilePath.java:1200)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
at hudson.FilePath.deleteRecursive(FilePath.java:1200)
at hudson.FilePath.access$1000(FilePath.java:195)
at hudson.FilePath$14.invoke(FilePath.java:1179)
at hudson.FilePath$14.invoke(FilePath.java:1176)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2731)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:332)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at ......remote call to sun04(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1435)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
at hudson.remoting.Channel.call(Channel.java:795)
at hudson.FilePath.act(FilePath.java:985)
at hudson.FilePath.act(FilePath.java:974)
at hudson.FilePath.deleteRecursive(FilePath.java:1176)
at hudson.plugins.ws_cleanup.Wipeout$DisposableImpl.dispose(Wipeout.java:110)
at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer$WorkItem.run(AsyncResourceDisposer.java:254)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.file.DirectoryNotEmptyException: /opt/jenkins/workspace/myjob-Solaris_ws-cleanup_1478635926754/local/.git/refs/remotes/origin
at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
at java.nio.file.Files.deleteIfExists(Files.java:1118)
at hudson.Util.tryOnceDeleteFile(Util.java:287)
at hudson.Util.deleteFile(Util.java:243)
at hudson.FilePath.deleteRecursive(FilePath.java:1209)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
at hudson.FilePath.deleteRecursive(FilePath.java:1200)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
at hudson.FilePath.deleteRecursive(FilePath.java:1200)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
at hudson.FilePath.deleteRecursive(FilePath.java:1200)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
at hudson.FilePath.deleteRecursive(FilePath.java:1200)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
at hudson.FilePath.deleteRecursive(FilePath.java:1200)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1218)
at hudson.FilePath.deleteRecursive(FilePath.java:1200)
at hudson.FilePath.access$1000(FilePath.java:195)
at hudson.FilePath$14.invoke(FilePath.java:1179)
at hudson.FilePath$14.invoke(FilePath.java:1176)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2731)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:332)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
... 1 more
java.io.IOException: Unable to delete '/opt/jenkins/workspace/myjob-Solaris_ws-cleanup_1478635926754/local/.git/refs/remotes/origin'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.

We are having the same issue on CentOS Linux release 7.2.1511 (Core). The plugin version is 0.32.

Directories seem to be marked for cleanup but their owner instead of being jenkins is another user.

Hi, we have the same issue still on CentOS Linux 6 and 7 platforms. We'd like these to just be removed and not reported so that we have to manually remove them. We end up with 100s, 1000s of these.

We have latest plugin versions:
Workspace Cleanup Plugin - 0.32
Resource Disposer Plugin - 0.3
and running jenkins master - 2.19.4

It seems this might be related to the problem we're having of the java heap being used up and then our master fails. I then have to restart master.

thanks
Gregor

gjphilp, the screenshot demonstrates how it is supposed to work. Jenkins tries to delete it but as it fail repeatedly so item is tracked in resource disposer until the directory get fried. It seems to be never in your case. It is reported for you attention as there is something that prevents the directory to be deleted for a long time. The cause needs to be found and eliminated in your case. Hover over the exception message to see the full exception that might contain the clue.

I am still experiencing the problem, and in my case I noticed that the build process is creating soft links that are owned by root. The cleanup process is not able to delete these, so it tracks them in the resource disposer. I am working with the developer to find how those links are being created, but this is something that should be addressed in the plugin as well.

stephan_fenton, note that this has nothing to do with the plugin or async deletion. The build just creates stuff Jenkins agent has no permission to clean.

The only thing I can think of to improve this is turn the async deletion off in case the project has a bad record on workspace deletion so such persistent problems will be presented to jobs owners eventually and not the admins.

I'm experiencing the same issue within an updated Jenkins (2.53) and ws-cleanup plugin (0.32).
I wanted to note that when I'm using the ws-cleanup into a pipeline, is working as expected. But when I'm using it into a regular Maven job it is leaving the famous directory with a timestamp (one per build!) and is filling the workspace of my slaves.

I think that this issue should be re-opened because there are more cases like this one.

I could replicate the issue if you disconnect the agent when the prebuild cleanup it is make, if you do that the temporal folder will never be deleted

gjphilp I had the same problem of having to manually remove hundreds of reports.

curl -s http://${JENKINS_URL}/administrativeMonitor/AsyncResourceDisposer/ -u${UTILITY_USER}:${UTILITY_PW} | tr '"' '\n' | grep 'stop-tracking' | cut -d '-' -f 3 | sort -n | while read ASYNC_THREAD; do curl http://${JENKINS_URL}/administrativeMonitor/AsyncResourceDisposer/stopTracking -u${UTILITY_USER}:${UTILITY_PW} -X POST --data "id=${ASYNC_THREAD}"; done

Heap usage dropped significantly immediately after cleanup of 337 threads. 

Guys, this ticket was resolved year and a half ago. Please, file a separate issue.

But the problem still persists (here too, BTW.). So why file a new issue for the VERY SAME problem? Reopening a ticket is common practice in such cases.

We also have this issue but only on Windows slaves (connected through Cygwin SSH).

Have the same issue. Jenkins 2.121.1 in docker container(workspace on volume) , ws-cleanup 0.34

Guys, this ticket was resolved year and a half ago. Please, file a separate issue.

Opened https://issues.jenkins-ci.org/browse/JENKINS-53579 now can we get this addressed?