Loading...

Type: Bug
Resolution: Unresolved
Priority: Critical
Component/s: core, workflow-multibranch-plugin, ws-cleanup-plugin
Labels:
None
Environment:
Linux
jenkins-2.60.2 LTS

Similar Issues:

Show

(Apologies if not filing this under the right components.)

I've had a problem on a software project lately, whereby we have a multi-branch pipeline built from a Jenkinsfile and during the automated workspace cleanup (either by requesting it from the scm/git plugin or when deleted branches are removed in the background by the pipeline multi-branch job) it fails to delete the workspaces.

Somewhere in this software project, there are some files and folders with somewhat unusual names, in that they use an extended charset. For instance, one project uses a node module with some test folders as follows:

$> ls -la
total 32
drwxr-xr-x 10 i316748 1694527156 340 Aug 8 14:28 .
drwxr-xr-x 14 i316748 1694527156 476 Aug 8 14:28 ..
-rw-r--r-- 1 i316748 1694527156 13 Aug 8 13:58 404.html
-rw-r--r-- 1 i316748 1694527156 5 Aug 8 13:58 a.txt
-rw-r--r-- 1 i316748 1694527156 5 Aug 8 13:58 b.txt
-rw-r--r-- 1 i316748 1694527156 21 Aug 8 13:58 c.js
drwxr-xr-x 4 i316748 1694527156 136 Aug 8 14:28 compress
drwxr-xr-x 4 i316748 1694527156 136 Aug 8 14:28 subdir
drwxr-xr-x 4 i316748 1694527156 136 Aug 8 14:28 subdir_with space
drwxr-xr-x 3 i316748 1694527156 102 Aug 8 14:28 中文

Jenkins will interrupt the entire recursive cleanup of the workspace as soon as it hits this file, and as a result we have remaining stray workspaces. It's sort of ok when it's triggered by usage from the Git or SCM plugins "force clean checkout" or similar options, as at least we can see it blow up (build is failed, error is reported in the console).

However, it's a bit sneakier in the case of the multi-branch pipeline scanner, as it fails to delete silently from a user perspective, and you'll only see the issue if you look at the Jenkins log files or when your sysadmin asks why you have hundreds of workspaces eating away at his precious disk space (which we can debate, but... ).

Anyways, I thought this would be easy, and that simply setting proper locales (LC_ALL, LANG, and so forth...) on the machines hosting my master and slaves, and that giving appropriate properties (file.encoding, sun.jnu.encoding, ...) to the Java processes running them would suffice...

Sadly, that's not the case.

And I still these in my logs, and stray workspaces remain:

WARNING: could not clean up workspace directory /home/jenkins/workspace/-page-search-sans-resultats-KHKRZ5WQ5FWRJIC7XKB7CKLDV5U4MILCU4MQPHZW2J3ABWOUCYWQ on jenkins-slave
java.io.IOException: remote file operation failed: /home/jenkins/workspace/-page-search-sans-resultats-KHKRZ5WQ5FWRJIC7XKB7CKLDV5U4MILCU4MQPHZW2J3ABWOUCYWQ at hudson.remoting.Channel@2404ed4d:jenkins-slave: java.io.IOException: Unable to delete '/home/jenkins/workspace/-page-search-sans-resultats-KHKRZ5WQ5FWRJIC7XKB7CKLDV5U4MILCU4MQPHZW2J3ABWOUCYWQ/hybris/bin/ext-content/npmancillary/resources/npm/node_modules/http-server/node_modules/ecstatic/test/public'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
at hudson.FilePath.act(FilePath.java:994)
at hudson.FilePath.act(FilePath.java:976)
at hudson.FilePath.deleteRecursive(FilePath.java:1178)
at jenkins.branch.WorkspaceLocatorImpl$Deleter$CleanupTask.run(WorkspaceLocatorImpl.java:211)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Unable to delete '/home/jenkins/workspace/-page-search-sans-resultats-KHKRZ5WQ5FWRJIC7XKB7CKLDV5U4MILCU4MQPHZW2J3ABWOUCYWQ/hybris/bin/ext-content/npmancillary/resources/npm/node_modules/http-server/node_modules/ecstatic/test/public'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
at hudson.Util.deleteFile(Util.java:250)
at hudson.FilePath.deleteRecursive(FilePath.java:1211)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.access$1000(FilePath.java:197)
at hudson.FilePath$14.invoke(FilePath.java:1181)
at hudson.FilePath$14.invoke(FilePath.java:1178)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2749)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:336)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at ......remote call to jenkins-slave(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
at hudson.remoting.Channel.call(Channel.java:830)
at hudson.FilePath.act(FilePath.java:987)
... 9 more
Caused by: java.nio.file.DirectoryNotEmptyException: /home/jenkins/workspace/-page-search-sans-resultats-KHKRZ5WQ5FWRJIC7XKB7CKLDV5U4MILCU4MQPHZW2J3ABWOUCYWQ/hybris/bin/ext-content/npmancillary/resources/npm/node_modules/http-server/node_modules/ecstatic/test/public
at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
at java.nio.file.Files.deleteIfExists(Files.java:1165)
at hudson.Util.tryOnceDeleteFile(Util.java:290)
at hudson.Util.deleteFile(Util.java:245)
at hudson.FilePath.deleteRecursive(FilePath.java:1211)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.access$1000(FilePath.java:197)
at hudson.FilePath$14.invoke(FilePath.java:1181)
at hudson.FilePath$14.invoke(FilePath.java:1178)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2749)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:336)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
... 4 more

Note that in this log you don't actually see why it fails, it just complains about the directly not being empty. I know use these env vars and properties on my build machines:

LC_ALL=en_US.UTF-8
LANG=en_US.UTF-8

-Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8

Originally the machines were plainly not configured, and the error was slightly different but I didn't keep the logs. It was, however, more explicit about the fact that it could not handle a malformed name, which led me to these tickets (same error logs):

* https://issues.jenkins-ci.org/browse/JENKINS-33478

* https://issues.jenkins-ci.org/browse/JENKINS-12610 (might actually be a duplicate of this one)

While I can see a benefit in Jenkins being locale and charset dependent on some things as it will help to identify bugs in the products it builds, here it's a bit of an issue as it happens really in the management of the build jobs themselves, and I'd expect admins and users to not have to worry about this.

I'd recommend to rewrite some of the filesystem manipulation code to be a bit more aggressive in its removal approach.

In the meantime, workarounds are doable but a bit ugly and unatural. Again, I'd simply expect not to have to worry about it. E.g.:

* crontabs and scripts to regularly delete workspaces,

* specific steps to implement in a pipeline to take care of it in a more resilient manner, e.g. something akin to:

post { always { sh("violent-workspace-cleanup.sh") } }

relates to

JENKINS-33478 Cleaning or deleting workspace fails with non-ASCII filename

Open

JENKINS-12610 Util.deleteRecursive fails for files using unmappable characters

Open

Details

Description

Attachments

Issue Links

Activity

People

Dates