-
Bug
-
Resolution: Fixed
-
Critical
-
None
-
Jenkins ver. 2.32.3, workspace cleanup plugin 1.0.5
if we run a lot of copies of the same job with workspace cleanup enabled - every next job will clean up workspaces on slaves with already running jobs. jobs are executed in parallel, so perhaps it's better to check if workspace is used currently and belongs to running job before the cleanup. it's important for us because we have ENOSPACE with cleanup disabled and many failed jobs with cleanup enabled. running job copies one-by-one is not an option for us, because it will take a lot of time.
thanks in advance
- is related to
-
JENKINS-58492 Improve robustness w.r.t. bad slave nodes
-
- Resolved
-
-
-
- Resolved
-
-
-
- Closed
-
[JENKINS-43269] workspace cleanup removes workspaces for running job
- we don't have any custom workspaces
- no. we have a lot of slaves, every slave is able to build the job
- no
- yes, from failed jobs. we have cleanup disabled at the moment, so no fresh logs. usually it looks like {code}+ mv /mnt/workspace/percona-server-5.6-param/BUILD_TYPE/debug/Host/centos5-64/mysql-test/test_results.subunit /mnt/workspace/percona-server-5.6-param/BUILD_TYPE/debug/Host/centos5-64
mv: cannot stat `/mnt/workspace/percona-server-5.6-param/BUILD_TYPE/debug/Host/centos5-64/mysql-test/test_results.subunit': No such file or directory{code} suddenly workspace disappears. - workspaces set to be cleaned up at the start of build. so, it looks like next started copy will destroy already running jobs
- because such a failures disappeared after disabling the cleanup
Uh, that is for https://wiki.jenkins-ci.org/display/JENKINS/Distributed+Workspace+Clean+plugin. I have fixed the component name and the wiki page so users can find what the component name is.
pjdarton
added a comment - I've hit this issue before myself.
I think it's down to this line where it's only considering the build which is running that build step and not considering the possibility that Jenkins might be running multiple runs of that build job in parallel on different slave nodes.
The repro case is fairly simple:
- Have multiple static slave nodes.
- Define a job with the "allow concurrent execution" flag set so you can have two (or more) builds of that job running at the same time, and ensure that the "restrict where build can run" labels are set to allow it to run on more than one of the static slave nodes.
- Trigger a build, wait until it's running, then trigger another. That'll result in two builds running in parallel, one on one slave node, one on another.
- When the first build completes, it'll run the cleanup phase and try to delete the workspace that's currently in use by the second build.
pjdarton
added a comment - I've hit this issue before myself.
I think it's down to this line where it's only considering the build which is running that build step and not considering the possibility that Jenkins might be running multiple runs of that build job in parallel on different slave nodes.
The repro case is fairly simple:
Have multiple static slave nodes.
Define a job with the "allow concurrent execution" flag set so you can have two (or more) builds of that job running at the same time, and ensure that the "restrict where build can run" labels are set to allow it to run on more than one of the static slave nodes.
Trigger a build, wait until it's running, then trigger another. That'll result in two builds running in parallel, one on one slave node, one on another.
When the first build completes, it'll run the cleanup phase and try to delete the workspace that's currently in use by the second build.
pjdarton
added a comment - I've created a PR that'll address this issue ... but until the plugin code has a Jenkinsfile (see PR#5) folks will have to build their own copy of the code.
pjdarton
added a comment - I've created a PR that'll address this issue ... but until the plugin code has a Jenkinsfile (see PR#5 ) folks will have to build their own copy of the code. pjdarton
added a comment - FAO tspengler and/or aheritier - you're both listed (in the pom.xml file) as developers of this plugin - could you take a look at the PRs I've raised?
pjdarton
added a comment - FAO tspengler and/or aheritier - you're both listed (in the pom.xml file) as developers of this plugin - could you take a look at the PRs I've raised? pjdarton I'm not active anymore on this plugin and many others.
I won't have the time to look at it before at least a month.
Could you consider to become a maintainer of this plugin ?
I will +1 your candidature
pjdarton
added a comment - abychko I've uploaded an enhanced hudson-wsclean-plugin.hpi
file.
Try ticking the "use history" check-box (in the "Advanced" config). That'll change the plugin's behavior from "hit all slaves that could run this project except the one running this build right now" to "hit all slaves that previously ran this build except any running this build right now".
That should allow you to have concurrent builds without a problem.
Note: By default, it'll continue to use the old "hit all slaves that could've run this project" logic which is largely unchanged and will still bork a concurrent build - you'll need to select the new behavior in your job config.
If you still have issues, create a log (manage jenkins -> system log) that logs de.jamba.hudson.plugin.wsclean at level "ALL", collect logs from that to see what it's doing wrong (as this code now logs what it's doing to the Jenkins logger as well as to the builds), and let me know further details.
pjdarton
added a comment - abychko I've uploaded an enhanced hudson-wsclean-plugin.hpi file.
Try ticking the "use history" check-box (in the "Advanced" config). That'll change the plugin's behavior from "hit all slaves that could run this project except the one running this build right now" to "hit all slaves that previously ran this build except any running this build right now".
That should allow you to have concurrent builds without a problem.
Note: By default, it'll continue to use the old "hit all slaves that could've run this project" logic which is largely unchanged and will still bork a concurrent build - you'll need to select the new behavior in your job config.
If you still have issues, create a log (manage jenkins -> system log) that logs de.jamba.hudson.plugin.wsclean at level "ALL", collect logs from that to see what it's doing wrong (as this code now logs what it's doing to the Jenkins logger as well as to the builds), and let me know further details. pjdarton
added a comment - aheritier I guess I could, although I'd prefer not to stay a maintainer long-term (I seem to adopt them much faster than I can let them go, and my boss would prefer me to get on with other things...)
So, sure, I'll take over long enough to kick out a new release with the functionality I want to add, but I don't want to stay involved if I can avoid doing so 
In the meantime, can you merge PR#5 ? It should just be a simple button-click to do that and that'll then get the jenkinsci build engine to notice the plugin and start doing CI builds of the PRs...
pjdarton
added a comment - aheritier I guess I could, although I'd prefer not to stay a maintainer long-term (I seem to adopt them much faster than I can let them go, and my boss would prefer me to get on with other things...)
So, sure, I'll take over long enough to kick out a new release with the functionality I want to add, but I don't want to stay involved if I can avoid doing so
In the meantime, can you merge PR#5 ? It should just be a simple button-click to do that and that'll then get the jenkinsci build engine to notice the plugin and start doing CI builds of the PRs... pjdarton
added a comment - aheritier I've requested access.
While I've linked back to your earlier approval in the comments here, if you could "approve" that PR then it may help speed things along.
Similarly, if you can grant write access to the github area then that would also be helpful.
pjdarton
added a comment - aheritier I've requested access .
While I've linked back to your earlier approval in the comments here, if you could "approve" that PR then it may help speed things along.
Similarly, if you can grant write access to the github area then that would also be helpful. pjdarton
added a comment - Update for anyone watching this:
There's now a Pull Request that contains a fix for this issue (plus other enhancements). Anyone can download the .hpi file of the bugfixed plugin from there and then upload that (Manage Jenkins -> Manage Plugins ->Advanced -> Upload plugin) to their own Jenkins server(s) to try it out.
It would be very helpful if everyone watching this could download and test that these changes fix the issue (and don't introduce any other big problems).
You're all welcome to take a look at the code changes and comment on there too, if you're so inclined.
Once I'm confident that everything is OK then I'll merge those changes in and release the new plugin officially.
pjdarton
added a comment - Update for anyone watching this:
There's now a Pull Request that contains a fix for this issue (plus other enhancements). Anyone can download the .hpi file of the bugfixed plugin from there and then upload that (Manage Jenkins -> Manage Plugins ->Advanced -> Upload plugin) to their own Jenkins server(s) to try it out.
It would be very helpful if everyone watching this could download and test that these changes fix the issue (and don't introduce any other big problems).
You're all welcome to take a look at the code changes and comment on there too, if you're so inclined.
Once I'm confident that everything is OK then I'll merge those changes in and release the new plugin officially. 
This certainly should not happen. Please provide more information about the situation (reproducer ideally) as well as plugin version (ws-cleanup was never released in version 1.0.5 as stated).