-
Improvement
-
Resolution: Fixed
-
Minor
-
None
'Lockable Resources' within Jenkins global configuration can grow quickly when lock names are selected which are dynamically created and are transient in nature.
Would be nice to clean-up the global configuration when locks released by removing the named lock value.
[JENKINS-38906] Remove lock resource name from global configuration when lock is released
Removing on release would not be safe. +1 for the transient parameter.
There should be a periodic background task looping on defined resources and removing those marked as transient and not used after a specific amount of time.
Also transient resources should not be displayed in the administration page as it does not make sense to configure them nor remove them.
Non-persistent "transient" locks would not be safe when used with Pipeline jobs, because Pipeline jobs running on other nodes can survive a restart of the Jenkins master. If the transient lock definitions aren't on disk, they won't be there when Jenkins master comes back.
I would bump this issue priority as the jenkins management UI (http://jenkinsurl/configure) becomes almost unusable after a bunch of non lockable resources. For me, the complete load takes 20 seconds, with 200 resources created. Our current workaround is shutting down jenkins, deleting the org.jenkins.plugins.lockableresources.LockableResourcesManager.xml and starting again
seanf witokondoria So... maybe there should be a separate page for locks instead of them being in global configure? (eg something similar to credentials) That way it wouldn't clutter the main configure page. Even on separate page, sounds like it would need to be paginated if there's so huge amounts of items.
It seems a combined approach would be valuable: split config (paginated) and self eraseable (non transient) locks.
IMHO, as a first baby step, a simple "delete all" button wuold suffice
+1 on this. We got so many locks and It's been such a pain to access the Jenkins config.
yeah, this is a significant problem and it is getting worse with time.
I like the idea of a separate configuration page to mitigate the pain it causes when trying to load the main configuration page. It if it easy, paging wouldn't be a terrible idea on that dedicated config page.
witokondoria suggested a "delete all", but that doesn't work well in cases (like mine) where I still have some non-pipeline jobs using lockable resources. AFAIK, non-pipeline jobs won't auto-create resources.
Maybe dynamically created resources could be automatically flagged as such? In conjunction with a "prune dynamic resources" button. Of course, the prune would be smart enough to skip over anything which is currently in use by a running job. In a future iteration maybe it can have an "automatic-prune on schedule" option.
As a shameful workaround:
def res = GlobalConfiguration.all().get(org.jenkins.plugins.lockableresources.LockableResourcesManager.class).resources res.retainAll {it -> it.locked}
A scheduled build with the previous script will wipe every non-locked resource. Whoever needs to keep non-pipeline created resources, tag them it properly and modify the expression to keep resources alive, such as:
def res = GlobalConfiguration.all().get(org.jenkins.plugins.lockableresources.LockableResourcesManager.class).resources
res.retainAll {it -> it.locked || it.labelsContain('handmade')}
Bear in mind that LockableResource methods should be approved if running via a pipeline job.
Why is there no simple unlock(resource) or removeResource(name)? Seems like an obvious thing to implement.
+1, there are cases where for example branch specific locks are required. That causes system config screen fill with thousands of locks.
using the github integration, each PR will create a hand full of these locks, loading the system config now takes 5 minutes; manually deleting won't end.
Please provide a way to remove them again from the pipeline that required it.
Any progress on this?
The routine purge process is really painful. The deeper I get into pipeline build, the more painful this becomes (e.g. the more transient locks)
+1 for transient flag
witokondoria any luck getting that script to work as a jenkins job? I keep getting odd errors when trying to run it as a script. I can run it the script console but not as a regular job in the "Build" section.
taylor01, that script has been working as a pipeline job since 10/17, on several Jenkins instances and versions. Wont the "odd errors" related to metrods needed to become approved via the instance scriptApproval service?
In addition to retainAll, you can use:
def all_lockable_resources = GlobalConfiguration.all().get(org.jenkins.plugins.lockableresources.LockableResourcesManager.class).resources
print all_lockable_resources
all_lockable_resources.removeAll { it.name.contains("whatever_you_want_to_filter") }
Bump - curious if anything can be done about this. Not having the ability to clean up locks that are auto-created seems like more than a "minor" priority... this could quickly balloon out of control.
Thanks a lot for the cleanup suggestion mip
vivek This one might be worth our team taking up, since locks are used rather heavily.
Using code from comments above, I have a Jenkins job that runs weekly to remove the ones that start with certain phrases. This may have race conditions (see later comments):
stage ("JENKINS-38906") { def manager = org.jenkins.plugins.lockableresources.LockableResourcesManager.get() def resources = manager.getResources().findAll { (!it.locked) && ( it.name.startsWith("docker_rpminstalled") || it.name.startsWith("docker-rpmbuild") || it.name.startsWith("rpm-deploy") ) } currentBuild.description = "${resources.size()} locks" resources.each { println "Removing ${it.name}" manager.getResources().remove(it) } manager.save() } // stage
aarondmarasco_vsi You should synchronize access to LocalbleResouceManager, otherwise you might cause race condition. Considering LocakbleResourcesManager relies on synchronization, it should actually be a trivial task to delete lock after it has been released.
We're using this to delete locks after our ansible plays:
@NonCPS
def deleteLocks(lockNames) {
def manager = org.jenkins.plugins.lockableresources.LockableResourcesManager.get()
synchronized (manager) {
manager.getResources().removeAll { r -> lockNames.contains(r.name) && !r.locked && !r.reserved }
manager.save()
}
}
Edit: I had time to study problem further. While this does resolve race condition when deleting item from list, it still isn't sufficient. Current lock allocation algorithm relies that locks are not deleted in any way. I think that's something I can fix. However I think that algorithm needs major rework. Everything related lock management has to be done with atomic operations - and to do so, management must be done in a single class. There might be some scalability issues when allocating hundreds of locks, that could be resolved as well.
Thanks skorhone for the heads-up. I don't need to worry about race conditions in my unique situation. However, I'll make a note in case others just see it and copypasta.
This should be fixed with the ephemeral lock support in release 2.6 - Everthing that is created automatically is now removed automatically.
+1. Alternatively, add a flag to the resource that marks it as 'transient' and so is not stored in the global config. Something like: