-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
CloudBees Jenkins Enterprise 2.32.2.1-rolling
lockable resources plugin version 1.x & 2.x
In strained situations, more resources than you asked for might get locked. This creates a bottleneck problem related to resources but it cannot be solved by adding more resources.
In worst cases will always lock all resources with a certain label instead of the required amount.
You can recreate the issue by configuring a couple (say 4) of lockable resources with a tag. Trigger a bunch of these jobs with NUM_RESOURCES between 1 - 4 to build up a queue of jobs waiting for their resources to become available. Eventually the error will start appearing.
properties([
parameters ([
string(name: 'NUM_RESOURCES')
])
]);
node {
def askFor = params.NUM_RESOURCES as Integer
lock(label: "FOO", quantity: askFor) {
def clients = org.jenkins.plugins.lockableresources.LockableResourcesManager.class.get().getResourcesFromBuild(currentBuild.getRawBuild())
echo("Asked for ${askFor} resources, got ${clients.size()}: ${clients.toString()}")
sleep (60 * 10)
if (askFor != clients.size()) {
error ("Wrong amount of resources acquired")
}
}
}
- links to
[JENKINS-46494] Certain conditions might lock more resources than required quantity
I think I just stumbled over this issue as well and the change in linked pull request https://github.com/jenkinsci/lockable-resources-plugin/pull/66 actually looks reasonable/applicable at a quick naive glance...
So here is my situation:
- Started playing around with lock based on 'label' (2 lockable resources with same label), but forgot to specify 'quantity' => so it reserved all available 2 resources (with this label)
- Fixing the problem by specifying 'quantity: 1'
- Triggering a new pipeline build with the fix, WHILE an old pipeline build WITHOUT THE FIX was still running AND was STILL LOCKING ALL 2 RESOURCES:
- The old build (#793, 20180417-133734-revUNKNOWN) WITHOUT the fix:
13:38:43 Trying to acquire lock on [Label: PhpStorm] 13:38:43 Found 0 available resource(s). Waiting for correct amount: 2. 13:38:43 [Label: PhpStorm] is locked, waiting... 13:41:36 Lock acquired on [Label: PhpStorm] ... 13:46:29 Lock released on resource [Label: PhpStorm]
- The new build (#794, 20180417-134149-revUNKNOWN) WITH the fix and right (lying logs) and wrong behaviour:
13:45:36 Trying to acquire lock on [Label: PhpStorm, Quantity: 1] 13:45:36 Found 0 available resource(s). Waiting for correct amount: 1. 13:45:36 [Label: PhpStorm, Quantity: 1] is locked, waiting... 13:46:29 Lock acquired on [Label: PhpStorm, Quantity: 1] ...
- The old build (#793, 20180417-133734-revUNKNOWN) WITHOUT the fix:
- So the new pipeline build had to correctly wait for the releasing of the lock, but it then incorrectly locked again for all 2 resources instead of just 1:
- Lockable Resources view:
- Lockable Resources view:
- Subsequent builds – after a coincidental pause/break that seemingly allowed to release all locks and give them completely back, instead of re-using them for queued/waiting builds – seemed to behave correctly then (and just lock 1 resource)
Reinhold Füreder
added a comment - - edited I think I just stumbled over this issue as well and the change in linked pull request https://github.com/jenkinsci/lockable-resources-plugin/pull/66 actually looks reasonable/applicable at a quick naive glance...
So here is my situation:
Started playing around with lock based on 'label' (2 lockable resources with same label), but forgot to specify 'quantity' => so it reserved all available 2 resources (with this label)
Fixing the problem by specifying 'quantity: 1'
Triggering a new pipeline build with the fix, WHILE an old pipeline build WITHOUT THE FIX was still running AND was STILL LOCKING ALL 2 RESOURCES:
The old build (#793, 20180417-133734-revUNKNOWN) WITHOUT the fix:
13:38:43 Trying to acquire lock on [Label: PhpStorm]
13:38:43 Found 0 available resource(s). Waiting for correct amount: 2.
13:38:43 [Label: PhpStorm] is locked, waiting...
13:41:36 Lock acquired on [Label: PhpStorm]
...
13:46:29 Lock released on resource [Label: PhpStorm]
The new build (#794, 20180417-134149-revUNKNOWN) WITH the fix and right (lying logs) and wrong behaviour:
13:45:36 Trying to acquire lock on [Label: PhpStorm, Quantity: 1]
13:45:36 Found 0 available resource(s). Waiting for correct amount: 1.
13:45:36 [Label: PhpStorm, Quantity: 1] is locked, waiting...
13:46:29 Lock acquired on [Label: PhpStorm, Quantity: 1]
...
So the new pipeline build had to correctly wait for the releasing of the lock, but it then incorrectly locked again for all 2 resources instead of just 1:
Lockable Resources view:
Subsequent builds – after a coincidental pause/break that seemingly allowed to release all locks and give them completely back, instead of re-using them for queued/waiting builds – seemed to behave correctly then (and just lock 1 resource)
Reinhold Füreder
added a comment - amuniz Since the github activities of mantonsson seem to indicate that he is unfortunately no longer active, could you (as one of the plugin owners) maybe take over this bug – in the hope that it is a low hanging fruit and the linked pull request actually makes really sense...
Reinhold Füreder
added a comment - amuniz Since the github activities of mantonsson seem to indicate that he is unfortunately no longer active, could you (as one of the plugin owners) maybe take over this bug – in the hope that it is a low hanging fruit and the linked pull request actually makes really sense... 
Marcus Antonsson
added a comment - reinholdfuereder It's not a very active account but I'm still active. 
I've unassigned myself and stopped the progress for now (unsure about the workflow here). Someone with the right privilege need to review the pull request and merge it for release.
Marcus Antonsson
added a comment - reinholdfuereder It's not a very active account but I'm still active.
I've unassigned myself and stopped the progress for now (unsure about the workflow here). Someone with the right privilege need to review the pull request and merge it for release. Reinhold Füreder
added a comment - mantonsson Sorry for my harsh words 
and thanks for the clarification... and of course thanks for the pull request...
Reinhold Füreder
added a comment - mantonsson Sorry for my harsh words and thanks for the clarification... and of course thanks for the pull request... 
Seeing this issue massively.
We have 9 labelled resources.
Build 1 reserves 4.
Build 2 reserves 4.
Build 3 wants to reserve 4, and waits.
Build 1 finishes and unlocks.
Now the bug: Build 3 locks 5, although in log it says it locks 4.
I'll happily play beta tester for a .hpi, this is bugging us quite much (we also often see builds reserve 4 to 5 when they only request 1, and this is really blocking)