Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
Jenkins 2.39
lockable-resources-plugin 1.11
Description
My pipeline consists of several exclusive sections like this, which should never be executed in parallel:
milestone() lock(resource: "unit-tests", inversePrecedence: true) { milestone() stage('Unit Tests') { } } milestone() lock(resource: "integration-tests", inversePrecedence: true) { milestone() stage('Integration Tests') { } }
This worked perfectly fine with lockable-resources-plugin 1.10.
However with version 1.11 the locking does not work properly anymore.
It happens that two builds are running the integration tests in parallel,
although this should not be possible given the above configuration.
Whenever this happens I have a look at the http://<jenkins>/lockable-resources overview.
It shows that the resource is locked by only one build, although two builds are just concurrently executing the integration tests.
I am afraid it has to do with the recently implemented features:
JENKINS-34268- lock multiple resources concurrentlyJENKINS-34273- add the number of resources lo lock from a given label
Since I consider this a bug and regression, I changed the priority to critical.
Attachments
Issue Links
- is duplicated by
-
-
- Resolved
-
-
-
- Resolved
-
-
-
- Closed
-
- links to
Activity
Phil McArdle
added a comment - I believe I hit the same issue. I have multiple stages queued to run in parallel, with some of them waiting on the same locks.
My experience was that when any lock was released, all the locks seemed to be.
Lock acquired on resource 1 {
Lock acquired on resource 2 {
}
Lock acquired on resource 1 {
}
}
In my case, a second lock was being acquired on resource 1 long before the log or the pipeline steps view showed it being released.
I've downgraded to 1.10 and it runs correctly. I'm not using inversePrecedence, quantity or label.
Phil McArdle
added a comment - I believe I hit the same issue. I have multiple stages queued to run in parallel, with some of them waiting on the same locks.
My experience was that when any lock was released, all the locks seemed to be.
Lock acquired on resource 1 {
Lock acquired on resource 2 {
}
Lock acquired on resource 1 {
}
}
In my case, a second lock was being acquired on resource 1 long before the log or the pipeline steps view showed it being released.
I've downgraded to 1.10 and it runs correctly. I'm not using inversePrecedence, quantity or label. PR up at https://github.com/jenkinsci/lockable-resources-plugin/pull/45, including a first commit with a test that seems to consistently reproduces the issue.
Code changed in jenkins
User: Andrew Bayer
Path:
src/test/java/org/jenkins/plugins/lockableresources/LockStepTest.java
http://jenkins-ci.org/commit/lockable-resources-plugin/19119d6cbb0af2057550d6886d1239b34d54b169
Log:
JENKINS-40879 Test to verify bug.
This test seems to fail pretty consistently - not 100% sure it'll keep
failing every time, but I haven't seen it pass yet except with a fix.
Code changed in jenkins
User: Andrew Bayer
Path:
src/main/java/org/jenkins/plugins/lockableresources/LockableResourcesManager.java
http://jenkins-ci.org/commit/lockable-resources-plugin/a7cc1aa694beed617aae302a7e04e770c81ec7aa
Log:
[FIXED JENKINS-40879] Only call proceed with resources to lock.
Previously, we were calling LockStepExecution.proceed with the full
list of resources - which was resulting in everything getting
unlocked if there was a next context, i.e., if anything else was
waiting on a resource to be unlocked. That was wrong, obviously.
This changes to instead only call proceed with the list of resource
names we actually need to lock for the next context, which fixes the
test in the previous commit.
I am also experiencing this bug. I am running 3 instances of the same pipeline job in parallel (build1, build2, build3), all in contention for Resource 'A'. build1 acquires 'A', the next 2 wait. When the first job releases Resource A (via a Java exception in the groovy script), both build2 and build3 log that they have acquired the resource and execute the critical section in parallel. Chaos ensues.