Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40787

Lockable Resource Lock with inversePrecedence=true not granting lock to newest build

XMLWordPrintable

      Lockable resources with inversePrecedence=true is not functioning correctly when used in the steps block of a Declarative Pipeline. The locked resource is locked correctly but when released it does not grant the lock to the newest build but instead the first one that requested it, as if inversePrecedence=false.

      I discovered that if you use the lock in the when block of the Declarative Pipeline the inversePrecedence behaves as expected.

          [JENKINS-40787] Lockable Resource Lock with inversePrecedence=true not granting lock to newest build

          Stefan Thurnherr added a comment - - edited

          Same problem discovered in our multi-branch Jenkinsfile pipeline (not in Declarative Pipeline context but same problem I guess...). The inversePrecedence:true doesn't seem to work reliably. Specifically, I've tried to protect our long-running integration tests with a LIFO queue using the following pipeline script:

          [...] // other stages to build, static code analysis, unit tests etc
stage('Integration testing') {
        milestone(label: 'IntegrationTests-AttemptingToStart')
        String lockResource = "integrationTests-${env.BRANCH_NAME}"
        lock(resource: lockResource, inversePrecedence: true) {
            try {
                integrationTestsStage() //runs different types of integrationTests in parallel, using more lock()s for protecting DB resource
            } finally {
                echo "Done with all integration tests."
                milestone(label:'IntegrationTests-Done') // abort all older builds (for same git branch) that haven't passed this milestone yet
            }
        }
}

          It did not work, the builds entered the integrationTests stage in strange orders and sometimes even two builds at the same time.

          Update a few hours later:
          After some more investigation I think the problem is that one cannot have nested lock()s in a pipeline. In our case above, the parallel steps run within integrationTestsStage each hold an additional resource (DB) lock. But whenever one of the inner locks is released, all other locks held by the same build also get released.

          I believe this is not only when using inversePrecedence, so may be a different bug?

          For the reference: Our environment is Jenkins 2.37 with latest versions of all pipeline-related plugins

          Stefan Thurnherr added a comment - - edited Same problem discovered in our multi-branch Jenkinsfile pipeline (not in Declarative Pipeline context but same problem I guess...). The inversePrecedence:true doesn't seem to work reliably. Specifically, I've tried to protect our long-running integration tests with a LIFO queue using the following pipeline script: [...] // other stages to build, static code analysis, unit tests etc stage( 'Integration testing' ) { milestone(label: 'IntegrationTests-AttemptingToStart' ) String lockResource = "integrationTests-${env.BRANCH_NAME}" lock(resource: lockResource, inversePrecedence: true ) { try { integrationTestsStage() //runs different types of integrationTests in parallel, using more lock()s for protecting DB resource } finally { echo "Done with all integration tests." milestone(label: 'IntegrationTests-Done' ) // abort all older builds ( for same git branch) that haven't passed this milestone yet } } } It did not work, the builds entered the integrationTests stage in strange orders and sometimes even two builds at the same time. Update a few hours later: After some more investigation I think the problem is that one cannot have nested lock()s in a pipeline. In our case above, the parallel steps run within integrationTestsStage each hold an additional resource (DB) lock. But whenever one of the inner locks is released, all other locks held by the same build also get released. I believe this is not only when using inversePrecedence, so may be a different bug? For the reference: Our environment is Jenkins 2.37 with latest versions of all pipeline-related plugins

          Stefan Thurnherr added a comment -

          chris_carson Could nested lock() statements be the cause of the your problems described in this issue as well, or should I create a new separate issue for this?

          Stefan Thurnherr added a comment - chris_carson Could nested lock() statements be the cause of the your problems described in this issue as well, or should I create a new separate issue for this?

          Chris Carson added a comment - - edited

          stefanthurnherr In my case I did not have nested locks. So I would create a new separate issue for that.

          Chris Carson added a comment - - edited stefanthurnherr In my case I did not have nested locks. So I would create a new separate issue for that.

          Stefan Thurnherr added a comment -

          Done: JENKINS-41183

          Stefan Thurnherr added a comment - Done: JENKINS-41183

            Unassigned Unassigned
            chris_carson Chris Carson
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: