Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40787

Lockable Resource Lock with inversePrecedence=true not granting lock to newest build

    XMLWordPrintable

Details

    Description

      Lockable resources with inversePrecedence=true is not functioning correctly when used in the steps block of a Declarative Pipeline. The locked resource is locked correctly but when released it does not grant the lock to the newest build but instead the first one that requested it, as if inversePrecedence=false.

      I discovered that if you use the lock in the when block of the Declarative Pipeline the inversePrecedence behaves as expected.

      Attachments

        Issue Links

          Activity

            stefanthurnherr Stefan Thurnherr added a comment - - edited

            Same problem discovered in our multi-branch Jenkinsfile pipeline (not in Declarative Pipeline context but same problem I guess...). The inversePrecedence:true doesn't seem to work reliably. Specifically, I've tried to protect our long-running integration tests with a LIFO queue using the following pipeline script:

            [...] // other stages to build, static code analysis, unit tests etc
            stage('Integration testing') {
                    milestone(label: 'IntegrationTests-AttemptingToStart')
                    String lockResource = "integrationTests-${env.BRANCH_NAME}"
                    lock(resource: lockResource, inversePrecedence: true) {
                        try {
                            integrationTestsStage() //runs different types of integrationTests in parallel, using more lock()s for protecting DB resource
                        } finally {
                            echo "Done with all integration tests."
                            milestone(label:'IntegrationTests-Done') // abort all older builds (for same git branch) that haven't passed this milestone yet
                        }
                    }
            }
            

            It did not work, the builds entered the integrationTests stage in strange orders and sometimes even two builds at the same time.

            Update a few hours later:
            After some more investigation I think the problem is that one cannot have nested lock()s in a pipeline. In our case above, the parallel steps run within integrationTestsStage each hold an additional resource (DB) lock. But whenever one of the inner locks is released, all other locks held by the same build also get released.

            I believe this is not only when using inversePrecedence, so may be a different bug?

            For the reference: Our environment is Jenkins 2.37 with latest versions of all pipeline-related plugins

            stefanthurnherr Stefan Thurnherr added a comment - - edited Same problem discovered in our multi-branch Jenkinsfile pipeline (not in Declarative Pipeline context but same problem I guess...). The inversePrecedence:true doesn't seem to work reliably. Specifically, I've tried to protect our long-running integration tests with a LIFO queue using the following pipeline script: [...] // other stages to build, static code analysis, unit tests etc stage( 'Integration testing' ) { milestone(label: 'IntegrationTests-AttemptingToStart' ) String lockResource = "integrationTests-${env.BRANCH_NAME}" lock(resource: lockResource, inversePrecedence: true ) { try { integrationTestsStage() //runs different types of integrationTests in parallel, using more lock()s for protecting DB resource } finally { echo "Done with all integration tests." milestone(label: 'IntegrationTests-Done' ) // abort all older builds ( for same git branch) that haven't passed this milestone yet } } } It did not work, the builds entered the integrationTests stage in strange orders and sometimes even two builds at the same time. Update a few hours later: After some more investigation I think the problem is that one cannot have nested lock()s in a pipeline. In our case above, the parallel steps run within integrationTestsStage each hold an additional resource (DB) lock. But whenever one of the inner locks is released, all other locks held by the same build also get released. I believe this is not only when using inversePrecedence, so may be a different bug? For the reference: Our environment is Jenkins 2.37 with latest versions of all pipeline-related plugins

            chris_carson Could nested lock() statements be the cause of the your problems described in this issue as well, or should I create a new separate issue for this?

            stefanthurnherr Stefan Thurnherr added a comment - chris_carson Could nested lock() statements be the cause of the your problems described in this issue as well, or should I create a new separate issue for this?
            chris_carson Chris Carson added a comment - - edited

            stefanthurnherr In my case I did not have nested locks. So I would create a new separate issue for that.

            chris_carson Chris Carson added a comment - - edited stefanthurnherr In my case I did not have nested locks. So I would create a new separate issue for that.
            stefanthurnherr Stefan Thurnherr added a comment - Done: JENKINS-41183

            People

              Unassigned Unassigned
              chris_carson Chris Carson
              Votes:
              2 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: