Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41183

Pipeline/Jenkinsfile: Nested lock() statements not working as expected

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Duplicate
    • Labels:
      None
    • Environment:
      Jenkins v2.37 with latest pipeline plugins, lockable resources plugin v1.11, running on JDK8
    • Similar Issues:

      Description

      Nested lock() statements in a Jenkinsfile pipeline do now work as expected. Here is an example use case (Jenkinsfile with git multi-branch pipeline) of what I tried to do :

      stage('Main build') {
        // fetch from git, build, unit tests etc.
      }
      stage('Integration testing') {
        milestone(label: 'IntegrationTests-AttemptingToStart')
        lock(resource: "integrationTests-${env.BRANCH_NAME}", inversePrecedence: true) {
          integrationTestsStage()
          milestone(label: 'IntegrationTests-Done')
        }
      }
      
      def integrationTestsStage() {
        def pipelineRunBranches = [:]
      
        pipelineRunBranches['itests-noDbData'] = {
          lock(resource: 'dbResourceLock-noDataPopulated') {
            node {
              // run lots of itests against DB with no data populated
            }
          }
        }
      
        pipelineRunBranches['itests-withDbData'] = {
          lock(resource: 'dbResourceLock-withDataPopulated') {
            node {
              // run some heavier itests against DB with test data populated
            }
          }
        }
      
        parallel pipelineRunBranches
      }
      

      Actual outcome is that as soon as one of the two parallel branches of the integration tests stage is done and releases its lock, then any build from the same branch waiting to enter the integration tests stage will acquire the lock and start running integration tests, although the first build is still running the other parallel branch of its integration tests stage.

      Workaround is to only use one single lockable resource (the outer one in our case), and making it a global instead of a per-branch lock (same resource label for all branches). This works fine, the only disadvantage is that we now have a global LIFO queue (instead of a per-branch LIFO queue) to enter the integration tests stage. This can cause some unexpected long delays for some builds.

      My guess is that the Lockable Resources Plugin" was originally designed for UI-configured Jenkins jobs (not scripted). Then one could only define required lockable resources per job, and all lockable resources would always be acquired at the start of a build, and be released at the end of the build. So releasing one lockable resources in that case is identical to releasing all lockable resources (happens at the same time anyway).

      I first thought that the issue has to do with using inversePrecedence: true - see also JENKINS-40787. But removing it didn't help in the example above.

        Attachments

          Issue Links

            Activity

            Hide
            adriankirchner Adrian Kirchner added a comment -

            I encountered the exact same problem in a multibranch pipeline job running three different branch jobs in parallel.

            Jenkins 2.41
            Lockable Resources plugin 1.11

            Show
            adriankirchner Adrian Kirchner added a comment - I encountered the exact same problem in a multibranch pipeline job running three different branch jobs in parallel. Jenkins 2.41 Lockable Resources plugin 1.11
            Hide
            marknysen Mark Nysen added a comment -

            I found the same problem and raised JENKINS-41868 against it. This seems to be a bug introduced in Lockable Resources 1.11. Rolling back to 1.10 fixed it for me. I suspect that the added clean up code is getting executed by the exiting of parallel jobs and it is removing locks that it shouldn't be removing. It removes the lock it should remove and also parent locks (nested) and sibling locks (from other tasks in the same job).

            It may be worth checking if rolling back to 1.10 fixes this for you too.

            Show
            marknysen Mark Nysen added a comment - I found the same problem and raised JENKINS-41868 against it. This seems to be a bug introduced in Lockable Resources 1.11. Rolling back to 1.10 fixed it for me. I suspect that the added clean up code is getting executed by the exiting of parallel jobs and it is removing locks that it shouldn't be removing. It removes the lock it should remove and also parent locks (nested) and sibling locks (from other tasks in the same job). It may be worth checking if rolling back to 1.10 fixes this for you too.
            Hide
            abayer Andrew Bayer added a comment -

            I suspect that this, like JENKINS-41868, is a duplicate of JENKINS-40879, but I'm waiting til I have a consistently reproducible test case for JENKINS-40879 and a fix before adding a test case for this as well.

            Show
            abayer Andrew Bayer added a comment - I suspect that this, like JENKINS-41868 , is a duplicate of JENKINS-40879 , but I'm waiting til I have a consistently reproducible test case for JENKINS-40879 and a fix before adding a test case for this as well.
            Hide
            abayer Andrew Bayer added a comment -

            On closer reading and inspection, yeah, this is a dupe of JENKINS-40879 - it's not the nesting that's at issue, it's the multiple locks.

            Show
            abayer Andrew Bayer added a comment - On closer reading and inspection, yeah, this is a dupe of JENKINS-40879 - it's not the nesting that's at issue, it's the multiple locks.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              stefanthurnherr Stefan Thurnherr
              Votes:
              3 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: