Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41183

Pipeline/Jenkinsfile: Nested lock() statements not working as expected

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Major Major
    • None
    • Jenkins v2.37 with latest pipeline plugins, lockable resources plugin v1.11, running on JDK8

      Nested lock() statements in a Jenkinsfile pipeline do now work as expected. Here is an example use case (Jenkinsfile with git multi-branch pipeline) of what I tried to do :

      stage('Main build') {
        // fetch from git, build, unit tests etc.
      }
      stage('Integration testing') {
        milestone(label: 'IntegrationTests-AttemptingToStart')
        lock(resource: "integrationTests-${env.BRANCH_NAME}", inversePrecedence: true) {
          integrationTestsStage()
          milestone(label: 'IntegrationTests-Done')
        }
      }
      
      def integrationTestsStage() {
        def pipelineRunBranches = [:]
      
        pipelineRunBranches['itests-noDbData'] = {
          lock(resource: 'dbResourceLock-noDataPopulated') {
            node {
              // run lots of itests against DB with no data populated
            }
          }
        }
      
        pipelineRunBranches['itests-withDbData'] = {
          lock(resource: 'dbResourceLock-withDataPopulated') {
            node {
              // run some heavier itests against DB with test data populated
            }
          }
        }
      
        parallel pipelineRunBranches
      }
      

      Actual outcome is that as soon as one of the two parallel branches of the integration tests stage is done and releases its lock, then any build from the same branch waiting to enter the integration tests stage will acquire the lock and start running integration tests, although the first build is still running the other parallel branch of its integration tests stage.

      Workaround is to only use one single lockable resource (the outer one in our case), and making it a global instead of a per-branch lock (same resource label for all branches). This works fine, the only disadvantage is that we now have a global LIFO queue (instead of a per-branch LIFO queue) to enter the integration tests stage. This can cause some unexpected long delays for some builds.

      My guess is that the Lockable Resources Plugin" was originally designed for UI-configured Jenkins jobs (not scripted). Then one could only define required lockable resources per job, and all lockable resources would always be acquired at the start of a build, and be released at the end of the build. So releasing one lockable resources in that case is identical to releasing all lockable resources (happens at the same time anyway).

      I first thought that the issue has to do with using inversePrecedence: true - see also JENKINS-40787. But removing it didn't help in the example above.

            Unassigned Unassigned
            stefanthurnherr Stefan Thurnherr
            Votes:
            3 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: