Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43336

Allow locking multiple stages in declarative pipeline

    • Declarative - 1.2

      It would be useful to be able to lock multiple stages as a single lock. For example, we usually have a stage to deploy to an environment and then another stage to run end-to-end tests on that environment, but there should be no other concurrent deployments until both stages have completed.

      Something like this:

      pipeline {
        stages {
          lock(resource: 'myResource', inversePrecedence: true) {
            stage('Deploy') {
              // deploy to environment
            }
      
            stage('E2E') {
              // run tests on the environment
              milestone 1
            }
          }
        }
      }

      Technically both stages could just be merged into a single stage but to me that defeats the purpose of stages.

          [JENKINS-43336] Allow locking multiple stages in declarative pipeline

          Roch Devost created issue -

          Daniel Geißler added a comment - - edited

          There may be some implementations in common with https://issues.jenkins-ci.org/browse/JENKINS-41334 ? As there needs to be a more variable way of structuring stages.

          Daniel Geißler added a comment - - edited There may be some implementations in common with  https://issues.jenkins-ci.org/browse/JENKINS-41334  ? As there needs to be a more variable way of structuring stages.

          dan turner added a comment -

          Hi guys, 

          Is there any chance of this being implemented?

          currently i have a pipeline where i create an apt repo on a linux agent, then pull from it on a different agent. I need those 2 stages to not be interrupted so ideally i would lock around both stages.

          I'm not sure of a way to do this currently?

           

          dan

           

          dan turner added a comment - Hi guys,  Is there any chance of this being implemented? currently i have a pipeline where i create an apt repo on a linux agent, then pull from it on a different agent. I need those 2 stages to not be interrupted so ideally i would lock around both stages. I'm not sure of a way to do this currently?   dan  

          dan turner added a comment -

          for me actually an alternative would be to be able to run steps within a stage on different agents but i'm not sure that makes sense

          dan turner added a comment - for me actually an alternative would be to be able to run steps within a stage on different agents but i'm not sure that makes sense

          Roch Devost added a comment - - edited

          danturn I didn't try it but if I remember correctly you can already run steps on different agents with something like this:

          stage('My Stage') {
            steps {
              node('some label') {
                // run step 1
              }
          
              node('another label') {
                // run step 2
              }
            }
          }

          You probably have to use `agent none` for this to work properly.

          Roch Devost added a comment - - edited danturn I didn't try it but if I remember correctly you can already run steps on different agents with something like this: stage( 'My Stage' ) { steps { node( 'some label' ) { // run step 1 } node( 'another label' ) { // run step 2 } } } You probably have to use `agent none` for this to work properly.

          dan turner added a comment -

          ooh ok, that would solve my problem
           
          i'll try it and update
           
          thanks

          dan turner added a comment - ooh ok, that would solve my problem   i'll try it and update   thanks

          dan turner added a comment -

          that works, thank you very much Roch!!

          dan turner added a comment - that works, thank you very much Roch!!

          Mateusz Was added a comment - - edited

          Alternatively it would be great if we could do something like lock() somewhere and unlock() in another place in pipeline.

          Example (similar to my own case):

          [...]
          def selectedNode = utils.selectAvailableResource()
          
          pipeline {
            agent none
              stages {
                stage('Checkout code') {
                  agent { label selectedNode }
                  steps {
                    lock('aResource')
                  }
                }
                stage('Build') {
                  agent { label selectedNode }
                  steps {
                    echo 'Still locked'
                  }
                }
                stage('Integration tests') {
                  agent { label selectedNode }
                  steps {
                    echo 'Still locked'
                  }
                }
                stage('Cleanup') {
                  agent { label selectedNode }
                  steps {
                    unlock('aResource')
                    echo 'Finally unlocked'
                  }
                }
              }
            }
          }

          Mateusz Was added a comment - - edited Alternatively it would be great if we could do something like lock() somewhere and unlock() in another place in pipeline. Example (similar to my own case): [...] def selectedNode = utils.selectAvailableResource() pipeline { agent none stages { stage( 'Checkout code' ) { agent { label selectedNode } steps { lock( 'aResource' ) } } stage( 'Build' ) { agent { label selectedNode } steps { echo 'Still locked' } } stage( 'Integration tests' ) { agent { label selectedNode } steps { echo 'Still locked' } } stage( 'Cleanup' ) { agent { label selectedNode } steps { unlock( 'aResource' ) echo 'Finally unlocked' } } } } }

          What matt_was suggested would open up many exiting possibilities.

          For example locking a resource prior to acquiring a node (best practice), but releasing it while retaining the build slot and workspace for further processing that does not need any locks.

          // acquire some kind of test resources
          def locks = lock(label: 'aResource', quantity: 1)
          node {
              stage('build'){
                 sh "mvn install -D${locks[0].name}"
                 unlock(locks)
              }
              // continoue processing without having to leave the node
              stage('analysis'){
                  sh "mvn sonar:sonar"
              }
          }

          having that return value would be awesome too (to ask for the name of a resource and possibly more properties), but this is discussed in other tickets.

          Daniel Geißler added a comment - What matt_was suggested would open up many exiting possibilities. For example locking a resource prior to acquiring a node (best practice), but releasing it while retaining the build slot and workspace for further processing that does not need any locks. // acquire some kind of test resources def locks = lock(label: 'aResource' , quantity: 1) node { stage( 'build' ){ sh "mvn install -D${locks[0].name}" unlock(locks) } // continoue processing without having to leave the node stage( 'analysis' ){ sh "mvn sonar:sonar" } } having that return value would be awesome too (to ask for the name of a resource and possibly more properties), but this is discussed in other tickets.

          David Crouch added a comment -

          This is the single issue that is preventing us from moving to declarative pipeline. We currently use scripted Jenkinsfiles for exactly this purpose: to be able to define multiple stages which can lock a resource atomically.

          In our case, we perform embedded hardware testing which requires the use of hardware resources to run testing. Because there are several stages in our pipeline which interact with the hardware (resetting, provisioning, testing, etc), the ability to lock across multiple stages is a necessity.

          I do like the concept of being able to control "lock" versus "unlock". I think that would solve the problem sufficiently for us, and open up new possibilities.

          David Crouch added a comment - This is the single issue that is preventing us from moving to declarative pipeline. We currently use scripted Jenkinsfiles for exactly this purpose: to be able to define multiple stages which can lock a resource atomically. In our case, we perform embedded hardware testing which requires the use of hardware resources to run testing. Because there are several stages in our pipeline which interact with the hardware (resetting, provisioning, testing, etc), the ability to lock across multiple stages is a necessity. I do like the concept of being able to control "lock" versus "unlock". I think that would solve the problem sufficiently for us, and open up new possibilities.

            abayer Andrew Bayer
            rochdev Roch Devost
            Votes:
            19 Vote for this issue
            Watchers:
            40 Start watching this issue

              Created:
              Updated:
              Resolved: