Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40966

Ability to leave Declarative node block w/ top-level agent for a single stage

    • Icon: New Feature New Feature
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • pipeline-model-definition 0.8.1

      I have a use case where I'd like to have a specific section of the pipeline limited to one concurrent step across all pipeline instances.

      Jenkinsfile
      pipeline {
        agent {
          docker {
            label 'label'
            image 'image'
          }
        }
      
        stages { 
          stage('locked_staged') {
            steps {
              //do some work
              lock(resource: 'resource', inversePrecedence:false) {
                //do locked work
              }
              //do more work
            }
          }
          
          stage('unlocked_stage') {
            steps {
              //work
            }
          }
      }
      

      This example would consume a node for the entire pipeline. Any node hitting the lock stage would current be blocked waiting on the lock. I would like for the node to be released once the lock is hit and resume once available.

      There is a partial workaround by specifically declaring agent at the appropriate stage.

      Jenkinsfile with workaround
      pipeline {
        agent none
      
        stages { 
           stage('locked_staged') {
              agent 'label'
              steps {
                lock(resource: 'resource', inversePrecedence:false) {
                  //do some work
                }
             }
          }
          
          stage('unlocked_stage') {
            steps {
              //work
            }
          }
      }
      

          [JENKINS-40966] Ability to leave Declarative node block w/ top-level agent for a single stage

          Patrick Wolf added a comment -

          you can also use the node step directly in a stage to get an executor.

          pipeline {
            agent none
            stages { 
              stage('locked_staged') {
                steps {
                  lock(resource: 'resource', inversePrecedence:false) {
                     echo "I'm locked"
                  }
                  node('my-label') {
                    echo "I'm on a node and not locked"
                  }
                }
              }
            }
          }
          

          Patrick Wolf added a comment - you can also use the node step directly in a stage to get an executor. pipeline { agent none stages { stage( 'locked_staged' ) { steps { lock(resource: 'resource' , inversePrecedence: false ) { echo "I'm locked" } node( 'my-label' ) { echo "I'm on a node and not locked" } } } } }

          Andrew Bayer added a comment -

          Hmm - good point. With agent on the stage, the lock isn't actually happening until you've got the executor on the agent. So if you've only got one executor on that agent (or all the executors are filled up with pipelines, either locked or running), the build may end up waiting a while to even get the executor before locking, and you end up with the locked pipelines still eating an executor. Hrrrrm. We might want to add a section to stage like options at the top-level - you can run the entire build in things like timeout or timestamps there, before even the top-level agent is invoked. Might make sense to have the equivalent pre-agent behavior on stage too...

          Andrew Bayer added a comment - Hmm - good point. With agent on the stage, the lock isn't actually happening until you've got the executor on the agent. So if you've only got one executor on that agent (or all the executors are filled up with pipelines, either locked or running), the build may end up waiting a while to even get the executor before locking, and you end up with the locked pipelines still eating an executor. Hrrrrm. We might want to add a section to stage like options at the top-level - you can run the entire build in things like timeout or timestamps there, before even the top-level agent is invoked. Might make sense to have the equivalent pre- agent behavior on stage too...

          Robby Pocase added a comment -

          Is it possible to lock before entering a node block? I would expect this to put the pipeline into flyweight while waiting on an executor and locked. Testing locally gave me a "No such DSL method 'lock' found among steps" - not sure if I'm missing an API extension plugin.

          pipeline {
            agent none
            stages { 
              stage('locked_staged') {
                steps {
                  node('my-label') {
                    echo "I'm on a node and not locked"
                  }
                  lock(resource: 'resource', inversePrecedence:false) {
                     echo "I'm locked"
                     node('my-label') {
                      echo "I'm on a node and locked"
                    }
                  }       
                }
              }
            }
          }
          

          Robby Pocase added a comment - Is it possible to lock before entering a node block? I would expect this to put the pipeline into flyweight while waiting on an executor and locked. Testing locally gave me a "No such DSL method 'lock' found among steps" - not sure if I'm missing an API extension plugin. pipeline { agent none stages { stage( 'locked_staged' ) { steps { node( 'my-label' ) { echo "I'm on a node and not locked" } lock(resource: 'resource' , inversePrecedence: false ) { echo "I'm locked" node( 'my-label' ) { echo "I'm on a node and locked" } } } } } }

          Patrick Wolf added a comment -

          You need to install the lockable-resources-plugin to use lock in Pipeline.

          In your example able the first node is not blocked but the second node would be locked.

          Patrick Wolf added a comment - You need to install the lockable-resources-plugin to use lock in Pipeline. In your example able the first node is not blocked but the second node would be locked.

          Andrew Bayer added a comment -

          So I'm generalizing this ticket! There are multiple cases where we want to be able to "leave" the node block created at the top-level by use of anything but agent none but then "return" to it after a stage with a different agent (particularly agent none) has completed. There's lock, there's the throttle step I've been working on in JENKINS-31801, there's CloudBees' proprietary checkpoint feature, there's the input step, and there's probably more I'm not thinking of.

          To do this, I think there are a few functional requirements:

          • Free up the original executor so that other builds can use it while we're not.
          • Somehow "lock" the original workspace so that it doesn't get reused or deleted while we're not on that executor - i.e., we want to make sure another run of the same Pipeline doesn't end up reusing it and instead creates "somePipeline@2" etc.
          • Ideally, keep the computer/node the original executor was on from being deleted due to RetentionStrategy until after we've come back to it and are truly done with it. This one is a very-nice-to-have rather than a hard requirement, though - you can work around this by using stash at the end of the stage before you leave the original executor and unstash at the beginning of the stage where you return to that agent/label/whatever. But stash is not very efficient, shall we say.
          • In the same vein, make sure we know the specific node/computer the original executor was on so we can return there. However, this is only useful if we've managed to "lock" the workspace to prevent reuse/deletion and we've managed to guarantee the node/computer won't be automatically deleted via a RetentionStrategy before we come back to it. If we can't guarantee reusability of the exact workspace and continued existence of the exact node/computer, knowing exactly where we ran before isn't necessarily of any use.
          • We need to decide whether we do this automatically whenever a per-stage agent is specified that does not include reuseNode true (an option on docker and dockerfile agent types) or if we require an additional configuration option (in the agent or on the stage itself) when using any agent but agent none. Usability is fun!

          So...yeah. I'm going to try to write up an RFC for this in the next few days, with the hope of getting that approved in the next week or two, followed by implementation in time for Declarative 1.2 (which is tentatively scheduled for late April).

          Andrew Bayer added a comment - So I'm generalizing this ticket! There are multiple cases where we want to be able to "leave" the node block created at the top-level by use of anything but agent none but then "return" to it after a stage with a different agent (particularly agent none ) has completed. There's lock , there's the throttle step I've been working on in  JENKINS-31801 , there's CloudBees' proprietary checkpoint feature, there's the input step, and there's probably more I'm not thinking of. To do this, I think there are a few functional requirements: Free up the original executor so that other builds can use it while we're not. Somehow "lock" the original workspace so that it doesn't get reused or deleted while we're not on that executor - i.e., we want to make sure another run of the same Pipeline doesn't end up reusing it and instead creates "somePipeline@2" etc. Ideally, keep the computer/node the original executor was on from being deleted due to RetentionStrategy until after we've come back to it and are truly done with it. This one is a very-nice-to-have rather than a hard requirement, though - you can work around this by using stash at the end of the stage before you leave the original executor and unstash at the beginning of the stage where you return to that agent/label/whatever. But stash is not very efficient, shall we say. In the same vein, make sure we know the specific node/computer the original executor was on so we can return there. However, this is only useful if we've managed to "lock" the workspace to prevent reuse/deletion and we've managed to guarantee the node/computer won't be automatically deleted via a RetentionStrategy before we come back to it. If we can't guarantee reusability of the exact workspace and continued existence of the exact node/computer, knowing exactly where we ran before isn't necessarily of any use. We need to decide whether we do this automatically whenever a per-stage agent is specified that does not include reuseNode true (an option on docker and dockerfile agent types) or if we require an additional configuration option (in the agent or on the stage itself) when using any agent but agent none . Usability is fun! So...yeah. I'm going to try to write up an RFC for this in the next few days, with the hope of getting that approved in the next week or two, followed by implementation in time for Declarative 1.2 (which is tentatively scheduled for late April).

          Philip Stroh added a comment -

          abayer Any news on this one? Especially when using only one input step in a declarative pipeline if feels really cumbersome to introduce a node or agent for each stage to workaround this issue. The issue feels more like a bug or flaw in declarative than like a new feature request.

          IMHO it's important that the fix provides not only the ability to leave the top-level node/agent for a single stage but for a single step within a stage. Otherwise you have to introduce an additional stage only to provide an input within declarative.

          Example:

          pipeline {
              agent none
              stages {
                  stage ("test stage") {
                      steps {
                          input "Continue?"
                          node("master") {
                              sh 'pwd'
                          }    
                      }    
                  }  
                  stage ("another stage") {
                      steps {
                          node("master") {
                              sh 'pwd'
                              echo 'all other stages need node step :('
                          }    
                      }    
                  }
              }
          }   
          

          Philip Stroh added a comment - abayer Any news on this one? Especially when using only one input step in a declarative pipeline if feels really cumbersome to introduce a node or agent for each stage to workaround this issue. The issue feels more like a bug or flaw in declarative than like a new feature request. IMHO it's important that the fix provides not only the ability to leave the top-level node/agent for a single stage but for a single step within a stage. Otherwise you have to introduce an additional stage only to provide an input within declarative. Example: pipeline {     agent none     stages {         stage ( "test stage" ) {             steps {                 input "Continue?"                 node( "master" ) {                     sh 'pwd'                 }                 }             }           stage ( "another stage" ) {             steps {                 node( "master" ) {                     sh 'pwd'                     echo 'all other stages need node step :('                 }                 }             }     } }   

            Unassigned Unassigned
            rpocase Robby Pocase
            Votes:
            3 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated: