There may be some implementations in common with https://issues.jenkins-ci.org/browse/JENKINS-41334 ? As there needs to be a more variable way of structuring stages.

Hi guys, 

Is there any chance of this being implemented?

currently i have a pipeline where i create an apt repo on a linux agent, then pull from it on a different agent. I need those 2 stages to not be interrupted so ideally i would lock around both stages.

I'm not sure of a way to do this currently?

dan

for me actually an alternative would be to be able to run steps within a stage on different agents but i'm not sure that makes sense

danturn I didn't try it but if I remember correctly you can already run steps on different agents with something like this:

stage('My Stage') {
  steps {
    node('some label') {
      // run step 1
    }

    node('another label') {
      // run step 2
    }
  }
}

You probably have to use `agent none` for this to work properly.

ooh ok, that would solve my problem
 
i'll try it and update
 
thanks

that works, thank you very much Roch!!

Alternatively it would be great if we could do something like lock() somewhere and unlock() in another place in pipeline.

Example (similar to my own case):

[...]
def selectedNode = utils.selectAvailableResource()

pipeline {
  agent none
    stages {
      stage('Checkout code') {
        agent { label selectedNode }
        steps {
          lock('aResource')
        }
      }
      stage('Build') {
        agent { label selectedNode }
        steps {
          echo 'Still locked'
        }
      }
      stage('Integration tests') {
        agent { label selectedNode }
        steps {
          echo 'Still locked'
        }
      }
      stage('Cleanup') {
        agent { label selectedNode }
        steps {
          unlock('aResource')
          echo 'Finally unlocked'
        }
      }
    }
  }
}

What matt_was suggested would open up many exiting possibilities.

For example locking a resource prior to acquiring a node (best practice), but releasing it while retaining the build slot and workspace for further processing that does not need any locks.

// acquire some kind of test resources
def locks = lock(label: 'aResource', quantity: 1)
node {
    stage('build'){
       sh "mvn install -D${locks[0].name}"
       unlock(locks)
    }
    // continoue processing without having to leave the node
    stage('analysis'){
        sh "mvn sonar:sonar"
    }
}

having that return value would be awesome too (to ask for the name of a resource and possibly more properties), but this is discussed in other tickets.

This is the single issue that is preventing us from moving to declarative pipeline. We currently use scripted Jenkinsfiles for exactly this purpose: to be able to define multiple stages which can lock a resource atomically.

In our case, we perform embedded hardware testing which requires the use of hardware resources to run testing. Because there are several stages in our pipeline which interact with the hardware (resetting, provisioning, testing, etc), the ability to lock across multiple stages is a necessity.

I do like the concept of being able to control "lock" versus "unlock". I think that would solve the problem sufficiently for us, and open up new possibilities.

+1 ^^

This is also the only issue preventing us from using this in our CICD system.  We rely on locking to lock external resources so we can build a queue and they do not effect eachother on multiple pipelines (If there is another way that would suffice but nothing has worked as yet).  stage by stage does not suffice.

The lock/unlock is also a highly desirable functionality.  We could make great use of it.

Initial WIP PR in lockable-resources for adding getLock(...) and releaseLock() steps is up at https://github.com/jenkinsci/lockable-resources-plugin/pull/64.

+1 To resource request. We use traditional pipes that have several stages that need to share a lock.

The combination of JENKINS-46809 and JENKINS-48380 will let you define a list of stages to run within a single top level (or parallel) stage, which will let you get what's desired here.

abayer I'm not so sure about this, as this only works if we would be allowed to wrap the substages inside an "lock" declaration, which, as of now, would be impossible as lock is not a valid direct child of a stage (has to be nested inside steps).

I tried

stage('Build Editor') {
    lock("myLock") {
        steps {
            //...
        }
    }
}

,

stage('Build Editor') {
    lock("myLock") {
        parallel {
            stage("ReallyBuild") {
                steps {
                    //...
                }
            }
        }
    }
}

 as well as 

stage('Build Editor') {
    parallel {
        lock("myLock") {
            stage("ReallyBuild") {
                steps {
                    //...
                }
            }
        }
    }
}

 to no avail, as it fails with

No "steps" or "parallel" to execute within stage

 or

No stages specified ...

JENKINS-46809 will give you what you need on top of what's already available from JENKINS-48380, so that you could do something like:

stage('Parent') {
  options {
    lock('myLock')
  }
  stages {
    stage('first child') {
      ...
    }
    stage('second child') {
      ...
    }
  }
}

abayer, I'm trying to lock multiple stages based on the Jenkins node on which the stages will execute. It seems like the only way to do this is using the `options` block that you specified:

stage('Parent') {
  options {
    lock("${env.NODE_NAME}")
  }
  stages {
    stage('first child') {
      ...
    }
    stage('second child') {
      ...
    }
  }
}

Unfortunately, the resource this attempts to lock is `null`:

Trying to acquire lock on [null]

It appears that this won't work because the stage's `options` directive occurs before entering the agent. The declarative pipeline syntax  documentation says: 

Inside a stage, the steps in the options directive are invoked before entering the agent or checking any when conditions.

 
Is there any way to lock multiple stages for a single env.NODE_NAME?  Ideally, other subsequent stages could still run in parallel with other builds (i.e. so limiting executors to 1 or disabling concurrent builds are not options).
 

Just wanted to +1 on rrueth's comment regarding locking multiple stages with lock named after $NODE_NAME. Pretty much in the same situation, and I'd like to know if this can be supported - either by enabling options block to execute after agent has been assigned for a given stage, or through some other means if this is not an option.

abayer do you know if this could be possible?

env is not available in options I think, but you could try getting the node name from the API:

stage('Parent') {
   options {
     lock("${currentBuild.getRawBuild().getExecutor().getOwner().getDisplayName()}")
   }
   stages {
      stage('first child') {
         ...
      }
      stage('second child') {
         ...
      }
   }
}

Hi, just wanted to add that we have the same problem where we need to lock multiple stages at once for atomicity (environment build and integration tests go together). Also, we need to parametrise the lock with the environment against which the build is done: it is ok to have parallel builds against different environments, but not against the same one.

As it looks, currently the multistage lock is available only in the options section which does not support environment parameters. Hence, it looks that we can not achieve what we need.

Bulk closing resolved issues.