Besides the actual label functionality, also the possibility to request a certain amount of resources from this label would be essential.

I would personally prefer something like

lock(label: 'fooBar', quantity: 2) {
  // whatever
}

Actually this is what keeps us from using pipelines.

aeon512 Could you explain a real use case where the quantity configuration makes sense please? It's not clear to me why one would want to acquire 2 resources but do not caring about which ones specifically.

Background:
We dynamically create virtual machines which acts as jenkins slave (using the swarm-plugin) to perform the actual build steps. Thereby, if an error occurs we can archive the whole VM such that a developer can investigate and solve the problem. Using linked snapshots the whole purpose of creating such a new VM takes simply seconds and works very well.

Problem solved by lockable resources:
Since the overall amount of memory and cores on the host machine is limited, we used the lockable resources plugin to manage the distribution of memory (slices) and cpu cores.

For example, let's say we have configured 16 resources (Memory Slice 1, Memory Slice 2, ..., Memory Slice 3), each representing e.g. 4 GB Memory. that is a total of 64 GB being available for virtual machines. When a job starts and wants to create a VM with e.g. 16 GB virtual memory, the job simply tries to request 4 slices ( =quantity ) of 4 GB memory

lock(label: 'memory', quantity: 4) {
   // actual build steps
}

If this succeeds there is enough memory available, and the job can continue. If not, the job blocks until the necessary amount of memory is available again (since other jobs have finished and shut down their VMs).

The same principle applies to CPU (cores) as well.

Does this explanation help?

Thanks for take the time to explain!

Makes sense, although nothing is preventing the steps inside the lock block to consume more CPU/RAM than virtually acquired, right?

Yes and no
No: Once the VM is configured to a certain amount of CPU/RAM, that amount is fixed and no process inside the VM can use more. If a process inside such a VM want's to consume more memory it simply receives an OutOfMemory exception.
Yes: However, in theory, you can lock a different amount than you configure the VM.

However, in our pipeline scripts we define a variable, e.g. required_memory_slices and use this variable to specify the necessary quantity as well as use this variable when setting up the virtual machine. Hence, in this case, it is guaranteed that we lock the same amount that we configure the VM to use, and hence no process can consume more CPU/RAM.

Ok, looks good. I've created a separate issue (JENKINS-34273) for quantity support in Pipeline as it can be handled independently of this one.

BTW you have there a nice CI configuration

Thanks And we are keen on switching to the new Pipeline - but JENKINS-34273 is absolutely necessary for us.

amuniz, maybe you could have a look at PR26 where I tried to implement the functionality for label and quantity. Since this is my first Jenkins plugin contribution (and my first actual Java implementation) I'm open for any kind of feedback. Would be great if we get this merged in.

lock(label: "MyLabelName", quantity: 3) {
    // body
}

Note however, that for closing this Issue the functionality to specify a list of resources is still missing. JENKINS-34273 however could be closed completely.
I couldn't figure out how the DataBoundSetter for this would need to be modified within the actual LockStep. The actual implementation for dealing with several required resources is already available.

Should the fix for this include exposing the names of the acquired resource within the body? For instance if I have something like:

lock (label: 'my-resources') {
  ... execution block ...
}

It would be useful to know whether resource1 or resource2 was acquired by the lock.

haney, I think the problem you mention would be solved by pursuing merging PR #20 tracked by JENKINS-31437.

the problem you mention would be solved by pursuing merging PR #20

No. That PR is only valid for freestyle jobs and it's about exposing arbitrary properties not the name of locked resources.

amuniz, oh, I see. Still, it would be great if one could set properties/environement variables for each resource and they would then be accessible within the lock execution block. Or it that possible already?

May be the locked resource might be return via given variable as requested in JENKINS-30269. This would be a similar way as if configured thru a job config.

Just curious has anyone tried nested locking as a workaround? I'm curious if that would work.
So instead of

    lock (resources: ['resource1', 'resource2']) {
      ... execution block ...
   }

it would be:

    lock('resource1) {
        lock('resource2) {
            ... execution block ...
        }
    }

Implemented in PR36.
The proposed DSL syntaxe is (each parameter is optional, some are aliases such as "label" / "labels" / "capability" / "capabilities"):

lock(resource: 'resource1 resource2', resources: ['resource1', 'resource2'], 
    label: 'label1 label2', labels: ['label1', 'label2'],
    capability: 'label1 label2', capabilities: ['label1', 'label2'],
    quantity: 2, variable: 'MY_VAR') {
        ... execution block ...
}

Note that "quantity" is only relevant for "labels" / "capabilities" (not "resources", since they are unique and defined by names)

Will PR36 be merged/released soon? This would resource allocation of some old ssh builders that we would like to do from pipelines. I there anything I can do to help? Is there a test build that we could run?

Code changed in jenkins
User: Antonio Muniz
Path:
src/main/java/org/jenkins/plugins/lockableresources/BackwardCompatibility.java
src/main/java/org/jenkins/plugins/lockableresources/LockStep.java
src/main/java/org/jenkins/plugins/lockableresources/LockStepExecution.java
src/main/java/org/jenkins/plugins/lockableresources/LockableResource.java
src/main/java/org/jenkins/plugins/lockableresources/LockableResourcesManager.java
src/main/java/org/jenkins/plugins/lockableresources/actions/LockableResourcesRootAction.java
src/main/java/org/jenkins/plugins/lockableresources/queue/LockRunListener.java
src/main/java/org/jenkins/plugins/lockableresources/queue/LockableResourcesStruct.java
src/main/java/org/jenkins/plugins/lockableresources/queue/QueuedContextStruct.java
src/main/resources/org/jenkins/plugins/lockableresources/LockStep/config.jelly
src/main/resources/org/jenkins/plugins/lockableresources/LockStep/help-label.html
src/main/resources/org/jenkins/plugins/lockableresources/LockStep/help-quantity.html
src/main/resources/org/jenkins/plugins/lockableresources/LockStep/help-resource.html
src/test/java/org/jenkins/plugins/lockableresources/LockStepTest.java
http://jenkins-ci.org/commit/lockable-resources-plugin/974572db98214f7b9293a303c9fd8371877250a0
Log:
Merge pull request #42 from amuniz/pr-26-fix

JENKINS-34268JENKINS-34273 Lock multiple resources with specific quantity

Compare: https://github.com/jenkinsci/lockable-resources-plugin/compare/ba48550fa5bc...974572db9821

Now that the PR that fixes this ticket is merged, I'd like to request a release of this plugin as soon as is appropriate.

Released as 1.11

amuniz
Hi! Nice to see this merged.
Looking at the merged content, I can't seem to find any pipeline syntax for actually knowing what resource was locked.
Specifically the proposed syntax from PR36 above: ", variable: 'MY_VAR'" is missing.
Our use case is that we wan't to specify a label containing node names for legacy builders that can't run the slave.jar (java7) anymore. We wan't to lock a resources (host name) from he label and then ssh to that machine any execute some build steps. This works with the FreestyleJob functionality of #Reserved resources variable name". Any thought of adding this to pipeline?

I'm also unable to successfully use the MY_VAR option. It's awesome to be able to lock multiple resources now, but can't recreate the capability of a Freestyle job without being able to query the results.

After implementing Anton Lundin's suggestion, I received the following error in my build (followed by a stacktrace):

org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use method

To fix it, I went to Manage Jenkins > In process Script Approval, clicked Approve on the pending signature approval, and reran the build.  I had to repeat the process 4-5 times to approve all the required signatures, but it worked great after that.

Signatures required for the above workaround:

method org.jenkins.plugins.lockableresources.LockableResource getName
method org.jenkins.plugins.lockableresources.LockableResourcesManager getResourcesFromBuild hudson.model.Run
method org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper getRawBuild
staticMethod org.jenkins.plugins.lockableresources.LockableResourcesManager get

glance

I did a really ugly workaround for the missing resourceVariable :

Did you happen to find any way of getting the resource that's locked in the current lock closure? Your workaround always returns the name of first locked resource. I'm trying to run label-based locks in parallel, so I cannot simply use getResourcesFromBuild(...)[0], but must find out the exact resource locked within current scope. Any ideas?

The hack grew quite a bit over the following days, and ended up as:

https://gist.github.com/glance-/aaa3c037757895798d4e1be5134bb843

I removed the need for extensive whitelisting by writing it as a pipeline library which later could be imported into the pipeline dsl by:
@Library("PipelineHelpers")
import PipelineHelpers.LockableResourcesHelper

We never use different locked resources in parallel so I have never needed to figure out if you can map locked resource to which block that locked it. If you ever do, please post the solution here for others to find.

We use locked resources in parallel, so the getResourcesFromBuild workaround didn't work for us.

https://github.com/jenkinsci/lockable-resources-plugin/pull/50 seems to work. (I haven't tried pull/49, but it appears to be a similar approach to the same fix.)

Anyone hoping to make use of locking a dynamic list of resources will probably find this feature (if it was ever implemented) doesn't work in v2.4 of the plugin.

So, I whipped up the following helper function which should allow you to do so:

def lockResources(listOfResources, closure) {
    if (listOfResources.size() > 1) {
        lockResources(listOfResources[1..-1], { lock(listOfResources[0]) { closure() }})
    } else {
        lock(listOfResources[0]) { closure() }
    }
}

Example usage:

def resources = ['foo', 'bar', 'baz']
lockResources(myResources, { 
    print('hello, world') 
})

chrisdickinsonnio Can you open a new ticket instead of commenting on a closed one (where nobody will ever see your comments, I stumbled on it by accident) if you have issues with the plugin? Anyway, locking of multiple resources is implemented since 2.3. The syntax (as generated by the snippet generator) is:

lock(extra: [[resource: 'b'], [label: 'c', quantity: 1]], resource: 'a') {
    // some block
}