Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34268

Lock multiple resources using the Pipeline lock step

    XMLWordPrintable

Details

    Description

      The current implementation of Pipeline lock step allows to block a single resource.

      It should be extended to cover all the functionality of the plugin (applicable to non-freestyle jobs) such as blocking resources by label or request a lock for N resources.

      The DSL must be something like this:

      lock (resources: ['resource1', 'resource2']) {
        ... execution block ...
      }
      

      or

      lock (label: 'my-resources') {
        ... execution block ...
      }
      

      The behavior of the label parameter would be equivalent to:

      lock (resources: ['resource3', 'resource4']) { // if both resource3 and resource4 are labeled as 'my-resources'
        ... execution block ...
      }
      

      Attachments

        Issue Links

          Activity

            aeon512 Florian Hug added a comment -

            Besides the actual label functionality, also the possibility to request a certain amount of resources from this label would be essential.

            I would personally prefer something like

            lock(label: 'fooBar', quantity: 2) {
              // whatever
            }
            

            Actually this is what keeps us from using pipelines.

            aeon512 Florian Hug added a comment - Besides the actual label functionality, also the possibility to request a certain amount of resources from this label would be essential. I would personally prefer something like lock(label: 'fooBar' , quantity: 2) { // whatever } Actually this is what keeps us from using pipelines.

            aeon512 Could you explain a real use case where the quantity configuration makes sense please? It's not clear to me why one would want to acquire 2 resources but do not caring about which ones specifically.

            amuniz Antonio Muñiz added a comment - aeon512 Could you explain a real use case where the quantity configuration makes sense please? It's not clear to me why one would want to acquire 2 resources but do not caring about which ones specifically.
            aeon512 Florian Hug added a comment - - edited

            Background:
            We dynamically create virtual machines which acts as jenkins slave (using the swarm-plugin) to perform the actual build steps. Thereby, if an error occurs we can archive the whole VM such that a developer can investigate and solve the problem. Using linked snapshots the whole purpose of creating such a new VM takes simply seconds and works very well.

            Problem solved by lockable resources:
            Since the overall amount of memory and cores on the host machine is limited, we used the lockable resources plugin to manage the distribution of memory (slices) and cpu cores.

            For example, let's say we have configured 16 resources (Memory Slice 1, Memory Slice 2, ..., Memory Slice 3), each representing e.g. 4 GB Memory. that is a total of 64 GB being available for virtual machines. When a job starts and wants to create a VM with e.g. 16 GB virtual memory, the job simply tries to request 4 slices ( =quantity ) of 4 GB memory

            lock(label: 'memory', quantity: 4) {
               // actual build steps
            }
            

            If this succeeds there is enough memory available, and the job can continue. If not, the job blocks until the necessary amount of memory is available again (since other jobs have finished and shut down their VMs).

            The same principle applies to CPU (cores) as well.

            Does this explanation help?

            aeon512 Florian Hug added a comment - - edited Background: We dynamically create virtual machines which acts as jenkins slave (using the swarm-plugin ) to perform the actual build steps. Thereby, if an error occurs we can archive the whole VM such that a developer can investigate and solve the problem. Using linked snapshots the whole purpose of creating such a new VM takes simply seconds and works very well. Problem solved by lockable resources: Since the overall amount of memory and cores on the host machine is limited, we used the lockable resources plugin to manage the distribution of memory (slices) and cpu cores. For example, let's say we have configured 16 resources (Memory Slice 1, Memory Slice 2, ..., Memory Slice 3), each representing e.g. 4 GB Memory. that is a total of 64 GB being available for virtual machines. When a job starts and wants to create a VM with e.g. 16 GB virtual memory, the job simply tries to request 4 slices ( =quantity ) of 4 GB memory lock(label: 'memory' , quantity: 4) { // actual build steps } If this succeeds there is enough memory available, and the job can continue. If not, the job blocks until the necessary amount of memory is available again (since other jobs have finished and shut down their VMs). The same principle applies to CPU (cores) as well. Does this explanation help?

            Thanks for take the time to explain!

            Makes sense, although nothing is preventing the steps inside the lock block to consume more CPU/RAM than virtually acquired, right?

            amuniz Antonio Muñiz added a comment - Thanks for take the time to explain! Makes sense, although nothing is preventing the steps inside the lock block to consume more CPU/RAM than virtually acquired, right?
            aeon512 Florian Hug added a comment -

            Yes and no
            No: Once the VM is configured to a certain amount of CPU/RAM, that amount is fixed and no process inside the VM can use more. If a process inside such a VM want's to consume more memory it simply receives an OutOfMemory exception.
            Yes: However, in theory, you can lock a different amount than you configure the VM.

            However, in our pipeline scripts we define a variable, e.g. required_memory_slices and use this variable to specify the necessary quantity as well as use this variable when setting up the virtual machine. Hence, in this case, it is guaranteed that we lock the same amount that we configure the VM to use, and hence no process can consume more CPU/RAM.

            aeon512 Florian Hug added a comment - Yes and no No: Once the VM is configured to a certain amount of CPU/RAM, that amount is fixed and no process inside the VM can use more. If a process inside such a VM want's to consume more memory it simply receives an OutOfMemory exception. Yes: However, in theory, you can lock a different amount than you configure the VM. However, in our pipeline scripts we define a variable, e.g. required_memory_slices and use this variable to specify the necessary quantity as well as use this variable when setting up the virtual machine. Hence, in this case, it is guaranteed that we lock the same amount that we configure the VM to use, and hence no process can consume more CPU/RAM.

            Ok, looks good. I've created a separate issue (JENKINS-34273) for quantity support in Pipeline as it can be handled independently of this one.

            BTW you have there a nice CI configuration

            amuniz Antonio Muñiz added a comment - Ok, looks good. I've created a separate issue ( JENKINS-34273 ) for quantity support in Pipeline as it can be handled independently of this one. BTW you have there a nice CI configuration
            aeon512 Florian Hug added a comment -

            Thanks And we are keen on switching to the new Pipeline - but JENKINS-34273 is absolutely necessary for us.

            aeon512 Florian Hug added a comment - Thanks And we are keen on switching to the new Pipeline - but JENKINS-34273 is absolutely necessary for us.
            aeon512 Florian Hug added a comment -

            amuniz, maybe you could have a look at PR26 where I tried to implement the functionality for label and quantity. Since this is my first Jenkins plugin contribution (and my first actual Java implementation) I'm open for any kind of feedback. Would be great if we get this merged in.

            lock(label: "MyLabelName", quantity: 3) {
                // body
            }
            

            Note however, that for closing this Issue the functionality to specify a list of resources is still missing. JENKINS-34273 however could be closed completely.
            I couldn't figure out how the DataBoundSetter for this would need to be modified within the actual LockStep. The actual implementation for dealing with several required resources is already available.

            aeon512 Florian Hug added a comment - amuniz , maybe you could have a look at PR26 where I tried to implement the functionality for label and quantity . Since this is my first Jenkins plugin contribution (and my first actual Java implementation) I'm open for any kind of feedback. Would be great if we get this merged in. lock(label: "MyLabelName" , quantity: 3) { // body } Note however, that for closing this Issue the functionality to specify a list of resources is still missing. JENKINS-34273 however could be closed completely. I couldn't figure out how the DataBoundSetter for this would need to be modified within the actual LockStep . The actual implementation for dealing with several required resources is already available.
            haney David Haney added a comment -

            Should the fix for this include exposing the names of the acquired resource within the body? For instance if I have something like:

            lock (label: 'my-resources') {
              ... execution block ...
            }
            

            It would be useful to know whether resource1 or resource2 was acquired by the lock.

            haney David Haney added a comment - Should the fix for this include exposing the names of the acquired resource within the body? For instance if I have something like: lock (label: 'my-resources' ) { ... execution block ... } It would be useful to know whether resource1 or resource2 was acquired by the lock.
            tjanez Tadej Janež added a comment -

            haney, I think the problem you mention would be solved by pursuing merging PR #20 tracked by JENKINS-31437.

            tjanez Tadej Janež added a comment - haney , I think the problem you mention would be solved by pursuing merging PR #20 tracked by JENKINS-31437 .

            the problem you mention would be solved by pursuing merging PR #20

            No. That PR is only valid for freestyle jobs and it's about exposing arbitrary properties not the name of locked resources.

            amuniz Antonio Muñiz added a comment - the problem you mention would be solved by pursuing merging PR #20 No. That PR is only valid for freestyle jobs and it's about exposing arbitrary properties not the name of locked resources.
            tjanez Tadej Janež added a comment -

            amuniz, oh, I see. Still, it would be great if one could set properties/environement variables for each resource and they would then be accessible within the lock execution block. Or it that possible already?

            tjanez Tadej Janež added a comment - amuniz , oh, I see. Still, it would be great if one could set properties/environement variables for each resource and they would then be accessible within the lock execution block. Or it that possible already?
            nwegner Niels Wegner added a comment -

            May be the locked resource might be return via given variable as requested in JENKINS-30269. This would be a similar way as if configured thru a job config.

            nwegner Niels Wegner added a comment - May be the locked resource might be return via given variable as requested in JENKINS-30269 . This would be a similar way as if configured thru a job config.
            rainwaj Justin Rainwater added a comment - - edited

            Just curious has anyone tried nested locking as a workaround? I'm curious if that would work.
            So instead of

                lock (resources: ['resource1', 'resource2']) {
                  ... execution block ...
               }
            

            it would be:

                lock('resource1) {
                    lock('resource2) {
                        ... execution block ...
                    }
                }
            
            rainwaj Justin Rainwater added a comment - - edited Just curious has anyone tried nested locking as a workaround? I'm curious if that would work. So instead of lock (resources: [ 'resource1' , 'resource2' ]) { ... execution block ... } it would be: lock('resource1) { lock('resource2) { ... execution block ... } }
            epsilon_b Epsilon B added a comment -

            Implemented in PR36.
            The proposed DSL syntaxe is (each parameter is optional, some are aliases such as "label" / "labels" / "capability" / "capabilities"):

            lock(resource: 'resource1 resource2', resources: ['resource1', 'resource2'], 
                label: 'label1 label2', labels: ['label1', 'label2'],
                capability: 'label1 label2', capabilities: ['label1', 'label2'],
                quantity: 2, variable: 'MY_VAR') {
                    ... execution block ...
            }
            

            Note that "quantity" is only relevant for "labels" / "capabilities" (not "resources", since they are unique and defined by names)

            epsilon_b Epsilon B added a comment - Implemented in PR36 . The proposed DSL syntaxe is (each parameter is optional, some are aliases such as "label" / "labels" / "capability" / "capabilities"): lock(resource: 'resource1 resource2', resources: ['resource1', 'resource2'], label: 'label1 label2', labels: ['label1', 'label2'], capability: 'label1 label2', capabilities: ['label1', 'label2'], quantity: 2, variable: 'MY_VAR') { ... execution block ... } Note that "quantity" is only relevant for "labels" / "capabilities" (not "resources", since they are unique and defined by names)

            Will PR36 be merged/released soon? This would resource allocation of some old ssh builders that we would like to do from pipelines. I there anything I can do to help? Is there a test build that we could run?

            fkykko Staffan Forsell added a comment - Will PR36 be merged/released soon? This would resource allocation of some old ssh builders that we would like to do from pipelines. I there anything I can do to help? Is there a test build that we could run?

            Code changed in jenkins
            User: Antonio Muniz
            Path:
            src/main/java/org/jenkins/plugins/lockableresources/BackwardCompatibility.java
            src/main/java/org/jenkins/plugins/lockableresources/LockStep.java
            src/main/java/org/jenkins/plugins/lockableresources/LockStepExecution.java
            src/main/java/org/jenkins/plugins/lockableresources/LockableResource.java
            src/main/java/org/jenkins/plugins/lockableresources/LockableResourcesManager.java
            src/main/java/org/jenkins/plugins/lockableresources/actions/LockableResourcesRootAction.java
            src/main/java/org/jenkins/plugins/lockableresources/queue/LockRunListener.java
            src/main/java/org/jenkins/plugins/lockableresources/queue/LockableResourcesStruct.java
            src/main/java/org/jenkins/plugins/lockableresources/queue/QueuedContextStruct.java
            src/main/resources/org/jenkins/plugins/lockableresources/LockStep/config.jelly
            src/main/resources/org/jenkins/plugins/lockableresources/LockStep/help-label.html
            src/main/resources/org/jenkins/plugins/lockableresources/LockStep/help-quantity.html
            src/main/resources/org/jenkins/plugins/lockableresources/LockStep/help-resource.html
            src/test/java/org/jenkins/plugins/lockableresources/LockStepTest.java
            http://jenkins-ci.org/commit/lockable-resources-plugin/974572db98214f7b9293a303c9fd8371877250a0
            Log:
            Merge pull request #42 from amuniz/pr-26-fix

            JENKINS-34268JENKINS-34273 Lock multiple resources with specific quantity

            Compare: https://github.com/jenkinsci/lockable-resources-plugin/compare/ba48550fa5bc...974572db9821

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Antonio Muniz Path: src/main/java/org/jenkins/plugins/lockableresources/BackwardCompatibility.java src/main/java/org/jenkins/plugins/lockableresources/LockStep.java src/main/java/org/jenkins/plugins/lockableresources/LockStepExecution.java src/main/java/org/jenkins/plugins/lockableresources/LockableResource.java src/main/java/org/jenkins/plugins/lockableresources/LockableResourcesManager.java src/main/java/org/jenkins/plugins/lockableresources/actions/LockableResourcesRootAction.java src/main/java/org/jenkins/plugins/lockableresources/queue/LockRunListener.java src/main/java/org/jenkins/plugins/lockableresources/queue/LockableResourcesStruct.java src/main/java/org/jenkins/plugins/lockableresources/queue/QueuedContextStruct.java src/main/resources/org/jenkins/plugins/lockableresources/LockStep/config.jelly src/main/resources/org/jenkins/plugins/lockableresources/LockStep/help-label.html src/main/resources/org/jenkins/plugins/lockableresources/LockStep/help-quantity.html src/main/resources/org/jenkins/plugins/lockableresources/LockStep/help-resource.html src/test/java/org/jenkins/plugins/lockableresources/LockStepTest.java http://jenkins-ci.org/commit/lockable-resources-plugin/974572db98214f7b9293a303c9fd8371877250a0 Log: Merge pull request #42 from amuniz/pr-26-fix JENKINS-34268 JENKINS-34273 Lock multiple resources with specific quantity Compare: https://github.com/jenkinsci/lockable-resources-plugin/compare/ba48550fa5bc...974572db9821

            Now that the PR that fixes this ticket is merged, I'd like to request a release of this plugin as soon as is appropriate.

            brendan_shape Brendan Mannix added a comment - Now that the PR that fixes this ticket is merged, I'd like to request a release of this plugin as soon as is appropriate.

            Released as 1.11

            amuniz Antonio Muñiz added a comment - Released as 1.11

            amuniz
            Hi! Nice to see this merged.
            Looking at the merged content, I can't seem to find any pipeline syntax for actually knowing what resource was locked.
            Specifically the proposed syntax from PR36 above: ", variable: 'MY_VAR'" is missing.
            Our use case is that we wan't to specify a label containing node names for legacy builders that can't run the slave.jar (java7) anymore. We wan't to lock a resources (host name) from he label and then ssh to that machine any execute some build steps. This works with the FreestyleJob functionality of #Reserved resources variable name". Any thought of adding this to pipeline?

            fkykko Staffan Forsell added a comment - amuniz Hi! Nice to see this merged. Looking at the merged content, I can't seem to find any pipeline syntax for actually knowing what resource was locked. Specifically the proposed syntax from PR36 above: ", variable: 'MY_VAR'" is missing. Our use case is that we wan't to specify a label containing node names for legacy builders that can't run the slave.jar (java7) anymore. We wan't to lock a resources (host name) from he label and then ssh to that machine any execute some build steps. This works with the FreestyleJob functionality of #Reserved resources variable name". Any thought of adding this to pipeline?
            koomietx Karl Schulz added a comment -

            I'm also unable to successfully use the MY_VAR option. It's awesome to be able to lock multiple resources now, but can't recreate the capability of a Freestyle job without being able to query the results.

            koomietx Karl Schulz added a comment - I'm also unable to successfully use the MY_VAR option. It's awesome to be able to lock multiple resources now, but can't recreate the capability of a Freestyle job without being able to query the results.
            glance Anton Lundin added a comment -

            I did a really ugly workaround for the missing resourceVariable :

            lock(label: 'LABEL', quantity: 1) {
            echo org.jenkins.plugins.lockableresources.LockableResourcesManager.class.get().getResourcesFromBuild(currentBuild.getRawBuild())[0].getName()
            }

            That will get you which resource you locked.

            glance Anton Lundin added a comment - I did a really ugly workaround for the missing resourceVariable : lock(label: 'LABEL', quantity: 1) { echo org.jenkins.plugins.lockableresources.LockableResourcesManager.class.get().getResourcesFromBuild(currentBuild.getRawBuild()) [0] .getName() } That will get you which resource you locked.
            dafalcon Dan Falcone added a comment - - edited

            After implementing Anton Lundin's suggestion, I received the following error in my build (followed by a stacktrace):

            org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use method

            To fix it, I went to Manage Jenkins > In process Script Approval, clicked Approve on the pending signature approval, and reran the build.  I had to repeat the process 4-5 times to approve all the required signatures, but it worked great after that.

            dafalcon Dan Falcone added a comment - - edited After implementing Anton Lundin's suggestion, I received the following error in my build (followed by a stacktrace): org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use method To fix it, I went to Manage Jenkins > In process Script Approval, clicked Approve on the pending signature approval, and reran the build.  I had to repeat the process 4-5 times to approve all the required signatures, but it worked great after that.
            mrozekma Michael Mrozek added a comment - - edited

            Signatures required for the above workaround:

            method org.jenkins.plugins.lockableresources.LockableResource getName
            method org.jenkins.plugins.lockableresources.LockableResourcesManager getResourcesFromBuild hudson.model.Run
            method org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper getRawBuild
            staticMethod org.jenkins.plugins.lockableresources.LockableResourcesManager get
            mrozekma Michael Mrozek added a comment - - edited Signatures required for the above workaround : method org.jenkins.plugins.lockableresources.LockableResource getName method org.jenkins.plugins.lockableresources.LockableResourcesManager getResourcesFromBuild hudson.model.Run method org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper getRawBuild staticMethod org.jenkins.plugins.lockableresources.LockableResourcesManager get
            mireksz Mirek Sz added a comment -

            glance

            I did a really ugly workaround for the missing resourceVariable :

            Did you happen to find any way of getting the resource that's locked in the current lock closure? Your workaround always returns the name of first locked resource. I'm trying to run label-based locks in parallel, so I cannot simply use getResourcesFromBuild(...)[0], but must find out the exact resource locked within current scope. Any ideas?

            mireksz Mirek Sz added a comment - glance I did a really ugly workaround for the missing resourceVariable : Did you happen to find any way of getting the resource that's locked in the current lock closure? Your workaround always returns the name of first locked resource. I'm trying to run label-based locks in parallel, so I cannot simply use getResourcesFromBuild(...) [0] , but must find out the exact resource locked within current scope. Any ideas?
            glance Anton Lundin added a comment -

            The hack grew quite a bit over the following days, and ended up as:

            https://gist.github.com/glance-/aaa3c037757895798d4e1be5134bb843

             

            I removed the need for extensive whitelisting by writing it as a pipeline library which later could be imported into the pipeline dsl by:
            @Library("PipelineHelpers")
            import PipelineHelpers.LockableResourcesHelper

             

            We never use different locked resources in parallel so I have never needed to figure out if you can map locked resource to which block that locked it. If you ever do, please post the solution here for others to find.

            glance Anton Lundin added a comment - The hack grew quite a bit over the following days, and ended up as: https://gist.github.com/glance-/aaa3c037757895798d4e1be5134bb843   I removed the need for extensive whitelisting by writing it as a pipeline library which later could be imported into the pipeline dsl by: @Library("PipelineHelpers") import PipelineHelpers.LockableResourcesHelper   We never use different locked resources in parallel so I have never needed to figure out if you can map locked resource to which block that locked it. If you ever do, please post the solution here for others to find.
            armb Alan Braggins added a comment - - edited

            We use locked resources in parallel, so the getResourcesFromBuild workaround didn't work for us.

            https://github.com/jenkinsci/lockable-resources-plugin/pull/50 seems to work. (I haven't tried pull/49, but it appears to be a similar approach to the same fix.)

            armb Alan Braggins added a comment - - edited We use locked resources in parallel, so the getResourcesFromBuild workaround didn't work for us. https://github.com/jenkinsci/lockable-resources-plugin/pull/50  seems to work. (I haven't tried pull/49, but it appears to be a similar approach to the same fix.)

            Anyone hoping to make use of locking a dynamic list of resources will probably find this feature (if it was ever implemented) doesn't work in v2.4 of the plugin.

            So, I whipped up the following helper function which should allow you to do so:

             

            def lockResources(listOfResources, closure) {
                if (listOfResources.size() > 1) {
                    lockResources(listOfResources[1..-1], { lock(listOfResources[0]) { closure() }})
                } else {
                    lock(listOfResources[0]) { closure() }
                }
            }
            

            Example usage:

            def resources = ['foo', 'bar', 'baz']
            lockResources(myResources, { 
                print('hello, world') 
            })
            

             

             

            chrisdickinsonnio Chris Dickinson added a comment - Anyone hoping to make use of locking a dynamic list of resources will probably find this feature (if it was ever implemented) doesn't work in v2.4 of the plugin. So, I whipped up the following helper function which should allow you to do so:   def lockResources(listOfResources, closure) { if (listOfResources.size() > 1) { lockResources(listOfResources[1..-1], { lock(listOfResources[0]) { closure() }}) } else { lock(listOfResources[0]) { closure() } } } Example usage: def resources = [ 'foo' , 'bar' , 'baz' ] lockResources(myResources, { print( 'hello, world' ) })    

            chrisdickinsonnio Can you open a new ticket instead of commenting on a closed one (where nobody will ever see your comments, I stumbled on it by accident) if you have issues with the plugin? Anyway, locking of multiple resources is implemented since 2.3. The syntax (as generated by the snippet generator) is:

            lock(extra: [[resource: 'b'], [label: 'c', quantity: 1]], resource: 'a') {
                // some block
            }
            

             

            tgr Tobias Gruetzmacher added a comment - chrisdickinsonnio Can you open a new ticket instead of commenting on a closed one (where nobody will ever see your comments, I stumbled on it by accident) if you have issues with the plugin? Anyway, locking of multiple resources is implemented since 2.3. The syntax (as generated by the snippet generator) is: lock(extra: [[resource: 'b' ], [label: 'c' , quantity: 1]], resource: 'a' ) { // some block }  

            People

              amuniz Antonio Muñiz
              amuniz Antonio Muñiz
              Votes:
              19 Vote for this issue
              Watchers:
              33 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: