Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-35359

Limit concurrent builds for 1 MultiBranch project

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      I am currently running the latest version of the Pipeline plugin and i have created a multi-branch job. Everything is running fine on my Jenkinsfile but i would like to limit only 1 concurrent build for my entire multi-branch project. I see that you can set the option individually for the sub-sequent (branch) job but i would like to know if there is a way to limit it for the entire folder.

        Attachments

          Issue Links

            Activity

            Hide
            sc_rsc Stefan Cordes added a comment -

            This is not a duplicate of  JENKINS-34547 , see https://issues.jenkins-ci.org/browse/JENKINS-34547?focusedCommentId=291934&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-291934

            (here it requested to have limit all branches of one multibranch project build one by one).

            Show
            sc_rsc Stefan Cordes added a comment - This is not a duplicate of  JENKINS-34547  , see https://issues.jenkins-ci.org/browse/JENKINS-34547?focusedCommentId=291934&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-291934 (here it requested to have limit all branches of one multibranch project build one by one).
            Hide
            sc_rsc Stefan Cordes added a comment -
            Show
            sc_rsc Stefan Cordes added a comment - Maybe https://wiki.jenkins-ci.org/display/JENKINS/Lockable+Resources+Plugin  helps workarounding the issue?
            Hide
            jglick Jesse Glick added a comment -

            Right use lock for that purpose.

            Show
            jglick Jesse Glick added a comment - Right use lock for that purpose.
            Hide
            marcodeneu Marc Neureiter added a comment -

            In my opinion this issue is not resolved with using `lock`. If you have 5 jobs and 4 are prevented from obtaining the lock you still have 5 running executors. This is exactly what Throttle Concurrent Builds would and SHOULD do. But we don't have something like this in Multibranch Builds currently...

            Show
            marcodeneu Marc Neureiter added a comment - In my opinion this issue is not  resolved with using `lock`. If you have 5 jobs and 4 are prevented from obtaining the lock you still have 5 running executors. This is exactly what Throttle Concurrent Builds would and SHOULD do. But we don't have something like this in Multibranch Builds currently...
            Hide
            jglick Jesse Glick added a comment -

            If you have 5 jobs and 4 are prevented from obtaining the lock you still have 5 running executors.

            Not if the lock is acquired outside any node.

            Show
            jglick Jesse Glick added a comment - If you have 5 jobs and 4 are prevented from obtaining the lock you still have 5 running executors. Not if the lock is acquired outside any node .
            Hide
            jglick Jesse Glick added a comment -

            Anyway you can also use the TCB plugin with Pipeline if you want. Not sure why this would be a distinct issue.

            Show
            jglick Jesse Glick added a comment - Anyway you can also use the TCB plugin with Pipeline if you want. Not sure why this would be a distinct issue.
            Hide
            marcodeneu Marc Neureiter added a comment -

            Jesse Glick You're right, thanks for the hint! Should be in the readme in my opinion. Though TCB doesn't apply on multibranch/organisation jobs (in my case).

            Show
            marcodeneu Marc Neureiter added a comment - Jesse Glick You're right, thanks for the hint! Should be in the readme in my opinion. Though TCB doesn't apply on multibranch/organisation jobs (in my case).
            Hide
            pbrokemann Kofi Brokemann added a comment -

            Hi,

            I ran into this problem running a multibranch/ organization job. Is there any way to limit concurrent jobs on multibranch/ organization without implementing lockable resources? I tried the TCB plugin approach but that doesn't work. Thank you

            Show
            pbrokemann Kofi Brokemann added a comment - Hi, I ran into this problem running a multibranch/ organization job. Is there any way to limit concurrent jobs on multibranch/ organization without implementing lockable resources? I tried the TCB plugin approach but that doesn't work. Thank you
            Hide
            dallasclement Dallas Clement added a comment -

            I just created an issue for this same problem today which appears to be a duplicate of this one. 

            https://issues.jenkins-ci.org/browse/JENKINS-46593

            I really do not like the lock approach to dealing with this problem.  There should be a way to limit the number of executors.

             

            Show
            dallasclement Dallas Clement added a comment - I just created an issue for this same problem today which appears to be a duplicate of this one.  https://issues.jenkins-ci.org/browse/JENKINS-46593 I really do not like the lock approach to dealing with this problem.  There should be a way to limit the number of executors.  
            Hide
            johannsig Jóhann Sigurðsson added a comment - - edited

            Has this been resolved? I don't want two pipeline builds to each occupy an executor at the same time

            Show
            johannsig Jóhann Sigurðsson added a comment - - edited Has this been resolved? I don't want two pipeline builds to each occupy an executor at the same time
            Hide
            johannsig Jóhann Sigurðsson added a comment - - edited

            For anyone so inclined, I came up with a workaround by:

            1. Limiting Master node executors to Pipeline builds
            2. Creating a local slave agent and hooking it up to Jenkins to take care of other jobs

            Declarative script example:

            agent {
              label {
                label "Master"    
              }
            }
            

             

             

            Show
            johannsig Jóhann Sigurðsson added a comment - - edited For anyone so inclined, I came up with a workaround by: Limiting Master node executors to Pipeline builds Creating a local slave agent and hooking it up to Jenkins to take care of other jobs Declarative script example: agent { label { label "Master" } }    
            Hide
            petersbattaglia Peter Battaglia added a comment -

            Is there a timeline for a fix for this issue – or a reasonable workaround?

            Show
            petersbattaglia Peter Battaglia added a comment - Is there a timeline for a fix for this issue – or a reasonable workaround?
            Hide
            jglick Jesse Glick added a comment -

            Use lock (outside node!).

            Show
            jglick Jesse Glick added a comment - Use lock (outside node !).
            Hide
            qhartman Quentin Hartman added a comment -

            I agree that the lock, even outside the context of node isn't really a solution here.

             

            I have two builds, A and B. Build A requires that it is build one at a time on a node. Build B does not. If what I've seen is correct, a resource locked by build A will prevent build B from running on that resource as well. This is not what I want.

            Am I missing something? There's no reason to waste the time waiting for A to finish when I have B builds waiting to go.

            Show
            qhartman Quentin Hartman added a comment - I agree that the lock, even outside the context of node isn't really a solution here.   I have two builds, A and B. Build A requires that it is build one at a time on a node. Build B does not. If what I've seen is correct, a resource locked by build A will prevent build B from running on that resource as well. This is not what I want. Am I missing something? There's no reason to waste the time waiting for A to finish when I have B builds waiting to go.
            Hide
            pjhayward Phillip Hayward added a comment -

            Lockable resource do not solve this issue.  Let me clarify my use case:

            I have a shared Jenkins installation.  We have a number of teams using multi-branch pipeline jobs.  Sometimes a team will have a large number of branches build concurrently - for example, they have a lot of feature branches that all depend on a given snapshot, and are configured to build when an upstream project builds.  Another example is when a project is migrated from SVN to git.  Regardless of the actual trigger, the result is that every other project is effectively blocked out because we have no realistic way to limit the concurrent branches for a given project.  Implementing a lockable resource for every single project could technically solve the issue, but would be overly burdensome to administrate, and would then restrict us to a single concurrent branch, where we would prefer to allow up to four concurrent branches per project.

             

            Show
            pjhayward Phillip Hayward added a comment - Lockable resource do not solve this issue.  Let me clarify my use case: I have a shared Jenkins installation.  We have a number of teams using multi-branch pipeline jobs.  Sometimes a team will have a large number of branches build concurrently - for example, they have a lot of feature branches that all depend on a given snapshot, and are configured to build when an upstream project builds.  Another example is when a project is migrated from SVN to git.  Regardless of the actual trigger, the result is that every other project is effectively blocked out because we have no realistic way to limit the concurrent branches for a given project.  Implementing a lockable resource for every single project could technically solve the issue, but would be overly burdensome to administrate, and would then restrict us to a single concurrent branch, where we would prefer to allow up to four concurrent branches per project.  
            Hide
            jglick Jesse Glick added a comment -

            By the way there is a Throttle builds branch property (Pipeline: rateLimitBuilds job property) which is appropriate for certain scenarios. Currently it only throttles builds within a branch project, though, not across branches. The feature is built into the multibranch system (i.e., branch-api, not this plugin) for historical reasons.

            Show
            jglick Jesse Glick added a comment - By the way there is a Throttle builds branch property (Pipeline: rateLimitBuilds job property) which is appropriate for certain scenarios. Currently it only throttles builds within a branch project, though, not across branches. The feature is built into the multibranch system (i.e., branch-api , not this plugin) for historical reasons.
            Hide
            pkruk Piotr Krukowiecki added a comment -

            I would like to request this too. I can only run 1 build at time (does not matter what branch).

            Using lockable resources would be a workaround - as I understand I need to configure jenkins manually (add lockable resource), the Jenkinsfile is not enough...

            Show
            pkruk Piotr Krukowiecki added a comment - I would like to request this too. I can only run 1 build at time (does not matter what branch). Using lockable resources would be a workaround - as I understand I need to configure jenkins manually (add lockable resource), the Jenkinsfile is not enough...
            Hide
            amidar Amit Dar added a comment - - edited

            I'm encountering this issue as well as a major drawback in the multi branch pipeline.

            the node i'm running on assigns the same workspace to two consecutive builds, which interrupt each other (the second build starts before the first is finished, this happens if two consecutive commits are done one after another during a short period of time)

            I kindly ask to address this issue -> would make a significant improvement!

            Show
            amidar Amit Dar added a comment - - edited I'm encountering this issue as well as a major drawback in the multi branch pipeline. the node i'm running on assigns the same workspace to two consecutive builds, which interrupt each other (the second build starts before the first is finished, this happens if two consecutive commits are done one after another during a short period of time) I kindly ask to address this issue -> would make a significant improvement!
            Hide
            llamahunter Richard Lee added a comment -

            This seems like an anti-pattern. Your branch builds should be independent of one another. If they need to share some external resource that has limited capacity, there should be a 'lockable resource' for that thing.

            Show
            llamahunter Richard Lee added a comment - This seems like an anti-pattern. Your branch builds should be independent of one another. If they need to share some external resource that has limited capacity, there should be a 'lockable resource' for that thing.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              zikphil Philippe Labat
              Votes:
              38 Vote for this issue
              Watchers:
              41 Start watching this issue

                Dates

                Created:
                Updated: