Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37588

Multibranch pipeline does not allow quiet time or throttle

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: branch-api-plugin
    • Labels:
      None
    • Environment:
      Running jenkins 2.15 with pipeline-multibranch 2.8.
    • Similar Issues:
    • Released As:
      branch-api-plugin-2.6.0

      Description

      Currently, for an individual pipeline job I can set a quiet time to avoid one-for-one commit-to-build ratio when there are many commits in a short time. I like to use the multibranch plugin to allow branch automation, but there does not appear to be a way for a multibranch config (either the meta-job itself or from within a Jenkinsfile) to define a quiet time or some other ability to throttle job runs.

        Attachments

          Issue Links

            Activity

            bsipos Brian Sipos created issue -
            oleg_nenashev Oleg Nenashev made changes -
            Field Original Value New Value
            Component/s workflow-plugin [ 18820 ]
            Component/s pipeline-_unsorted [ 21692 ]
            oleg_nenashev Oleg Nenashev made changes -
            Assignee Oleg Nenashev [ oleg_nenashev ] Jesse Glick [ jglick ]
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Changed the component.
            Actually there are many throttling options in Pipeline, but Quite Period is definitely not one of them

            Show
            oleg_nenashev Oleg Nenashev added a comment - Changed the component. Actually there are many throttling options in Pipeline, but Quite Period is definitely not one of them
            Hide
            bsipos Brian Sipos added a comment -

            My issue is with the multibranch pipeline though. I don't see any way to configure the multibranch job which will make use of job throttling on the auto-generated branch jobs.

            I see an option to "suppress automatic SCM triggering" but then the jobs will always need to be manually triggered. I don't see a way to have the auto-created per-branch job contain a "build periodically" or similar parameter. I just have a read-only view on the job configuration with unchecked options for "Build Triggers" section.

            Show
            bsipos Brian Sipos added a comment - My issue is with the multibranch pipeline though. I don't see any way to configure the multibranch job which will make use of job throttling on the auto-generated branch jobs. I see an option to "suppress automatic SCM triggering" but then the jobs will always need to be manually triggered. I don't see a way to have the auto-created per-branch job contain a "build periodically" or similar parameter. I just have a read-only view on the job configuration with unchecked options for "Build Triggers" section.
            abayer Andrew Bayer made changes -
            Component/s pipeline-general [ 21692 ]
            abayer Andrew Bayer made changes -
            Component/s workflow-plugin [ 18820 ]
            Hide
            gaalandr Andras Gaal added a comment -

            Oleg could please share some documentation regarding the throttling options of multipipeline?

            Show
            gaalandr Andras Gaal added a comment - Oleg could please share some documentation regarding the throttling options of multipipeline?
            Hide
            jglick Jesse Glick added a comment -

            try: Pipeline Syntax » properties » Throttle builds (have not personally ever used it)

            Show
            jglick Jesse Glick added a comment - try: Pipeline Syntax » properties » Throttle builds (have not personally ever used it)
            Hide
            gaalandr Andras Gaal added a comment -

            Thanks Jesse,

            I missed "Pipeline Syntax » properties" somehow. I will try to test "Throttle builds".

            However not all the options are listed there. For example I could set up 'upstream' project. With this code:

            properties([
              pipelineTriggers([
                upstream(
                  threshold: hudson.model.Result.SUCCESS,
                  upstreamProjects: 'jobB'
                )
              ])
            ])
            

            I got this code from a comment somewhere.

            Is there a documentation somewhere? Or where is the related code where I can find it out?

            Thanks,
            Andras

            Show
            gaalandr Andras Gaal added a comment - Thanks Jesse, I missed "Pipeline Syntax » properties" somehow. I will try to test "Throttle builds". However not all the options are listed there. For example I could set up 'upstream' project. With this code: properties([ pipelineTriggers([ upstream( threshold: hudson.model.Result.SUCCESS, upstreamProjects: 'jobB' ) ]) ]) I got this code from a comment somewhere. Is there a documentation somewhere? Or where is the related code where I can find it out? Thanks, Andras
            Hide
            jglick Jesse Glick added a comment -

            not all the options are listed there

            I am not sure I understand what options you are expecting to see and what you actually saw. Sounds like an independent bug at any rate.

            Documentation is sparse to say the least.

            The code is in workflow-job-plugin but there is not much to it: basically, any JobProperty compatible with Pipeline can be specified, and should appear as an option under properties. In turn the trigger property should offer any Trigger compatible with Pipeline. upstream refers to ReverseBuildTrigger in Jenkins core (for 1.x users the syntax is different).

            Show
            jglick Jesse Glick added a comment - not all the options are listed there I am not sure I understand what options you are expecting to see and what you actually saw. Sounds like an independent bug at any rate. Documentation is sparse to say the least. The code is in workflow-job-plugin but there is not much to it: basically, any JobProperty compatible with Pipeline can be specified, and should appear as an option under properties . In turn the trigger property should offer any Trigger compatible with Pipeline. upstream refers to ReverseBuildTrigger in Jenkins core (for 1.x users the syntax is different).
            rhutchison Ryan Hutchison made changes -
            Link This issue is duplicated by JENKINS-40464 [ JENKINS-40464 ]
            bluephoenix Felix Barros made changes -
            Summary Multibranch pipleine does not allow quiet time or throttle Multibranch pipeline does not allow quiet time or throttle
            Hide
            peternhale Pete Hale added a comment -

            I am trying to figure out how to apply quietPeriod to our multibranch pipeline jobs. As describe above, it is supposed to be possible to do this using properties DSL.

            If I follow the source for properties DSL (JobPropertyStep), it takes a list of JobProperty instances and applies those to the branch job.

            The quiet period "property" is defined in two ways; a global quiet period (GlobalQuietPeriodConfiguration) and also on the job (AbstractProject). 

            AFAIK, there does not seem to be a path to the property quietPeriod via properties DSL.

            Has anyone figured out how to set quiet period on a multibranch job?

            Show
            peternhale Pete Hale added a comment - I am trying to figure out how to apply quietPeriod to our multibranch pipeline jobs. As describe above, it is supposed to be possible to do this using properties DSL. If I follow the source for properties DSL (JobPropertyStep), it takes a list of JobProperty instances and applies those to the branch job. The quiet period "property" is defined in two ways; a global quiet period (GlobalQuietPeriodConfiguration) and also on the job (AbstractProject).  AFAIK, there does not seem to be a path to the property quietPeriod via properties DSL. Has anyone figured out how to set quiet period on a multibranch job?
            jglick Jesse Glick made changes -
            Assignee Jesse Glick [ jglick ]
            Hide
            jglick Jesse Glick added a comment -

            AFAIK there is no support for quiet periods in branch projects. Generally speaking, this system was introduced in the early days of Hudson to work around limitations in CVS.

            Show
            jglick Jesse Glick added a comment - AFAIK there is no support for quiet periods in branch projects. Generally speaking, this system was introduced in the early days of Hudson to work around limitations in CVS.
            jglick Jesse Glick made changes -
            Component/s branch-api-plugin [ 18621 ]
            Component/s pipeline [ 21692 ]
            Hide
            obscurerichard Richard Bullington-McGuire added a comment -

            GitHub is sometimes sending multiple webhook notifications for a single push within a second, and causing multiple jobs to spawn for us in the multibranch pipeline. Having a quiet period would have been a nice workaround.

            Show
            obscurerichard Richard Bullington-McGuire added a comment - GitHub is sometimes sending multiple webhook notifications for a single push within a second, and causing multiple jobs to spawn for us in the multibranch pipeline. Having a quiet period would have been a nice workaround.
            Hide
            akrapfl Adam Krapfl added a comment -

            I feel there are legitimate use cases for this.  Jesse Glick can you explain why this is no longer needed?  I'm struggling to come up with alternative approaches in a Jenkinsfile without a quiet period.

            Show
            akrapfl Adam Krapfl added a comment - I feel there are legitimate use cases for this.  Jesse Glick can you explain why this is no longer needed?  I'm struggling to come up with alternative approaches in a Jenkinsfile without a quiet period.
            Hide
            fdrueke Felix Drueke added a comment - - edited

            Kohsuke described the quiet feature in this (old) article.

            I still find that it is valid and needed (especially for longer builds).

            Show
            fdrueke Felix Drueke added a comment - - edited Kohsuke described the quiet feature in this (old) article . I still find that it is valid and needed (especially for longer builds).
            Hide
            remal Semyon Levin added a comment -

            We have quite complex dependencies and really need throttling options in Jenkinsfile...

            Show
            remal Semyon Levin added a comment - We have quite complex dependencies and really need throttling options in Jenkinsfile...
            Hide
            johnstosh John Muczynski added a comment -

            We have one project which needs a large quiet time while multiple commits are manually handled. But all the other projects have simple atomic commits. So we're in this battle of setting the global quiet time to 6 minutes vs 1 minute.

            So this feature would help. I'd run two Jenkins masters to solve the issue, but that hits other snags. We have several Jenkins worker machines that are needed for all these projects and the workers can't be shared among multiple masters.

             

             

            Show
            johnstosh John Muczynski added a comment - We have one project which needs a large quiet time while multiple commits are manually handled. But all the other projects have simple atomic commits. So we're in this battle of setting the global quiet time to 6 minutes vs 1 minute. So this feature would help. I'd run two Jenkins masters to solve the issue, but that hits other snags. We have several Jenkins worker machines that are needed for all these projects and the workers can't be shared among multiple masters.    
            Hide
            jglick Jesse Glick added a comment -

            Was already implemented for Declarative Pipeline (JENKINS-51227), but is apparently not available in Scripted.

            Show
            jglick Jesse Glick added a comment - Was already implemented for Declarative Pipeline ( JENKINS-51227 ), but is apparently not available in Scripted.
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-51227 [ JENKINS-51227 ]
            Hide
            jglick Jesse Glick added a comment -

            Rate limiting is possible, though, as far as I know.

            Show
            jglick Jesse Glick added a comment - Rate limiting is possible, though, as far as I know .
            Hide
            johnstosh John Muczynski added a comment -

            We do already use rate limiting (1 build at a time).

            Well, the Declarative Pipeline support might work for me. I'll have to get a new Jenkins version ...

            We're using multibranch pipelines.

             

            pipeline {
                agent any
                options {| | | | quietPeriod(15)| | | | }
            Show
            johnstosh John Muczynski added a comment - We do already use rate limiting (1 build at a time). Well, the Declarative Pipeline support might work for me. I'll have to get a new Jenkins version ... We're using multibranch pipelines.   pipeline {     agent any     options {| | | | quietPeriod(15)| | | | }
            Hide
            johnstosh John Muczynski added a comment -

            Ah, okay, as Jesse commented on the other issue: "Nothing to do with Declarative vs. Scripted, only multibranch vs. standalone."

             

            options {
                buildDiscarder(logRotator(numToKeepStr: '90', artifactNumToKeepStr: '5'))
                retry(1)
                skipDefaultCheckout()
                disableConcurrentBuilds()
                quietPeriod(600)
            }

             
            org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
            WorkflowScript: 350: Invalid option type "quietPeriod". Valid option types: [buildDiscarder, catchError, disableConcurrentBuilds, overrideIndexTriggers, retry, script, skipDefaultCheckout, skipStagesAfterUnstable, timeout, timestamps, waitUntil, withContext, withCredentials, withEnv, ws] @ line 350, column 9.
            quietPeriod(600)
            ^

            General error during semantic analysis: There's no @DataBoundConstructor on any constructor of class jenkins.model.GlobalQuietPeriodConfiguration

            org.kohsuke.stapler.NoStaplerConstructorException: There's no @DataBoundConstructor on any constructor of class jenkins.model.GlobalQuietPeriodConfiguration
            at org.kohsuke.stapler.ClassDescriptor.loadConstructorParamNames(ClassDescriptor.java:265)
            at org.jenkinsci.plugins.structs.describable.DescribableModel.<init>(DescribableModel.java:122)

            Show
            johnstosh John Muczynski added a comment - Ah, okay, as Jesse commented on the other issue: "Nothing to do with Declarative vs. Scripted, only multibranch vs. standalone."   options { buildDiscarder(logRotator(numToKeepStr: '90', artifactNumToKeepStr: '5')) retry(1) skipDefaultCheckout() disableConcurrentBuilds() quietPeriod(600) }   org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: WorkflowScript: 350: Invalid option type "quietPeriod". Valid option types: [buildDiscarder, catchError, disableConcurrentBuilds, overrideIndexTriggers, retry, script, skipDefaultCheckout, skipStagesAfterUnstable, timeout, timestamps, waitUntil, withContext, withCredentials, withEnv, ws] @ line 350, column 9. quietPeriod(600) ^ General error during semantic analysis: There's no @DataBoundConstructor on any constructor of class jenkins.model.GlobalQuietPeriodConfiguration org.kohsuke.stapler.NoStaplerConstructorException: There's no @DataBoundConstructor on any constructor of class jenkins.model.GlobalQuietPeriodConfiguration at org.kohsuke.stapler.ClassDescriptor.loadConstructorParamNames(ClassDescriptor.java:265) at org.jenkinsci.plugins.structs.describable.DescribableModel.<init>(DescribableModel.java:122)
            Hide
            johnstosh John Muczynski added a comment -

            Ooph, my bad.  I didn't make sure to update Declarative to version 1.3.2

            It looks like it is accepting the quietPeriod now, after updating blue ocean.

            Show
            johnstosh John Muczynski added a comment - Ooph, my bad.  I didn't make sure to update Declarative to version 1.3.2 It looks like it is accepting the quietPeriod now, after updating blue ocean.
            Hide
            yanstoic753 Yan Corneille added a comment -

            As noticed by Jesse Glick comment-350957 this does not seem to be available in Scripted pipelines. Are there any plans for that ?

            Show
            yanstoic753 Yan Corneille added a comment - As noticed by Jesse Glick  comment-350957 this does not seem to be available in Scripted pipelines. Are there any plans for that ?
            ogmueller Oliver Mueller made changes -
            Comment [ A very important hint is, that the quietPeriod option has to be written *WITHOUT* braces:

            so instead of writing:
            {code:java}
            quietPeriod(600){code}
            you need to write:
            {code:java}
            quietPeriod 600{code}
            jenkins will not give you a notice, warning or anything alike when using it with braces.

             

              ]
            Hide
            ogmueller Oliver Mueller added a comment - - edited

            We are using Jenkins 2.195, Multibranch 2.21 and Declarative version 1.3.9.

            Option quietPeriod seems to be accepted, but it does NOT seem to work.

            We are using a declarative multibranch pipeline with these options: 

            pipeline {
              agent any
              options {
                quietPeriod 1200
                timeout(time: 4, unit: 'HOURS')
                disableConcurrentBuilds()
              }
            ...

            The pipeline will get executed immediately.

            Show
            ogmueller Oliver Mueller added a comment - - edited We are using Jenkins 2.195, Multibranch 2.21 and Declarative version 1.3.9. Option quietPeriod seems to be accepted, but it does NOT seem to work. We are using a declarative multibranch pipeline with these options:  pipeline { agent any options { quietPeriod 1200 timeout(time: 4, unit: 'HOURS' ) disableConcurrentBuilds() } ... The pipeline will get executed immediately.
            Hide
            javi_martin Javi Martin added a comment -

            we are facing the same problem (Jenkins 2.190.1, Multibranch 2.21 and Declarative 1.3.9).

            The quietPeriod option seems to be working when calling the job from another job by sending the branch name in the URL "/projects/Build-MultibranchProject/master", but it does not work when the webhook calls the Multibranch project. 

            Multibranch plugin seems to be executing the required branch job doing a "Build now" and it bypasses the quietPeriod options.

             

             

            Show
            javi_martin Javi Martin added a comment - we are facing the same problem (Jenkins 2.190.1, Multibranch 2.21 and Declarative 1.3.9). The quietPeriod option seems to be working when calling the job from another job by sending the branch name in the URL "/projects/Build-MultibranchProject/master", but it does not work when the webhook calls the Multibranch project.  Multibranch plugin seems to be executing the required branch job doing a "Build now" and it bypasses the quietPeriod options.    
            bitwiseman Liam Newman made changes -
            Link This issue is duplicated by JENKINS-54052 [ JENKINS-54052 ]
            bitwiseman Liam Newman made changes -
            Remote Link This issue links to "PR-131 (Web Link)" [ 24005 ]
            Hide
            vincenzo_cerbone vincenzo cerbone added a comment -

            Successfully tested with Declarative (Jenkins 2.200) but had no luck to set it in Scripted fashion. As someone stated before, there's no way neither to get it generated with pipeline-syntax generator, nor trying something like this:

            properties([
                pipelineTriggers([
            				[
            					$class: 'HudsonStartupTrigger',
            					quietPeriod: '2'
            				]
            			])
            ])
            

            will produce:

             no known implementation of class hudson.triggers.Trigger is named HudsonStartupTrigger
            
            Show
            vincenzo_cerbone vincenzo cerbone added a comment - Successfully tested with Declarative (Jenkins 2.200) but had no luck to set it in Scripted fashion. As someone stated before, there's no way neither to get it generated with pipeline-syntax generator, nor trying something like this: properties([ pipelineTriggers([ [ $class: 'HudsonStartupTrigger' , quietPeriod: '2' ] ]) ]) will produce: no known implementation of class hudson.triggers.Trigger is named HudsonStartupTrigger
            Hide
            addib Dibakar Aditya added a comment - - edited

            vincenzo cerbone, did you see any success using this in a multibranch pipeline project? In our setup, the Declarative syntax with standalone pipeline project works as intended, but not in multibranch yet. This functionality is much needed, especially with the maven release plugin.

            Show
            addib Dibakar Aditya added a comment - - edited vincenzo cerbone , did you see any success using this in a multibranch pipeline project? In our setup, the Declarative syntax with standalone pipeline project works as intended, but not in multibranch yet. This functionality is much needed, especially with the maven release plugin.
            Hide
            vincenzo_cerbone vincenzo cerbone added a comment -

            Dibakar Aditya I tested it (Declarative) in multibranch as Oliver Mueller reported and I had luck to see it working. Unfortunately I needed it to work in scripted mode because JTE (Jenking Template Engine) does not support Declarative.
            I ended up using this instruction and it works!

            }}{{currentBuild.rawBuild.getParent().setQuietPeriod(10){{}}

            About maven-release-plugin, I prefer not to use it anymor. Now I'm using :
            https://github.com/aleksandr-m/gitflow-maven-plugin{{}}

            Show
            vincenzo_cerbone vincenzo cerbone added a comment - Dibakar Aditya I tested it (Declarative) in multibranch as Oliver Mueller reported and I had luck to see it working. Unfortunately I needed it to work in scripted mode because JTE (Jenking Template Engine) does not support Declarative. I ended up using this instruction and it works! }}{{currentBuild.rawBuild.getParent().setQuietPeriod(10) {{}} About maven-release-plugin, I prefer not to use it anymor. Now I'm using : https://github.com/aleksandr-m/gitflow-maven-plugin {{}}
            Hide
            addib Dibakar Aditya added a comment - - edited

            vincenzo cerbone, I have tried both the Declarative and Scripted approach that you suggested on Jenkins 2.204.1 with the latest version of plugins. Though I can see the quiet period being set in the job configuration page of the branch, multibranch still chooses to ignore it.

            Show
            addib Dibakar Aditya added a comment - - edited vincenzo cerbone , I have tried both the Declarative and Scripted approach that you suggested on Jenkins 2.204.1 with the latest version of plugins. Though I can see the quiet period being set in the job configuration page of the branch, multibranch still chooses to ignore it.
            Hide
            jdcskillet Joe Cavanaugh added a comment - - edited

            I know I got this working when I was using Polling on the Multibranch pipeline. The quietPeriod was respected... however, when I moved to webhooks to kick off the pipeline through Bitbucket the quietPeriod(60) is no longer respected. I have gotten multiple complaints from Developers on this one as well.

                options {
                     buildDiscarder logRotator(numToKeepStr: '5')
                     disableConcurrentBuilds()
                     disableResume()
                     quietPeriod 60
                     timeout(time: 3, unit: 'HOURS')
                     skipStagesAfterUnstable()
                     parallelsAlwaysFailFast()
                 }
            Show
            jdcskillet Joe Cavanaugh added a comment - - edited I know I got this working when I was using Polling on the Multibranch pipeline. The quietPeriod was respected... however, when I moved to webhooks to kick off the pipeline through Bitbucket the quietPeriod(60) is no longer respected. I have gotten multiple complaints from Developers on this one as well.     options {         buildDiscarder logRotator(numToKeepStr:  '5' )         disableConcurrentBuilds()         disableResume()         quietPeriod 60         timeout(time: 3, unit:  'HOURS' )         skipStagesAfterUnstable()         parallelsAlwaysFailFast()     }
            Hide
            brianjmurrell Brian J Murrell added a comment -

            This would be useful in GitHub if one has a GitHub Action that pushes a commit to one's PR – to prevent the the PR from building prior to the GH Actions adding the new commit.

             

            Show
            brianjmurrell Brian J Murrell added a comment - This would be useful in GitHub if one has a GitHub Action that pushes a commit to one's PR – to prevent the the PR from building prior to the GH Actions adding the new commit.  
            Hide
            hemp Chris Hemp added a comment -

            ^ Related to Github - With dependabot being enabled for a repo to update dependencies, it can be common to have a few PRs merge at once.  This quiet period would be helpful to have in those situations. 

            Show
            hemp Chris Hemp added a comment - ^ Related to Github - With dependabot being enabled for a repo to update dependencies, it can be common to have a few PRs merge at once.  This quiet period would be helpful to have in those situations. 
            bitwiseman Liam Newman made changes -
            Released As branch-api-plugin-2.6.0
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Closed [ 6 ]
            Hide
            deiwin Deiwin Sarjas added a comment -

            Would it be possible possible to configure this once for the entire GitHub Organization? I read the thread and the PRs but I don't see how one could do that.

            Show
            deiwin Deiwin Sarjas added a comment - Would it be possible possible to configure this once for the entire GitHub Organization ? I read the thread and the PRs but I don't see how one could do that.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              bsipos Brian Sipos
              Votes:
              49 Vote for this issue
              Watchers:
              54 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: