Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44411

Support throttling of blocks without node() in the throttle() step

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      We have a limit on the number of licenses we can use concurrently. Before pipeline as code we could use the throttle to limit the number of jobs to 16.
      With pipeline as code and 2.0 of the plugin we cannot limit jobs unless we create the jobs and trigger in parallel.
      Is there any intention to make the lateste plugin work in the same way as before without having to do parallel jobs

        Attachments

          Activity

          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          Hi Robert,

          I do not exactly understand what's wrong with the current implementation. throttle() step works well without Parallel, e.g. the following sample works well:

          throttle(['test_2']) {
            node() {
                  sh "sleep 500"
                  echo "Done"
            }
          }

          The thing is that the Pipeline actually starts, but node() step will synchronize on the category

          Show
          oleg_nenashev Oleg Nenashev added a comment - Hi Robert, I do not exactly understand what's wrong with the current implementation. throttle() step works well without Parallel, e.g. the following sample works well: throttle( ['test_2'] ) {   node() {         sh "sleep 500"         echo "Done"   } } The thing is that the Pipeline actually starts, but node() step will synchronize on the category
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          Anyway, it worth providing such example in the docs

          Show
          oleg_nenashev Oleg Nenashev added a comment - Anyway, it worth providing such example in the docs
          Hide
          rob_squire Robert Squire added a comment -

          I have done the following:
          case (target_option != "HOST" && comp_tool == "ARM"):
          throttle(['ARM_LICENSES'])
          {
          stage("Build libraries")
          {
          try
          {
          sh ''' mkdir BUILD
          cd BUILD
          cmake -GNinja -DCTEST_GENERATE_XUNIT_FILES=ON -DCMAKE_TOOLCHAIN_FILE=$CBS/toolchains/ARMCortexR4Toolchain.cmake -DCBS_RUN_VPUML=OFF ${WORKSPACE}/sdmos/cmake/targets/repository
          ninja -v '''
          currentBuild.result = "${env.job_pass}"
          }
          catch (Exception err)
          {
          currentBuild.result = "${env.job_fail}"
          CommonUtilities.stage_failure("Build libraries")
          println err
          throw err
          }
          }
          }

          If it set ARM_LICENSES to 2 in Jenkins config then trigger threee pipelines that has this in I see all three jobs running the build stage at the same time.
          Do I need to move to beore the node as in the example?

          Show
          rob_squire Robert Squire added a comment - I have done the following: case (target_option != "HOST" && comp_tool == "ARM"): throttle( ['ARM_LICENSES'] ) { stage("Build libraries") { try { sh ''' mkdir BUILD cd BUILD cmake -GNinja -DCTEST_GENERATE_XUNIT_FILES=ON -DCMAKE_TOOLCHAIN_FILE=$CBS/toolchains/ARMCortexR4Toolchain.cmake -DCBS_RUN_VPUML=OFF ${WORKSPACE}/sdmos/cmake/targets/repository ninja -v ''' currentBuild.result = "${env.job_pass}" } catch (Exception err) { currentBuild.result = "${env.job_fail}" CommonUtilities.stage_failure("Build libraries") println err throw err } } } If it set ARM_LICENSES to 2 in Jenkins config then trigger threee pipelines that has this in I see all three jobs running the build stage at the same time. Do I need to move to beore the node as in the example?
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          Yes, throttling will work on the node() level only. so the node() definition should be within the throttle() block.

          Andrew Bayer just to be flexible with support of other throttling scopes, maybe we could add something like a  ThrottlePlaceholderTask? Maybe a flyweight one... It would be similar with node() acquisition logic, which is not a state of the art design though. But it would help. WDYT?

          Show
          oleg_nenashev Oleg Nenashev added a comment - Yes, throttling will work on the node() level only. so the node() definition should be within the throttle() block. Andrew Bayer just to be flexible with support of other throttling scopes, maybe we could add something like a  ThrottlePlaceholderTask? Maybe a flyweight one... It would be similar with node() acquisition logic, which is not a state of the art design though. But it would help. WDYT?
          Hide
          rob_squire Robert Squire added a comment -

          Ok if I put it before node then the other job waits.
          Thanks for the help

          Show
          rob_squire Robert Squire added a comment - Ok if I put it before node then the other job waits. Thanks for the help
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          If you do not mind, let's keep this bug open. I will discuss it with Andrew Bayer and probably convert it to RFE

          Show
          oleg_nenashev Oleg Nenashev added a comment - If you do not mind, let's keep this bug open. I will discuss it with Andrew Bayer and probably convert it to RFE
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          Created https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/49 with documentation improvements. Robert Squire I will appreciate feedback in the pull request.

          Regarding this issue, I am going to convert it to RFE

          Show
          oleg_nenashev Oleg Nenashev added a comment - Created https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/49 with documentation improvements. Robert Squire I will appreciate feedback in the pull request. Regarding this issue, I am going to convert it to RFE
          Hide
          rob_squire Robert Squire added a comment -

          Ok happy to give feedback

          Show
          rob_squire Robert Squire added a comment - Ok happy to give feedback
          Hide
          rob_squire Robert Squire added a comment -

          In a pipeline job we do sometimes have two build stages one might use the arm license and another build in the same pipeline might use msbuild to build a host exe. It would be good to be able to limit the arm build but allow msbuild builds to run without any throttling.

          Show
          rob_squire Robert Squire added a comment - In a pipeline job we do sometimes have two build stages one might use the arm license and another build in the same pipeline might use msbuild to build a host exe. It would be good to be able to limit the arm build but allow msbuild builds to run without any throttling.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          README.md
          http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/ad8072a541fcacc0175b6468092e19392e0b91e4
          Log:
          JENKINS-44411 - Extend documentation to clarify throttle() behavior and supported use-cases

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: README.md http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/ad8072a541fcacc0175b6468092e19392e0b91e4 Log: JENKINS-44411 - Extend documentation to clarify throttle() behavior and supported use-cases
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          README.md
          http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/68f1f3fa1cd3d96ae71675008bbff0e4ba61ab52
          Log:
          JENKINS-44411 - Drop the confusing sentence

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: README.md http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/68f1f3fa1cd3d96ae71675008bbff0e4ba61ab52 Log: JENKINS-44411 - Drop the confusing sentence
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          README.md
          http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/7c61f6119c67ffb046bcf3afb596671b12011bd8
          Log:
          Merge pull request #49 from oleg-nenashev/docs/JENKINS-44411-clarify-throttle-step

          JENKINS-44411 - Extend documentation to clarify throttle() behavior

          Compare: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/compare/1497858da1f6...7c61f6119c67

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: README.md http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/7c61f6119c67ffb046bcf3afb596671b12011bd8 Log: Merge pull request #49 from oleg-nenashev/docs/ JENKINS-44411 -clarify-throttle-step JENKINS-44411 - Extend documentation to clarify throttle() behavior Compare: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/compare/1497858da1f6...7c61f6119c67
          Hide
          rob_squire Robert Squire added a comment -

          The new documentation is now clear on the use of throttle for pipelines.

          Show
          rob_squire Robert Squire added a comment - The new documentation is now clear on the use of throttle for pipelines.
          Hide
          cyberjoac Joachim Hagege added a comment -

          Hi, any update on support of declarative pipeline flows ? 
          Thanks

          Show
          cyberjoac Joachim Hagege added a comment - Hi, any update on support of declarative pipeline flows ?  Thanks
          Hide
          5im Simon K added a comment -

          Hi, we switched to the lock-Plugin for this use case:
          https://wiki.jenkins.io/display/JENKINS/Lockable+Resources+Plugin

          Worked as aspected

          Show
          5im Simon K added a comment - Hi, we switched to the lock-Plugin for this use case: https://wiki.jenkins.io/display/JENKINS/Lockable+Resources+Plugin Worked as aspected
          Hide
          marcus_phi Marcus Philip added a comment - - edited

          Lockable resources solves some use cases in declarative pipelines, but not all.

          I want to limit builds of a declarative pipeline project to one build per node. I can't achieve this.

          The documentation in "Example 3: Throttling of declarative pipelines" is lacking/misguiding. I've tested, and it has no throttling effect on a single declarative Pipeline job. Does it only throttle between jobs using same category? (And in that case, are different branches in a multi-branch pipeline considered different jobs?) IMHO, that feature is not as useful as being able to throttle on the same job.

          Show
          marcus_phi Marcus Philip added a comment - - edited Lockable resources solves some use cases in declarative pipelines, but not all. I want to limit builds of a declarative pipeline project to one build per node. I can't achieve this. The documentation in "Example 3: Throttling of declarative pipelines" is lacking/misguiding. I've tested, and it has no throttling effect on a single declarative Pipeline job. Does it only throttle between jobs using same category? (And in that case, are different branches in a multi-branch pipeline considered different jobs?) IMHO, that feature is not as useful as being able to throttle on the same job.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            rob_squire Robert Squire
            Votes:
            12 Vote for this issue
            Watchers:
            15 Start watching this issue

              Dates

              Created:
              Updated: