I am looking to use Throttle Concurrent Builds in a WorkFlow. I have jobs that perform automated, scheduled testing, as well as user-initiated jobs. All these jobs compete for the same set of hardware resources to run tests against. I'm currently using BuildFlows for all this and Throttle Concurrent Builds works fine. My problem is that I want to use the WorkFlow plugin for the automated jobs but TCB does not natively support WorkFlows.

      I've read that people dedicate a Node per unit of hardware to solve this problem. That's not a good solution for me since I can have 10-20 hardware units. I've also ready about people using a WorkFlow Stage, but this also doesn't work because of the user-initiated jobs that are outside of the WorkFlow.

      I've seen a presentation by CloudBees that appears to indicate that a WorkFlow can be used with any plugin, even if the plugin does not have native WorkFlow support. Is this possible for TCB and/or is there TCB support planned for WorkFlow?

      Thanks,

      Dave

          [JENKINS-31801] Pipeline Support in Throttle Concurrent Builds

          Jesse Glick added a comment -

          It would be awesome to have multiple executors, while being able to ensure that only one job requiring certain ressources (like an iOS simulator) is sent to a Node at a time.

          You can do this already with the lock step.

          Jesse Glick added a comment - It would be awesome to have multiple executors, while being able to ensure that only one job requiring certain ressources (like an iOS simulator) is sent to a Node at a time. You can do this already with the lock step.

          You can do this already with the lock step.

          AFAIK only at a global level — in that particular example, you'd want to be able to lock the "ios-simulator" resource per node (as only one simulator can run on a machine simultaneously).

          Christopher Orr added a comment - You can do this already with the lock step. AFAIK only at a global level — in that particular example, you'd want to be able to lock the "ios-simulator" resource per node (as only one simulator can run on a machine simultaneously).

          Andrew Bayer added a comment -

          jglick Not exactly - the use case that Throttle was written for originally was as follows:

          • Multiple {{Node}}s with multiple executors on each of them.
          • Two different jobs (or types of jobs) - let's say backend tests and frontend tests.
          • You can only run one backend test and one frontend test at a time on each node, but you can run one of each.
          • So Throttle lets you say, for a given job or "category" applied to multiple jobs, "You can run X instances of this job/jobs with this category on a given node (or all nodes) at the same time"
          • The result is that you can have four nodes with two executors each, with each node running one backend test job and one frontend test job. Tada.

          I couldn't figure out a way to get this to work with lock because lock is running before the node step. So we can't do something like lock(NODE_NAME + "-this-branch"). I might be able to rework lock (or add a new step there) to do something like I'm doing in the throttle PR with a context, but honestly, I think it makes most sense to use throttle - this fits into the overall model of the plugin and allows interoperability with non-Pipeline jobs too.

          Andrew Bayer added a comment - jglick Not exactly - the use case that Throttle was written for originally was as follows: Multiple {{Node}}s with multiple executors on each of them. Two different jobs (or types of jobs) - let's say backend tests and frontend tests. You can only run one backend test and one frontend test at a time on each node, but you can run one of each. So Throttle lets you say, for a given job or "category" applied to multiple jobs, "You can run X instances of this job/jobs with this category on a given node (or all nodes) at the same time" The result is that you can have four nodes with two executors each, with each node running one backend test job and one frontend test job. Tada. I couldn't figure out a way to get this to work with lock because lock is running before the node step. So we can't do something like lock(NODE_NAME + "-this-branch") . I might be able to rework lock (or add a new step there) to do something like I'm doing in the throttle PR with a context, but honestly, I think it makes most sense to use throttle - this fits into the overall model of the plugin and allows interoperability with non-Pipeline jobs too.

          Jesse Glick added a comment -

          lock is running before the node step

          Well, you can give each node extra executors and

          node('backend-or-frontend') {
            lock("frontend-$NODE_NAME") {
              sh './run-frontend-tests'
            }
          }
          

          Of course this ignores freestyle interoperability etc., and means that waiting builds will be displayed as “on” one of the executors even though they are idle (perhaps fixable with a tryLock step, or perhaps PlaceholderExecutable display should consider PauseAction).

          Jesse Glick added a comment - lock is running before the node step Well, you can give each node extra executors and node( 'backend-or-frontend' ) { lock( "frontend-$NODE_NAME" ) { sh './run-frontend-tests' } } Of course this ignores freestyle interoperability etc., and means that waiting builds will be displayed as “on” one of the executors even though they are idle (perhaps fixable with a tryLock step, or perhaps PlaceholderExecutable display should consider PauseAction ).

          Another use case for the plugin that I don't think can be solved by using lock is the case where people need to limit the total concurrent builds across all nodes. I use this for throttling deployments due to memory constraints.

          Benjamin Manuel added a comment - Another use case for the plugin that I don't think can be solved by using lock is the case where people need to limit the total concurrent builds across all nodes. I use this for throttling deployments due to memory constraints.

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          pom.xml
          src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java
          src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java
          src/main/java/hudson/plugins/throttleconcurrents/pipeline/ThrottleStep.java
          src/main/java/hudson/plugins/throttleconcurrents/pipeline/ThrottleStepExecution.java
          src/main/resources/hudson/plugins/throttleconcurrents/Messages.properties
          src/main/resources/hudson/plugins/throttleconcurrents/ThrottleJobProperty/help.html
          src/main/resources/hudson/plugins/throttleconcurrents/pipeline/Messages.properties
          src/main/resources/hudson/plugins/throttleconcurrents/pipeline/ThrottleStep/config.jelly
          src/main/resources/hudson/plugins/throttleconcurrents/pipeline/ThrottleStep/help-categories.html
          src/test/java/hudson/plugins/throttleconcurrents/ThrottleConcurrentTest.java
          src/test/java/hudson/plugins/throttleconcurrents/ThrottleStepTest.java
          http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/0bbd0792eb88ff33c81cc0efa5e04b651542fb6d
          Log:
          JENKINS-31801 Add Pipeline throttle(category) step (#46)

          • JENKINS-31801 Initial work on throttle(category) step - needs tests
          • Reworked to no longer rely on StepExecutions.
          • Add a trailing newline to messages.
          • Make findbugs happy.
          • Whoops, this needs to take a block
          • Initial test, actually working

          Needed to bump to newer dependency versions, most notably to get
          PlaceholderTask.getNode(). Still a work in progress, mind you.

          • Cleanup, commenting, javadoc
          • Test across all nodes

          Also discovered that Run<?,?> is a very bad Map key.

          • Add interop with freestyle test
          • Add snippet generator support.
          • Add snippetizer support and test
          • Review comments
          • Check for pending PlaceholderTasks as well.
          • Allow multiple comma-separated categories
          • Minor review responses, moving step UI to correct dir
          • Go away, empty category names!
          • Fixed up snippetizer, switched to a list of strings for the throttle step
          • Check for and respond to duplicate or non-existent category names
          • Adding help for ThrottleJobProperty pointing out it doesn't work for Pipeline
          • unmodifiableList

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: pom.xml src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java src/main/java/hudson/plugins/throttleconcurrents/pipeline/ThrottleStep.java src/main/java/hudson/plugins/throttleconcurrents/pipeline/ThrottleStepExecution.java src/main/resources/hudson/plugins/throttleconcurrents/Messages.properties src/main/resources/hudson/plugins/throttleconcurrents/ThrottleJobProperty/help.html src/main/resources/hudson/plugins/throttleconcurrents/pipeline/Messages.properties src/main/resources/hudson/plugins/throttleconcurrents/pipeline/ThrottleStep/config.jelly src/main/resources/hudson/plugins/throttleconcurrents/pipeline/ThrottleStep/help-categories.html src/test/java/hudson/plugins/throttleconcurrents/ThrottleConcurrentTest.java src/test/java/hudson/plugins/throttleconcurrents/ThrottleStepTest.java http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/0bbd0792eb88ff33c81cc0efa5e04b651542fb6d Log: JENKINS-31801 Add Pipeline throttle(category) step (#46) JENKINS-31801 Initial work on throttle(category) step - needs tests Reworked to no longer rely on StepExecutions. Add a trailing newline to messages. Make findbugs happy. Whoops, this needs to take a block Initial test, actually working Needed to bump to newer dependency versions, most notably to get PlaceholderTask.getNode(). Still a work in progress, mind you. Cleanup, commenting, javadoc Test across all nodes Also discovered that Run<?,?> is a very bad Map key. Add interop with freestyle test Add snippet generator support. Add snippetizer support and test Review comments Check for pending PlaceholderTasks as well. Allow multiple comma-separated categories Minor review responses, moving step UI to correct dir Go away, empty category names! Fixed up snippetizer, switched to a list of strings for the throttle step Check for and respond to duplicate or non-existent category names Adding help for ThrottleJobProperty pointing out it doesn't work for Pipeline unmodifiableList

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          CHANGELOG.md
          LICENSE.txt
          README
          README.md
          doc/images/abstractProject_jobProperty.png
          doc/images/abstractProject_matrixFlags.png
          doc/images/global_categoryConfig.png
          pom.xml
          http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/e5d829ba44d89e60ac9cf1e6e12691e8d14e81aa
          Log:
          JENKINS-31801 - Documentation Revamp + Pipeline support docs (#47)

          • Add License
          • Revamp Readme from Wiki, add Pipeline Support documentation towards JENKINS-31801
          • Migrate changelog from Wiki, add JENKINS-31801 reference in "Coming Soon"
          • Update Parent POM
          • Update documentation to reflect the current state of Pipeline support

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: CHANGELOG.md LICENSE.txt README README.md doc/images/abstractProject_jobProperty.png doc/images/abstractProject_matrixFlags.png doc/images/global_categoryConfig.png pom.xml http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/e5d829ba44d89e60ac9cf1e6e12691e8d14e81aa Log: JENKINS-31801 - Documentation Revamp + Pipeline support docs (#47) Add License Revamp Readme from Wiki, add Pipeline Support documentation towards JENKINS-31801 Migrate changelog from Wiki, add JENKINS-31801 reference in "Coming Soon" Update Parent POM Update documentation to reflect the current state of Pipeline support

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          README
          README.md
          doc/images/abstractProject_jobProperty.png
          doc/images/abstractProject_matrixFlags.png
          doc/images/global_categoryConfig.png
          http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/b157b72c075e70bb83ca3ea964da97ef5febdf38
          Log:
          Revamp Readme from Wiki, add Pipeline Support documentation towards JENKINS-31801

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: README README.md doc/images/abstractProject_jobProperty.png doc/images/abstractProject_matrixFlags.png doc/images/global_categoryConfig.png http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/b157b72c075e70bb83ca3ea964da97ef5febdf38 Log: Revamp Readme from Wiki, add Pipeline Support documentation towards JENKINS-31801

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          CHANGELOG.md
          http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/eb74255621c82d5df3b1caf20c0e6f3f7e0071e6
          Log:
          Migrate changelog from Wiki, add JENKINS-31801 reference in "Coming Soon"

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: CHANGELOG.md http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/eb74255621c82d5df3b1caf20c0e6f3f7e0071e6 Log: Migrate changelog from Wiki, add JENKINS-31801 reference in "Coming Soon"

          Oleg Nenashev added a comment -

          Oleg Nenashev added a comment - The feature has been released in the TCB plugin 2.0. https://github.com/jenkinsci/throttle-concurrent-builds-plugin/blob/master/CHANGELOG.md#20

            abayer Andrew Bayer
            daveml Dave Lawrence
            Votes:
            35 Vote for this issue
            Watchers:
            52 Start watching this issue

              Created:
              Updated:
              Resolved: