I am looking to use Throttle Concurrent Builds in a WorkFlow. I have jobs that perform automated, scheduled testing, as well as user-initiated jobs. All these jobs compete for the same set of hardware resources to run tests against. I'm currently using BuildFlows for all this and Throttle Concurrent Builds works fine. My problem is that I want to use the WorkFlow plugin for the automated jobs but TCB does not natively support WorkFlows.

      I've read that people dedicate a Node per unit of hardware to solve this problem. That's not a good solution for me since I can have 10-20 hardware units. I've also ready about people using a WorkFlow Stage, but this also doesn't work because of the user-initiated jobs that are outside of the WorkFlow.

      I've seen a presentation by CloudBees that appears to indicate that a WorkFlow can be used with any plugin, even if the plugin does not have native WorkFlow support. Is this possible for TCB and/or is there TCB support planned for WorkFlow?

      Thanks,

      Dave

          [JENKINS-31801] Pipeline Support in Throttle Concurrent Builds

          Andrew Bayer added a comment -

          Andrew Bayer added a comment - Looks like https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/32 is pertinent - diving in there.

          Andrew Bayer added a comment -

          So that change does make it possible to use TCB in its normal way with Workflow jobs - i.e., you can't use TCB within the Workflow itself, but you can use it to throttle groups of Workflow jobs, etc. I'm guessing that's not all you want, but is that a start?

          Andrew Bayer added a comment - So that change does make it possible to use TCB in its normal way with Workflow jobs - i.e., you can't use TCB within the Workflow itself, but you can use it to throttle groups of Workflow jobs, etc. I'm guessing that's not all you want, but is that a start?

          Andrew Bayer added a comment -

          Ping?

          Andrew Bayer added a comment - Ping?

          Oleg Nenashev added a comment - - edited

          daveml. Yes, this is a known case (was also a blocker at my previous company). Actually I would say that workflow's implementation is not very suitable to remote hardware management due to the failover robustness (which cannot be disabled IIRC). If Jenkins or its node restarts, WF will continue the execution and will restore ONLY it's own state. Currently the state of the remote stuff cannot be restored from Workflow => jobs may behave unreliably.

          > I've seen a presentation by CloudBees that appears to indicate that a WorkFlow can be used with any plugin

          I doubt it's correct. Could you refer this presentation? If it is misguiding, we will need to fix it.

          Oleg Nenashev added a comment - - edited daveml . Yes, this is a known case (was also a blocker at my previous company). Actually I would say that workflow's implementation is not very suitable to remote hardware management due to the failover robustness (which cannot be disabled IIRC). If Jenkins or its node restarts, WF will continue the execution and will restore ONLY it's own state. Currently the state of the remote stuff cannot be restored from Workflow => jobs may behave unreliably. > I've seen a presentation by CloudBees that appears to indicate that a WorkFlow can be used with any plugin I doubt it's correct. Could you refer this presentation? If it is misguiding, we will need to fix it.

          Andrew Bayer added a comment -

          jglick - pinging you on this to warn you I wanna talk this over with you tomorrow. =)

          Andrew Bayer added a comment - jglick - pinging you on this to warn you I wanna talk this over with you tomorrow. =)

          Dave Lawrence added a comment -

          Thanks to all who have responded. I appreciate the effort.

          To Andrew's point "...but you can use it to throttle groups of Workflow jobs..."
          Using TCB within the Workflow is where the value is. Applying a throttle over tan entire Workflow causes other problems as I'm sure you can imagine.

          Oleg:
          I'm sorry but I could not find the presentation. I know Cyrille gave it and it was older, perhaps Jan 2015. As for the Workflow failover, I understand the point, but it's not a concern for me. Hopefully TCB support can get worked into Workflow.

          Dave

          Dave Lawrence added a comment - Thanks to all who have responded. I appreciate the effort. To Andrew's point "...but you can use it to throttle groups of Workflow jobs..." Using TCB within the Workflow is where the value is. Applying a throttle over tan entire Workflow causes other problems as I'm sure you can imagine. Oleg: I'm sorry but I could not find the presentation. I know Cyrille gave it and it was older, perhaps Jan 2015. As for the Workflow failover, I understand the point, but it's not a concern for me. Hopefully TCB support can get worked into Workflow. Dave

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          COMPATIBILITY.md
          http://jenkins-ci.org/commit/workflow-plugin/5578cc034790402d514557648f076db44143ed02
          Log:
          Add throttle-concurrent-builds plugin to the list

          https://issues.jenkins-ci.org/browse/JENKINS-31801

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: COMPATIBILITY.md http://jenkins-ci.org/commit/workflow-plugin/5578cc034790402d514557648f076db44143ed02 Log: Add throttle-concurrent-builds plugin to the list https://issues.jenkins-ci.org/browse/JENKINS-31801

          Jesse Glick added a comment -

          The easy integration would be to make the JobProperty applicable to Job, and call SubTask.getOwnerTask. The nicer integration would be a step that you could call on a particular node.

          Jesse Glick added a comment - The easy integration would be to make the JobProperty applicable to Job , and call SubTask.getOwnerTask . The nicer integration would be a step that you could call on a particular node .

          mcrooney added a comment -

          I suppose a work-around in the meantime could be to create a new node running on the master (ssh localhost or run slave.jar), that has one executor and a label, and then require that label in your job. You can decrease the number of executors on your master by one if you want it to be identical from a bandwidth perspective!

          mcrooney added a comment - I suppose a work-around in the meantime could be to create a new node running on the master (ssh localhost or run slave.jar), that has one executor and a label, and then require that label in your job. You can decrease the number of executors on your master by one if you want it to be identical from a bandwidth perspective!

          Andrew Bayer added a comment -

          fwiw, I'm going to try to take another look at this soon.

          Andrew Bayer added a comment - fwiw, I'm going to try to take another look at this soon.

          mcrooney added a comment -

          Thanks Andrew, that'd be awesome! For now I am using the executor hack I described above, but it isn't very elegant so it'd be awesome to have native support for this.

          mcrooney added a comment - Thanks Andrew, that'd be awesome! For now I am using the executor hack I described above, but it isn't very elegant so it'd be awesome to have native support for this.

          Any news/progress on the ability to use the Throttle from Pipelines? It would be awesome to have multiple executors, while being able to ensure that only one job requiring certain ressources (like an iOS simulator) is sent to a Node at a time.

          Niels Frydenholm added a comment - Any news/progress on the ability to use the Throttle from Pipelines? It would be awesome to have multiple executors, while being able to ensure that only one job requiring certain ressources (like an iOS simulator) is sent to a Node at a time.

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          pom.xml
          src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java
          src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java
          src/main/java/hudson/plugins/throttleconcurrents/pipeline/ThrottleStep.java
          src/main/java/hudson/plugins/throttleconcurrents/pipeline/ThrottleStepExecution.java
          src/main/java/hudson/plugins/throttleconcurrents/pipeline/ThrottledStepInfo.java
          src/main/resources/hudson/plugins/throttleconcurrents/Messages.properties
          http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/af877cd2ebd90c669a2bc1f87bcc0b3814a8a108
          Log:
          JENKINS-31801 Initial work on throttle(category) step - needs tests

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: pom.xml src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java src/main/java/hudson/plugins/throttleconcurrents/pipeline/ThrottleStep.java src/main/java/hudson/plugins/throttleconcurrents/pipeline/ThrottleStepExecution.java src/main/java/hudson/plugins/throttleconcurrents/pipeline/ThrottledStepInfo.java src/main/resources/hudson/plugins/throttleconcurrents/Messages.properties http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/af877cd2ebd90c669a2bc1f87bcc0b3814a8a108 Log: JENKINS-31801 Initial work on throttle(category) step - needs tests

          Andrew Bayer added a comment -

          Very initial work-in-progress PR up at https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/46 to add a throttle("some-category") step.

          Andrew Bayer added a comment - Very initial work-in-progress PR up at https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/46  to add a throttle("some-category") step.

          Jesse Glick added a comment -

          It would be awesome to have multiple executors, while being able to ensure that only one job requiring certain ressources (like an iOS simulator) is sent to a Node at a time.

          You can do this already with the lock step.

          Jesse Glick added a comment - It would be awesome to have multiple executors, while being able to ensure that only one job requiring certain ressources (like an iOS simulator) is sent to a Node at a time. You can do this already with the lock step.

          You can do this already with the lock step.

          AFAIK only at a global level — in that particular example, you'd want to be able to lock the "ios-simulator" resource per node (as only one simulator can run on a machine simultaneously).

          Christopher Orr added a comment - You can do this already with the lock step. AFAIK only at a global level — in that particular example, you'd want to be able to lock the "ios-simulator" resource per node (as only one simulator can run on a machine simultaneously).

          Andrew Bayer added a comment -

          jglick Not exactly - the use case that Throttle was written for originally was as follows:

          • Multiple {{Node}}s with multiple executors on each of them.
          • Two different jobs (or types of jobs) - let's say backend tests and frontend tests.
          • You can only run one backend test and one frontend test at a time on each node, but you can run one of each.
          • So Throttle lets you say, for a given job or "category" applied to multiple jobs, "You can run X instances of this job/jobs with this category on a given node (or all nodes) at the same time"
          • The result is that you can have four nodes with two executors each, with each node running one backend test job and one frontend test job. Tada.

          I couldn't figure out a way to get this to work with lock because lock is running before the node step. So we can't do something like lock(NODE_NAME + "-this-branch"). I might be able to rework lock (or add a new step there) to do something like I'm doing in the throttle PR with a context, but honestly, I think it makes most sense to use throttle - this fits into the overall model of the plugin and allows interoperability with non-Pipeline jobs too.

          Andrew Bayer added a comment - jglick Not exactly - the use case that Throttle was written for originally was as follows: Multiple {{Node}}s with multiple executors on each of them. Two different jobs (or types of jobs) - let's say backend tests and frontend tests. You can only run one backend test and one frontend test at a time on each node, but you can run one of each. So Throttle lets you say, for a given job or "category" applied to multiple jobs, "You can run X instances of this job/jobs with this category on a given node (or all nodes) at the same time" The result is that you can have four nodes with two executors each, with each node running one backend test job and one frontend test job. Tada. I couldn't figure out a way to get this to work with lock because lock is running before the node step. So we can't do something like lock(NODE_NAME + "-this-branch") . I might be able to rework lock (or add a new step there) to do something like I'm doing in the throttle PR with a context, but honestly, I think it makes most sense to use throttle - this fits into the overall model of the plugin and allows interoperability with non-Pipeline jobs too.

          Jesse Glick added a comment -

          lock is running before the node step

          Well, you can give each node extra executors and

          node('backend-or-frontend') {
            lock("frontend-$NODE_NAME") {
              sh './run-frontend-tests'
            }
          }
          

          Of course this ignores freestyle interoperability etc., and means that waiting builds will be displayed as “on” one of the executors even though they are idle (perhaps fixable with a tryLock step, or perhaps PlaceholderExecutable display should consider PauseAction).

          Jesse Glick added a comment - lock is running before the node step Well, you can give each node extra executors and node( 'backend-or-frontend' ) { lock( "frontend-$NODE_NAME" ) { sh './run-frontend-tests' } } Of course this ignores freestyle interoperability etc., and means that waiting builds will be displayed as “on” one of the executors even though they are idle (perhaps fixable with a tryLock step, or perhaps PlaceholderExecutable display should consider PauseAction ).

          Another use case for the plugin that I don't think can be solved by using lock is the case where people need to limit the total concurrent builds across all nodes. I use this for throttling deployments due to memory constraints.

          Benjamin Manuel added a comment - Another use case for the plugin that I don't think can be solved by using lock is the case where people need to limit the total concurrent builds across all nodes. I use this for throttling deployments due to memory constraints.

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          pom.xml
          src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java
          src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java
          src/main/java/hudson/plugins/throttleconcurrents/pipeline/ThrottleStep.java
          src/main/java/hudson/plugins/throttleconcurrents/pipeline/ThrottleStepExecution.java
          src/main/resources/hudson/plugins/throttleconcurrents/Messages.properties
          src/main/resources/hudson/plugins/throttleconcurrents/ThrottleJobProperty/help.html
          src/main/resources/hudson/plugins/throttleconcurrents/pipeline/Messages.properties
          src/main/resources/hudson/plugins/throttleconcurrents/pipeline/ThrottleStep/config.jelly
          src/main/resources/hudson/plugins/throttleconcurrents/pipeline/ThrottleStep/help-categories.html
          src/test/java/hudson/plugins/throttleconcurrents/ThrottleConcurrentTest.java
          src/test/java/hudson/plugins/throttleconcurrents/ThrottleStepTest.java
          http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/0bbd0792eb88ff33c81cc0efa5e04b651542fb6d
          Log:
          JENKINS-31801 Add Pipeline throttle(category) step (#46)

          • JENKINS-31801 Initial work on throttle(category) step - needs tests
          • Reworked to no longer rely on StepExecutions.
          • Add a trailing newline to messages.
          • Make findbugs happy.
          • Whoops, this needs to take a block
          • Initial test, actually working

          Needed to bump to newer dependency versions, most notably to get
          PlaceholderTask.getNode(). Still a work in progress, mind you.

          • Cleanup, commenting, javadoc
          • Test across all nodes

          Also discovered that Run<?,?> is a very bad Map key.

          • Add interop with freestyle test
          • Add snippet generator support.
          • Add snippetizer support and test
          • Review comments
          • Check for pending PlaceholderTasks as well.
          • Allow multiple comma-separated categories
          • Minor review responses, moving step UI to correct dir
          • Go away, empty category names!
          • Fixed up snippetizer, switched to a list of strings for the throttle step
          • Check for and respond to duplicate or non-existent category names
          • Adding help for ThrottleJobProperty pointing out it doesn't work for Pipeline
          • unmodifiableList

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: pom.xml src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java src/main/java/hudson/plugins/throttleconcurrents/pipeline/ThrottleStep.java src/main/java/hudson/plugins/throttleconcurrents/pipeline/ThrottleStepExecution.java src/main/resources/hudson/plugins/throttleconcurrents/Messages.properties src/main/resources/hudson/plugins/throttleconcurrents/ThrottleJobProperty/help.html src/main/resources/hudson/plugins/throttleconcurrents/pipeline/Messages.properties src/main/resources/hudson/plugins/throttleconcurrents/pipeline/ThrottleStep/config.jelly src/main/resources/hudson/plugins/throttleconcurrents/pipeline/ThrottleStep/help-categories.html src/test/java/hudson/plugins/throttleconcurrents/ThrottleConcurrentTest.java src/test/java/hudson/plugins/throttleconcurrents/ThrottleStepTest.java http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/0bbd0792eb88ff33c81cc0efa5e04b651542fb6d Log: JENKINS-31801 Add Pipeline throttle(category) step (#46) JENKINS-31801 Initial work on throttle(category) step - needs tests Reworked to no longer rely on StepExecutions. Add a trailing newline to messages. Make findbugs happy. Whoops, this needs to take a block Initial test, actually working Needed to bump to newer dependency versions, most notably to get PlaceholderTask.getNode(). Still a work in progress, mind you. Cleanup, commenting, javadoc Test across all nodes Also discovered that Run<?,?> is a very bad Map key. Add interop with freestyle test Add snippet generator support. Add snippetizer support and test Review comments Check for pending PlaceholderTasks as well. Allow multiple comma-separated categories Minor review responses, moving step UI to correct dir Go away, empty category names! Fixed up snippetizer, switched to a list of strings for the throttle step Check for and respond to duplicate or non-existent category names Adding help for ThrottleJobProperty pointing out it doesn't work for Pipeline unmodifiableList

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          CHANGELOG.md
          LICENSE.txt
          README
          README.md
          doc/images/abstractProject_jobProperty.png
          doc/images/abstractProject_matrixFlags.png
          doc/images/global_categoryConfig.png
          pom.xml
          http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/e5d829ba44d89e60ac9cf1e6e12691e8d14e81aa
          Log:
          JENKINS-31801 - Documentation Revamp + Pipeline support docs (#47)

          • Add License
          • Revamp Readme from Wiki, add Pipeline Support documentation towards JENKINS-31801
          • Migrate changelog from Wiki, add JENKINS-31801 reference in "Coming Soon"
          • Update Parent POM
          • Update documentation to reflect the current state of Pipeline support

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: CHANGELOG.md LICENSE.txt README README.md doc/images/abstractProject_jobProperty.png doc/images/abstractProject_matrixFlags.png doc/images/global_categoryConfig.png pom.xml http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/e5d829ba44d89e60ac9cf1e6e12691e8d14e81aa Log: JENKINS-31801 - Documentation Revamp + Pipeline support docs (#47) Add License Revamp Readme from Wiki, add Pipeline Support documentation towards JENKINS-31801 Migrate changelog from Wiki, add JENKINS-31801 reference in "Coming Soon" Update Parent POM Update documentation to reflect the current state of Pipeline support

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          README
          README.md
          doc/images/abstractProject_jobProperty.png
          doc/images/abstractProject_matrixFlags.png
          doc/images/global_categoryConfig.png
          http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/b157b72c075e70bb83ca3ea964da97ef5febdf38
          Log:
          Revamp Readme from Wiki, add Pipeline Support documentation towards JENKINS-31801

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: README README.md doc/images/abstractProject_jobProperty.png doc/images/abstractProject_matrixFlags.png doc/images/global_categoryConfig.png http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/b157b72c075e70bb83ca3ea964da97ef5febdf38 Log: Revamp Readme from Wiki, add Pipeline Support documentation towards JENKINS-31801

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          CHANGELOG.md
          http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/eb74255621c82d5df3b1caf20c0e6f3f7e0071e6
          Log:
          Migrate changelog from Wiki, add JENKINS-31801 reference in "Coming Soon"

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: CHANGELOG.md http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/eb74255621c82d5df3b1caf20c0e6f3f7e0071e6 Log: Migrate changelog from Wiki, add JENKINS-31801 reference in "Coming Soon"

          Oleg Nenashev added a comment -

          Oleg Nenashev added a comment - The feature has been released in the TCB plugin 2.0. https://github.com/jenkinsci/throttle-concurrent-builds-plugin/blob/master/CHANGELOG.md#20

            abayer Andrew Bayer
            daveml Dave Lawrence
            Votes:
            35 Vote for this issue
            Watchers:
            52 Start watching this issue

              Created:
              Updated:
              Resolved: