Lately we have been using several different Jenkins plugins to control the execution of a diverse set of jobs, some of which may not be run in parallel to others, one of which being the throttle builds plugin. One limitation of the throttling plugin is that it applies to the entire job, making it impossible to throttle a subset of the operations performed with a larger, more complex job.

For example, lets suppose you have a job that builds some code, runs some operations on the build, runs some unit tests against it, generates a package, and publishes the results somewhere. Further, lets suppose that some of the unit tests that job runs can not be run in parallel with other unit tests on the system. It would be nice to be able to have a build launch and run some of the initial processes in parallel to other processes on the system, but then when it comes time to execute the unit tests to then apply the throttling logic as it would normally be applied, releasing the lock once the tests are complete.

I probably should point out that this behavior is provided exactly as described by the "exclusion" plugin, which we are currently using. However, the problem is that sometimes we'd like entire jobs to be excluded from a build until a locked resource is available while other times we'd like only part of a job to be blocked when a locked resource is in use. Currently we are using both the throttle build and exclusion plugin together to achieve this result, however this strategy breaks down once you want to manage a single resource with both plugins. Using my example above, sometimes we have "monolithic" jobs which run unit tests as one part of a larger operation while other times we have individual jobs responsible for running unit tests independently from any other job. In the former case we don't want the monolithic job to be completely blocked while the independent unit test job is running so the exclusion plugin seems to apply. However, when multiple copies of the standalone test job are triggered we don't want them all consuming executors and just sitting there waiting for a resource lock since they will be total idle and thus wasting valuable build box resources. Thus the throttle build plugin would seem to apply.

I don't believe it is possible to combine these two independent plugins to achieve these intended results. At best we'd need to create two independent configurations for managing the same resource, one in each plugin... and even then the results wouldn't quite be as desired. For example, we would not be able to use the throttle build plugin on our monolithic builds because we need the build to launch and start executing the safe operations in parallel with other work and only block when it needs to run tests. However, if the monolithic tests are running when one of the standalone test jobs trigger it will be launched and will consume an executor on the target agent while it sits idle waiting for the monolithic build to complete it's tests.

What I'd like to see here is some additional options added to the throttle build plugin which would allow custom build steps to be inserted as build operations within a job demarking the start and end points of the area to be throttled - much the same way the exclusion plugin works. However, these custom build steps would then allow the user to select from a list of available categories defined within the global Jenkins config so as to define the category to be 'locked' for the duration of the operation. In this way, jobs that want to be blocked in queue until the lock is available could use the features currently available in the throttling build plugin, while those jobs that want to only block a subset of their operations could use the latter features... and both would make use of the same categorization options effectively eliminating the duplication caused when using multiple plugins.