See JENKINS-5083 - description cribbed from there:

In addition to the original limit reques: total number of concurrent builds (the original request),
I would like to see an additional limit: total number of concurrent builds per slave.

This is to handle a specific requirement I have: for some of my jobs, I can run builds concurrently on different machines, but I cannot have builds running concurrently on the same machine.

At the moment, the way I achieve this is by only having one executor per slave, but that imposes a stronger restriction that I really need. Having a concurrent-per-machine limit would meet my requirements exactly.

I'm splitting this off into a separate issue specifically for the "max concurrent builds of a job per node" functionality because it can be implemented in a plugin against Hudson as is - but I couldn't find anything near as elegant a way to limit the total number of concurrent builds of a job across all nodes, so I think that'll end up being a change in core.

Here's how I'm doing the max concurrent builds per node functionality (or as I'm calling it in the plugin, the throttle concurrent builds functionality):

• We'll assign a label (say, "job_foo_eligible") to the slave(s) in question.
• We'll add a job property to the job - in it, we'll specify the maximum number of concurrent builds per node and the label we put on the slaves.
• In addition to the job property, there'll be three additional classes in the plugin:
  • The LabelFinder - it does the actual work, getting the non-dynamic labels assigned to a node and checking to see if any jobs have the throttle concurrent builds job property with any of those labels as the designated label. It then takes that list of jobs and checks each one to see if it's already running X times on the node (where X is the max concurrent builds per node we specified in the job property), and for each job that isn't already running X times, it adds "$ORIGLABEL__not_throttled" to the list of dynamic labels it returns - e.g., with the original label "job_foo_eligible", the label that actually gets returned is "job_foo_eligible__not_throttled".
  • The RunListener, which calls the same private method for onStarted and onFinalized - if the job the run belongs to has the throttle concurrent builds job property, refresh the labels by running Node.getAssignedLabels on each node. This makes sure the labels are up to date when jobs start/finish.
  • The QueueDecisionHandler, which, like the one in the matrixtieparent plugin, takes advantage of the fact that QueueDecisionHandler.shouldSchedule gets called before the actual job scheduling happens - so we're able to sneak in there and run setAssignedLabel on the job to assign it to "job_foo_eligible___not_throttled". Tada!

I should have a prototype plugin ready in the next few days.