Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-6586

Support max number of concurrent builds of a given job per node

    • Icon: New Feature New Feature
    • Resolution: Fixed
    • Icon: Major Major
    • core
    • None

      See JENKINS-5083 - description cribbed from there:

      In addition to the original limit reques: total number of concurrent builds (the original request),
      I would like to see an additional limit: total number of concurrent builds per slave.

      This is to handle a specific requirement I have: for some of my jobs, I can run builds concurrently on different machines, but I cannot have builds running concurrently on the same machine.

      At the moment, the way I achieve this is by only having one executor per slave, but that imposes a stronger restriction that I really need. Having a concurrent-per-machine limit would meet my requirements exactly.

      I'm splitting this off into a separate issue specifically for the "max concurrent builds of a job per node" functionality because it can be implemented in a plugin against Hudson as is - but I couldn't find anything near as elegant a way to limit the total number of concurrent builds of a job across all nodes, so I think that'll end up being a change in core.

      Here's how I'm doing the max concurrent builds per node functionality (or as I'm calling it in the plugin, the throttle concurrent builds functionality):

      • We'll assign a label (say, "job_foo_eligible") to the slave(s) in question.
      • We'll add a job property to the job - in it, we'll specify the maximum number of concurrent builds per node and the label we put on the slaves.
      • In addition to the job property, there'll be three additional classes in the plugin:
        • The LabelFinder - it does the actual work, getting the non-dynamic labels assigned to a node and checking to see if any jobs have the throttle concurrent builds job property with any of those labels as the designated label. It then takes that list of jobs and checks each one to see if it's already running X times on the node (where X is the max concurrent builds per node we specified in the job property), and for each job that isn't already running X times, it adds "$ORIGLABEL__not_throttled" to the list of dynamic labels it returns - e.g., with the original label "job_foo_eligible", the label that actually gets returned is "job_foo_eligible__not_throttled".
        • The RunListener, which calls the same private method for onStarted and onFinalized - if the job the run belongs to has the throttle concurrent builds job property, refresh the labels by running Node.getAssignedLabels on each node. This makes sure the labels are up to date when jobs start/finish.
        • The QueueDecisionHandler, which, like the one in the matrixtieparent plugin, takes advantage of the fact that QueueDecisionHandler.shouldSchedule gets called before the actual job scheduling happens - so we're able to sneak in there and run setAssignedLabel on the job to assign it to "job_foo_eligible___not_throttled". Tada!

      I should have a prototype plugin ready in the next few days.

          [JENKINS-6586] Support max number of concurrent builds of a given job per node

          Andrew Bayer added a comment -

          Well, this ended up being kind of annoying - getting the labels to properly reset their internal lists of nodes connected to them was a giant pain. But I've got a rough draft up: http://github.com/hudson/hudson-throttle-concurrent-builds-plugin

          Andrew Bayer added a comment - Well, this ended up being kind of annoying - getting the labels to properly reset their internal lists of nodes connected to them was a giant pain. But I've got a rough draft up: http://github.com/hudson/hudson-throttle-concurrent-builds-plugin

          Andrew Bayer added a comment -

          I've released a plugin to do this - it works with 1.360 of Hudson and later: http://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin

          Andrew Bayer added a comment - I've released a plugin to do this - it works with 1.360 of Hudson and later: http://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin

          Do you know if there is a way to limit the number of nodes a jenkins job would take at one given time?

          Benjamin Leheup added a comment - Do you know if there is a way to limit the number of nodes a jenkins job would take at one given time?

            abayer Andrew Bayer
            abayer Andrew Bayer
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: