• Icon: Improvement Improvement
    • Resolution: Fixed
    • Icon: Major Major
    • other
    • None
    • Platform: All, OS: All

          [JENKINS-2072] Make the Queue implementation replaceable

          mdonohue added a comment -

          These enhancement requests depend on a replaceable queue, since they are not
          necessarily appropriate for all users.

          mdonohue added a comment - These enhancement requests depend on a replaceable queue, since they are not necessarily appropriate for all users.

          Ringo De Smet added a comment -

          Wrote this comment since I couldn't find another way to only add myself in CC

          Ringo De Smet added a comment - Wrote this comment since I couldn't find another way to only add myself in CC

          sunfire added a comment -

          Has anyone looked at this "codewise"? Any ideas of complexity and so on?
          I might be able to commit some time to get some of the work done since we're
          having "problems" with Maven dependency builds that occur out-of-order, which
          results in unnecessary builds.

          sunfire added a comment - Has anyone looked at this "codewise"? Any ideas of complexity and so on? I might be able to commit some time to get some of the work done since we're having "problems" with Maven dependency builds that occur out-of-order, which results in unnecessary builds.

          duality72 added a comment -

          cc

          duality72 added a comment - cc

          fireresq added a comment -

          added CC

          fireresq added a comment - added CC

          danrubel added a comment -

          What if the queue ordering was separated from the algorithm(s) that determined the ordering?

          More specifically, what if each job had an integer representing its priority where smaller numbers
          represented higher priority and jobs with the same numeric priority were built on first come first serve
          basis. All jobs with a given priority would be built and completed before any jobs with lower priority
          were built.

          Given this internal queue implementation, you could have
          1) a field in the job configuration page allowing you to manually specify the priority
          2) an extension point for zero or more algorithms to adjust multiple job priorities when a single job's
          configuration is saved
          3) a facility for a user to manually and temporarily "bump up" a job's priority for a single build

          Separating queue ordering from the the algorithm(s) that determined the ordering allows you to keep the
          queue implementation internal and straightforward, while allowing others to explore different algorithms
          to suit their individual needs. Algorithms can also be composited, applying first one algorithm then
          another to achieve the desired effect.

          danrubel added a comment - What if the queue ordering was separated from the algorithm(s) that determined the ordering? More specifically, what if each job had an integer representing its priority where smaller numbers represented higher priority and jobs with the same numeric priority were built on first come first serve basis. All jobs with a given priority would be built and completed before any jobs with lower priority were built. Given this internal queue implementation, you could have 1) a field in the job configuration page allowing you to manually specify the priority 2) an extension point for zero or more algorithms to adjust multiple job priorities when a single job's configuration is saved 3) a facility for a user to manually and temporarily "bump up" a job's priority for a single build Separating queue ordering from the the algorithm(s) that determined the ordering allows you to keep the queue implementation internal and straightforward, while allowing others to explore different algorithms to suit their individual needs. Algorithms can also be composited, applying first one algorithm then another to achieve the desired effect.

          mdonohue added a comment -

          Ultimately it would be nice to have an extension point just for determining each
          projects build ordering, without having to muck with the entire queue
          implementation. That's the end goal.

          In the interim, I can see how a real number priority could solve a number of the
          problems, but it is also lacking. Notably, it is not very object oriented, so
          doing things like tracking why a job was assigned a given priority is very
          difficult to extract from a real number.

          Composability would be nice, but I don't think it is realistic to expect to
          compose arbitrary priority systems. For example, I would like to queue jobs
          according to their expected run time - faster jobs go first. This maps to a
          real number easily. So do user supplied priorities. But composing them seems
          awfully complicated. How do I define a function from real to real that does
          something meaningful with the initial priority, but still produces a queue that
          orders things according to the expected run time?

          Similarly for the bump up feature. I'd rather have some sort of proper
          datastructure that allows for a predicate that moves a job to the top, rather
          than taking the maximum current priority and adding 1 to it. With a
          datastructure, the manually promoted job can be highlighted in the queue as
          promoted, for example.

          mdonohue added a comment - Ultimately it would be nice to have an extension point just for determining each projects build ordering, without having to muck with the entire queue implementation. That's the end goal. In the interim, I can see how a real number priority could solve a number of the problems, but it is also lacking. Notably, it is not very object oriented, so doing things like tracking why a job was assigned a given priority is very difficult to extract from a real number. Composability would be nice, but I don't think it is realistic to expect to compose arbitrary priority systems. For example, I would like to queue jobs according to their expected run time - faster jobs go first. This maps to a real number easily. So do user supplied priorities. But composing them seems awfully complicated. How do I define a function from real to real that does something meaningful with the initial priority, but still produces a queue that orders things according to the expected run time? Similarly for the bump up feature. I'd rather have some sort of proper datastructure that allows for a predicate that moves a job to the top, rather than taking the maximum current priority and adding 1 to it. With a datastructure, the manually promoted job can be highlighted in the queue as promoted, for example.

          danrubel added a comment -

          You could have an extension point for providing a java.util.Comparator for
          determining which job should be built first. Those could also be composed in that
          if a comparator could not determine which job should be built first, then it
          passes the responsibility down the chain of command. In fact the first comparator
          in chain of command could check for jobs that have been manually/temporarily
          bumped-up in priority by a developer waiting for a build to happen. Perhaps this
          is more along the lines of what you are thinking?

          danrubel added a comment - You could have an extension point for providing a java.util.Comparator for determining which job should be built first. Those could also be composed in that if a comparator could not determine which job should be built first, then it passes the responsibility down the chain of command. In fact the first comparator in chain of command could check for jobs that have been manually/temporarily bumped-up in priority by a developer waiting for a build to happen. Perhaps this is more along the lines of what you are thinking?

          At this point enough aspects of a queue is pluggable that I consider this issue fixed.

          If people need additional aspects pluggable, please open separate issues.

          Kohsuke Kawaguchi added a comment - At this point enough aspects of a queue is pluggable that I consider this issue fixed. If people need additional aspects pluggable, please open separate issues.

            kohsuke Kohsuke Kawaguchi
            mdonohue mdonohue
            Votes:
            17 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: