Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-28107

Matrix parent should take an executor slot until the point where children are triggered

    XMLWordPrintable

Details

    Description

      When lots of matrix jobs are queued because a slave has been unavailable, all of these are started at the same time because these do not take up executor slots.

      As the parent job still accesses the VCS, all of these accesses happen in parallel then and may starve each other for resources.

      It would be great if the job would take an executor slot, and relinquish it when it starts triggering the child jobs.

      Attachments

        Activity

          danielbeck Daniel Beck added a comment -

          Possible workaround: Don't restrict the parent build to a slave?

          danielbeck Daniel Beck added a comment - Possible workaround: Don't restrict the parent build to a slave?
          simonrichter Simon Richter added a comment -

          That helps distribute the load a bit, but is still suboptimal (rather than 30 concurrent parent jobs I now have 10).

          simonrichter Simon Richter added a comment - That helps distribute the load a bit, but is still suboptimal (rather than 30 concurrent parent jobs I now have 10).
          danielbeck Daniel Beck added a comment -

          That helps distribute the load a bit, but is still suboptimal (rather than 30 concurrent parent jobs I now have 10).

          Isn't that just due to you currently already experiencing this load? If this would have been the config to begin with, they'd have started at different times (unless all have the same trigger, and then the quiet period may be a workaround).

          danielbeck Daniel Beck added a comment - That helps distribute the load a bit, but is still suboptimal (rather than 30 concurrent parent jobs I now have 10). Isn't that just due to you currently already experiencing this load? If this would have been the config to begin with, they'd have started at different times (unless all have the same trigger, and then the quiet period may be a workaround).
          simonrichter Simon Richter added a comment -

          Indeed, that's why I prioritized it as "Minor" – it is possible to work around the problem, but I'd rather see that the system avoids getting into a situation where more processes run on a single system than I have executors configured for it.

          simonrichter Simon Richter added a comment - Indeed, that's why I prioritized it as "Minor" – it is possible to work around the problem, but I'd rather see that the system avoids getting into a situation where more processes run on a single system than I have executors configured for it.
          danielbeck Daniel Beck added a comment -

          I cannot think of a good solution to this issue. Moving between executors, or changing a job from "heavy-weight" to fly-weight isn't really possible AFAIK.

          The best approach I can think of right now is to limit the number of matrix jobs allowed to run in parallel using something like Throttle Concurrent Builds, but that's of course a different limitation (you don't care about how many matrix parents are running after they've finished SCM operations).

          danielbeck Daniel Beck added a comment - I cannot think of a good solution to this issue. Moving between executors, or changing a job from "heavy-weight" to fly-weight isn't really possible AFAIK. The best approach I can think of right now is to limit the number of matrix jobs allowed to run in parallel using something like Throttle Concurrent Builds, but that's of course a different limitation (you don't care about how many matrix parents are running after they've finished SCM operations).

          People

            kohsuke Kohsuke Kawaguchi
            simonrichter Simon Richter
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: