Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-28107

Matrix parent should take an executor slot until the point where children are triggered

      When lots of matrix jobs are queued because a slave has been unavailable, all of these are started at the same time because these do not take up executor slots.

      As the parent job still accesses the VCS, all of these accesses happen in parallel then and may starve each other for resources.

      It would be great if the job would take an executor slot, and relinquish it when it starts triggering the child jobs.

          [JENKINS-28107] Matrix parent should take an executor slot until the point where children are triggered

          Daniel Beck added a comment -

          Possible workaround: Don't restrict the parent build to a slave?

          Daniel Beck added a comment - Possible workaround: Don't restrict the parent build to a slave?

          Simon Richter added a comment -

          That helps distribute the load a bit, but is still suboptimal (rather than 30 concurrent parent jobs I now have 10).

          Simon Richter added a comment - That helps distribute the load a bit, but is still suboptimal (rather than 30 concurrent parent jobs I now have 10).

          Daniel Beck added a comment -

          That helps distribute the load a bit, but is still suboptimal (rather than 30 concurrent parent jobs I now have 10).

          Isn't that just due to you currently already experiencing this load? If this would have been the config to begin with, they'd have started at different times (unless all have the same trigger, and then the quiet period may be a workaround).

          Daniel Beck added a comment - That helps distribute the load a bit, but is still suboptimal (rather than 30 concurrent parent jobs I now have 10). Isn't that just due to you currently already experiencing this load? If this would have been the config to begin with, they'd have started at different times (unless all have the same trigger, and then the quiet period may be a workaround).

          Simon Richter added a comment -

          Indeed, that's why I prioritized it as "Minor" – it is possible to work around the problem, but I'd rather see that the system avoids getting into a situation where more processes run on a single system than I have executors configured for it.

          Simon Richter added a comment - Indeed, that's why I prioritized it as "Minor" – it is possible to work around the problem, but I'd rather see that the system avoids getting into a situation where more processes run on a single system than I have executors configured for it.

          Daniel Beck added a comment -

          I cannot think of a good solution to this issue. Moving between executors, or changing a job from "heavy-weight" to fly-weight isn't really possible AFAIK.

          The best approach I can think of right now is to limit the number of matrix jobs allowed to run in parallel using something like Throttle Concurrent Builds, but that's of course a different limitation (you don't care about how many matrix parents are running after they've finished SCM operations).

          Daniel Beck added a comment - I cannot think of a good solution to this issue. Moving between executors, or changing a job from "heavy-weight" to fly-weight isn't really possible AFAIK. The best approach I can think of right now is to limit the number of matrix jobs allowed to run in parallel using something like Throttle Concurrent Builds, but that's of course a different limitation (you don't care about how many matrix parents are running after they've finished SCM operations).

            kohsuke Kohsuke Kawaguchi
            simonrichter Simon Richter
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: