Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44085

have a simple way to limit the number of parallel branches that run concurrently

XMLWordPrintable

      If you have a large number of branches for a parallel step you may want to be nice and just run a maximum of 10 branches at any one time.

      For a single pipeline project you could use the lockable-resource-plugin to achieve this but for a multi-branch you need to create a lock for each PR in advance. (You would want all PRs / SCM branch jobs) to be able to run 10 of the branches in parallel not just 10 across all of the the jobs.

      You could also use the throttle-concurrent-builds but that only works with nodes and has the same limitations - that the categories need to be set up ahead of time.

      Even if the categories/resources would not need to be setup ahead of time (e.g. enhance the creation of the resource to let it have a capacity of N) you would end up polluting the global configuration for something that is inherently Run based.

      Therefore it would be desirable to implement an extra option to the parallel step to limit that maximum number of branches that would be run at one time.

      e.g. given the following code only 10 branches would run at once.

      def branches [:]
      for (int i=0;i<1000;i++) {
        def thing = "$i"
        map[thing] = {
          echo "$thing"
        }
      }
      branches[maxConcurrent] = 10
      parallel branches
      

      the current workaround would be to use a BlockingDeque and manually adding in / removing values when starting and exiting the parallel block, or by using the waitUntil and something like an AtomicInteger

            Unassigned Unassigned
            teilo James Nord
            Votes:
            63 Vote for this issue
            Watchers:
            65 Start watching this issue

              Created:
              Updated: