Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44085

have a simple way to limit the number of parallel branches that run concurrently

XMLWordPrintable

      If you have a large number of branches for a parallel step you may want to be nice and just run a maximum of 10 branches at any one time.

      For a single pipeline project you could use the lockable-resource-plugin to achieve this but for a multi-branch you need to create a lock for each PR in advance. (You would want all PRs / SCM branch jobs) to be able to run 10 of the branches in parallel not just 10 across all of the the jobs.

      You could also use the throttle-concurrent-builds but that only works with nodes and has the same limitations - that the categories need to be set up ahead of time.

      Even if the categories/resources would not need to be setup ahead of time (e.g. enhance the creation of the resource to let it have a capacity of N) you would end up polluting the global configuration for something that is inherently Run based.

      Therefore it would be desirable to implement an extra option to the parallel step to limit that maximum number of branches that would be run at one time.

      e.g. given the following code only 10 branches would run at once.

      def branches [:]
for (int i=0;i<1000;i++) {
  def thing = "$i"
  map[thing] = {
    echo "$thing"
  }
}
branches[maxConcurrent] = 10
parallel branches

      the current workaround would be to use a BlockingDeque and manually adding in / removing values when starting and exiting the parallel block, or by using the waitUntil and something like an AtomicInteger

        1. image-2021-01-26-16-09-24-657.png
          image-2021-01-26-16-09-24-657.png
          61 kB
        2. image-2021-01-26-15-53-37-872.png
          image-2021-01-26-15-53-37-872.png
          22 kB
        3. jenkins-parallel-waits.jpg
          jenkins-parallel-waits.jpg
          64 kB
        4. image-2017-10-31-07-51-15-375.png
          image-2017-10-31-07-51-15-375.png
          240 kB

          [JENKINS-44085] have a simple way to limit the number of parallel branches that run concurrently

          James Nord created issue -

          James Nord added a comment -

          potential workaround:

          def branches = [:]

// setup a latch
MAX_CONCURRENT = 10
latch = new java.util.concurrent.LinkedBlockingDeque(MAX_CONCURRENT)
// put a number of items into the queue to allow that number of branches to run
for (int i=0;i<MAX_CONCURRENT;i++) {
    latch.offer("$i")
}

for (int i=0; i < 500; i++) {
    def name = "$i"
    branches[name] = {
        def thing = null
        // this will not allow proceeding until there is something in the queue.
        waitUntil {
            thing = latch.pollFirst();
            return thing != null;
        }
        try {
            echo "Hello from $name"
            sleep time: 5, unit: 'SECONDS'
            echo "Goodbye from $name"
        }
        finally {
           // put something back into the queue to allow others to proceed
            latch.offer(thing)
        }
    }
}

timestamps {
    parallel branches
}

          James Nord added a comment - potential workaround: def branches = [:] // setup a latch MAX_CONCURRENT = 10 latch = new java.util.concurrent.LinkedBlockingDeque(MAX_CONCURRENT) // put a number of items into the queue to allow that number of branches to run for ( int i=0;i<MAX_CONCURRENT;i++) { latch.offer( "$i" ) } for ( int i=0; i < 500; i++) { def name = "$i" branches[name] = { def thing = null // this will not allow proceeding until there is something in the queue. waitUntil { thing = latch.pollFirst(); return thing != null ; } try { echo "Hello from $name" sleep time: 5, unit: 'SECONDS' echo "Goodbye from $name" } finally { // put something back into the queue to allow others to proceed latch.offer(thing) } } } timestamps { parallel branches }
          Jesse Glick made changes -
          Labels Original: pipeline
          Jesse Glick made changes -
          Component/s New: workflow-cps-plugin [ 21713 ]
          Component/s Original: pipeline [ 21692 ]
          Jesse Glick made changes -
          Link New: This issue relates to JENKINS-26125 [ JENKINS-26125 ]
          Andrew Bayer made changes -
          Link New: This issue is duplicated by JENKINS-46236 [ JENKINS-46236 ]

          Mor L added a comment - - edited

          I think the suggestion to have it as a parameter to the parallel step as suggested in https://issues.jenkins-ci.org/browse/JENKINS-46236 is better than defining it in the branches/jobs map. would love to see it implemented - and thanks for the workaround.

          Mor L added a comment - - edited I think the suggestion to have it as a parameter to the parallel step as suggested in https://issues.jenkins-ci.org/browse/JENKINS-46236  is better than defining it in the branches/jobs map. would love to see it implemented - and thanks for the workaround.
          Tom Larrow made changes -
          Attachment New: image-2017-10-31-07-51-15-375.png [ 40174 ]

          Tom Larrow added a comment -

          Agreed, some simple syntax to limit the parallelization would be great, because when you start dynamically building parallel steps, it is easy to get out of control...

           

          Tom Larrow added a comment - Agreed, some simple syntax to limit the parallelization would be great, because when you start dynamically building parallel steps, it is easy to get out of control...  
          Jared Kauppila made changes -
          Assignee New: Jared Kauppila [ jakauppila ]

            Unassigned Unassigned
            teilo James Nord
            Votes:
            63 Vote for this issue
            Watchers:
            65 Start watching this issue

              Created:
              Updated: