Throttle parallel step in pipeline script

This issue is archived. You can view it, but you can't modify it. Learn more

XMLWordPrintable

      It appears that Jenkins pipelines have no way of throttling parallel closures execution.

      Given the example:

      stage('Test') {
        steps {
          script {
            testing_closures = [one: { print("starting one"); sleep 10; print("finishing one") },
                                two: { print("starting two"); sleep 10; print("finishing two") },
                                three: { print("starting three"); sleep 10; print("finishing three") },
                                four: { print("starting four"); sleep 10; print("finishing four") },
                                five: { print("starting five"); sleep 10; print("finishing five") },
                                six: { print("starting six"); sleep 10; print("finishing six") }]
            parallel(testing_closures)
          }
        }
      }
      

      The main goal is to throttle those closures. If we don't want for all six of them to run concurrently, but let's say only 3 at a time, we'd need an argument for 'parallel' step:
      parallel(closures: testing_closures, maxThreadCount: 3)

      Currently, as far as I know, there are no workarounds for this issue. If JENKINS-46235 is resolved, the following workaround could be used:

      1. Dynamically create 3 lockable resources via LockableResourcesManager::createResourceWithLabel() with build-unique labels
      2. Lock them by label in all of the closures
      3. The closures will wait for each other to finish and only 3 at the time would be running.
      4. Delete the closures (no such API method. JENKINS-46235 is for exposing 'delete' method for lockable resource)

            Assignee:
            Unassigned
            Reporter:
            Mirek Sz
            Archiver:
            Jenkins Service Account

              Created:
              Updated:
              Resolved:
              Archived: