Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46236

Throttle parallel step in pipeline script

XMLWordPrintable

    • Icon: Improvement Improvement
    • Resolution: Duplicate
    • Icon: Major Major
    • workflow-cps-plugin
    • None

      It appears that Jenkins pipelines have no way of throttling parallel closures execution.

      Given the example:

      stage('Test') {
  steps {
    script {
      testing_closures = [one: { print("starting one"); sleep 10; print("finishing one") },
                          two: { print("starting two"); sleep 10; print("finishing two") },
                          three: { print("starting three"); sleep 10; print("finishing three") },
                          four: { print("starting four"); sleep 10; print("finishing four") },
                          five: { print("starting five"); sleep 10; print("finishing five") },
                          six: { print("starting six"); sleep 10; print("finishing six") }]
      parallel(testing_closures)
    }
  }
}

      The main goal is to throttle those closures. If we don't want for all six of them to run concurrently, but let's say only 3 at a time, we'd need an argument for 'parallel' step:
      parallel(closures: testing_closures, maxThreadCount: 3)

      Currently, as far as I know, there are no workarounds for this issue. If JENKINS-46235 is resolved, the following workaround could be used:

      1. Dynamically create 3 lockable resources via LockableResourcesManager::createResourceWithLabel() with build-unique labels
      2. Lock them by label in all of the closures
      3. The closures will wait for each other to finish and only 3 at the time would be running.
      4. Delete the closures (no such API method. JENKINS-46235 is for exposing 'delete' method for lockable resource)

          [JENKINS-46236] Throttle parallel step in pipeline script

          Sorin Sbarnea added a comment -

          Thanks, that describes exactly what I was looking for. Maybe the only thing I would like to see is using "concurrency" instead of "maxThreadCount" as the name of the parameter.

          This would be in sync with other similar places like https://www.cloudbees.com/blog/parallelism-and-distributed-builds-jenkins

          Sorin Sbarnea added a comment - Thanks, that describes exactly what I was looking for. Maybe the only thing I would like to see is using "concurrency" instead of "maxThreadCount" as the name of the parameter. This would be in sync with other similar places like https://www.cloudbees.com/blog/parallelism-and-distributed-builds-jenkins

          Tom Larrow added a comment -

          This would come in very helpful as we refactor how much of our testing works, and split selenium tests to spawn in parallel.  For many of our applications, this would spawn 3-5 threads, but for others it would spawn 75-100.  The first is workable, the second is not.  Having a way to limit the number of concurrent threads would allow us to build this functionality.

          Tom Larrow added a comment - This would come in very helpful as we refactor how much of our testing works, and split selenium tests to spawn in parallel.  For many of our applications, this would spawn 3-5 threads, but for others it would spawn 75-100.  The first is workable, the second is not.  Having a way to limit the number of concurrent threads would allow us to build this functionality.

            Unassigned Unassigned
            mireksz Mirek Sz
            Votes:
            2 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: