Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45553

Parallel pipeline execution scales poorly

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical

    Description

      Execution of parallel blocks scales poorly for values of N > 100.  With ~50 nodes (each with 4 executors, for a total of ~200 slots), the following pipeline job takes extraordinarily long to execute:

       

      def stepsForParallel = [:]
      for (int i = 0; i < Integer.valueOf(params.SUB_JOBS); i++) {
        def s = "subjob_${i}" 
        stepsForParallel[s] = {
          node("darwin") {
            echo "hello"
          }
        }
      }
      parallel stepsForParallel
      

       

      SUB_JOBS   Time (sec)
      ---------------------
       100         10
       200         40
       300         96
       400        214
       500        392
       600        660
       700        960
       800       1500
       900       2220
      1000       gave up...

      At no point does the underlying system become taxed (CPU utilization is very low, as this is a very beefy system – 28 cores, 128GB RAM, SSDs)

      CPU and Thread CPU Time Sampling (via VisualVM) are attached for reference.

       

       

       

       

       

      Attachments

        Issue Links

          Activity

            People

              jglick Jesse Glick
              tskrainar Tom Skrainar
              Votes:
              4 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: