• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical

      Execution of parallel blocks scales poorly for values of N > 100.  With ~50 nodes (each with 4 executors, for a total of ~200 slots), the following pipeline job takes extraordinarily long to execute:

       

      def stepsForParallel = [:]
      for (int i = 0; i < Integer.valueOf(params.SUB_JOBS); i++) {
        def s = "subjob_${i}" 
        stepsForParallel[s] = {
          node("darwin") {
            echo "hello"
          }
        }
      }
      parallel stepsForParallel
      

       

      SUB_JOBS   Time (sec)
      ---------------------
       100         10
       200         40
       300         96
       400        214
       500        392
       600        660
       700        960
       800       1500
       900       2220
      1000       gave up...

      At no point does the underlying system become taxed (CPU utilization is very low, as this is a very beefy system – 28 cores, 128GB RAM, SSDs)

      CPU and Thread CPU Time Sampling (via VisualVM) are attached for reference.

       

       

       

       

       

          [JENKINS-45553] Parallel pipeline execution scales poorly

          Tom Skrainar created issue -
          Jesse Glick made changes -
          Labels Original: Pipeline performance New: performance
          Tom Skrainar made changes -
          Attachment New: JENKINS-45553_20170725.tgz [ 39026 ]
          Tom Skrainar made changes -
          Attachment New: JENKINS-45553_20170725.tgz [ 39027 ]
          Tom Skrainar made changes -
          Attachment Original: JENKINS-45553_20170725.tgz [ 39026 ]
          Jesse Glick made changes -
          Component/s New: core [ 15593 ]
          Component/s New: workflow-durable-task-step-plugin [ 21715 ]
          Component/s New: workflow-support-plugin [ 21719 ]
          Component/s Original: pipeline [ 21692 ]
          Labels Original: performance New: performance pipeline
          Jesse Glick made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]
          Jesse Glick made changes -
          Assignee New: Jesse Glick [ jglick ]
          Jesse Glick made changes -
          Link New: This issue relates to JENKINS-38223 [ JENKINS-38223 ]
          Jesse Glick made changes -
          Link New: This issue relates to JENKINS-34542 [ JENKINS-34542 ]
          Jesse Glick made changes -
          Remote Link New: This issue links to "workflow-support PR 38 (Web Link)" [ 17350 ]

            jglick Jesse Glick
            tskrainar Tom Skrainar
            Votes:
            4 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated:
              Resolved: