Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45553

Parallel pipeline execution scales poorly

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Environment:
    • Similar Issues:

      Description

      Execution of parallel blocks scales poorly for values of N > 100.  With ~50 nodes (each with 4 executors, for a total of ~200 slots), the following pipeline job takes extraordinarily long to execute:

       

      def stepsForParallel = [:]
      for (int i = 0; i < Integer.valueOf(params.SUB_JOBS); i++) {
        def s = "subjob_${i}" 
        stepsForParallel[s] = {
          node("darwin") {
            echo "hello"
          }
        }
      }
      parallel stepsForParallel
      

       

      SUB_JOBS   Time (sec)
      ---------------------
       100         10
       200         40
       300         96
       400        214
       500        392
       600        660
       700        960
       800       1500
       900       2220
      1000       gave up...

      At no point does the underlying system become taxed (CPU utilization is very low, as this is a very beefy system – 28 cores, 128GB RAM, SSDs)

      CPU and Thread CPU Time Sampling (via VisualVM) are attached for reference.

       

       

       

       

       

        Attachments

          Issue Links

            Activity

            tskrainar Tom Skrainar created issue -
            jglick Jesse Glick made changes -
            Field Original Value New Value
            Labels Pipeline performance performance
            tskrainar Tom Skrainar made changes -
            Attachment JENKINS-45553_20170725.tgz [ 39026 ]
            tskrainar Tom Skrainar made changes -
            Attachment JENKINS-45553_20170725.tgz [ 39027 ]
            tskrainar Tom Skrainar made changes -
            Attachment JENKINS-45553_20170725.tgz [ 39026 ]
            jglick Jesse Glick made changes -
            Component/s core [ 15593 ]
            Component/s workflow-durable-task-step-plugin [ 21715 ]
            Component/s workflow-support-plugin [ 21719 ]
            Component/s pipeline [ 21692 ]
            Labels performance performance pipeline
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Assignee Jesse Glick [ jglick ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-38223 [ JENKINS-38223 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-34542 [ JENKINS-34542 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "workflow-support PR 38 (Web Link)" [ 17350 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "workflow-durable-task-step PR 44 (Web Link)" [ 17351 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "workflow-durable-task-step PR 45 (Web Link)" [ 17352 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-26132 [ JENKINS-26132 ]
            jglick Jesse Glick made changes -
            Labels performance pipeline lts-candidate performance pipeline
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-42556 [ JENKINS-42556 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "core PR 2947 (Web Link)" [ 17353 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "core PR 2948 (Web Link)" [ 17354 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "workflow-durable-task-step PR 46 (Web Link)" [ 17355 ]
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-40934 [ JENKINS-40934 ]
            abayer Andrew Bayer made changes -
            Link This issue is duplicated by JENKINS-45876 [ JENKINS-45876 ]
            jglick Jesse Glick made changes -
            Link This issue depends on JENKINS-36547 [ JENKINS-36547 ]
            jglick Jesse Glick made changes -
            Link This issue depends on JENKINS-38381 [ JENKINS-38381 ]
            jglick Jesse Glick made changes -
            Resolution Fixed [ 1 ]
            Status In Review [ 10005 ] Resolved [ 5 ]
            olivergondza Oliver Gondža made changes -
            Labels lts-candidate performance pipeline performance pipeline
            svanoort Sam Van Oort made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

              People

              Assignee:
              jglick Jesse Glick
              Reporter:
              tskrainar Tom Skrainar
              Votes:
              4 Vote for this issue
              Watchers:
              13 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: