[JENKINS-45553] Parallel pipeline execution scales poorly - Jenkins Jira

XML

Word

Printable

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Critical
Resolution: Fixed
Component/s: core, workflow-durable-task-step-plugin, workflow-support-plugin
Labels:
- performance
- pipeline
Environment:

Hide
Jenkins 2.69
Pipeline 2.5

Ubuntu 16.04
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)
JVM args: -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+ParallelRefProcEnabled -XX:+UseStringDeduplication -XX:+UnlockDiagnosticVMOptions -XX:G1SummarizeRSetStatsPeriod=1 -server -XX:+AlwaysPreTouch -Djenkins.install.runSetupWizard=false -Dgroovy.use.classvalue=true -Xmx8192m -Xms8192m

Show
Jenkins 2.69 Pipeline 2.5 Ubuntu 16.04 OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode) JVM args: -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+ParallelRefProcEnabled -XX:+UseStringDeduplication -XX:+UnlockDiagnosticVMOptions -XX:G1SummarizeRSetStatsPeriod=1 -server -XX:+AlwaysPreTouch -Djenkins.install.runSetupWizard=false -Dgroovy.use.classvalue=true -Xmx8192m -Xms8192m

Similar Issues:

Show

Description

Execution of parallel blocks scales poorly for values of N > 100. With ~50 nodes (each with 4 executors, for a total of ~200 slots), the following pipeline job takes extraordinarily long to execute:

def stepsForParallel = [:]
for (int i = 0; i < Integer.valueOf(params.SUB_JOBS); i++) {
  def s = "subjob_${i}" 
  stepsForParallel[s] = {
    node("darwin") {
      echo "hello"
    }
  }
}
parallel stepsForParallel

SUB_JOBS   Time (sec)
---------------------
 100         10
 200         40
 300         96
 400        214
 500        392
 600        660
 700        960
 800       1500
 900       2220
1000       gave up...

At no point does the underlying system become taxed (CPU utilization is very low, as this is a very beefy system – 28 cores, 128GB RAM, SSDs)

CPU and Thread CPU Time Sampling (via VisualVM) are attached for reference.

Attachments

Options
- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

Attachments

JENKINS-45553_20170725.tgz
17 kB
2017-07-25 23:17
VisualVM_CPU_Sampling.png
69 kB
2017-07-14 22:58
VisualVM_Thread_CPU_Time_Sampling_After_Run.png
105 kB
2017-07-14 23:05
VisualVM_Thread_CPU_Time_Sampling.png
84 kB
2017-07-14 22:58

Issue Links

depends on

JENKINS-38381 [JEP-210] Optimize log handling in Pipeline and Durable Task

Resolved

JENKINS-36547 Queue.Task.getFullDisplayName is a poor choice of key for LoadBalancer.CONSISTENT_HASH

Resolved

is duplicated by

JENKINS-45876 Jenkins becomes extremely slow while running a lot of tests parallel

Resolved

relates to

JENKINS-34542 Hang in ExecutorStepExecution

Resolved

JENKINS-42556 PlaceholderTask.runForDisplay vulnerable to AccessDeniedException

Resolved

JENKINS-38223 FlowNode.isRunning is not very useful

Closed

JENKINS-26132 Executor should show the current stage the flow run is in

Resolved

JENKINS-40934 LogActionImpl listener inefficient; poor performance queuing large parallel workloads

Resolved

links to

core PR 2947

core PR 2948

workflow-durable-task-step PR 44

workflow-durable-task-step PR 45

workflow-durable-task-step PR 46

workflow-support PR 38

(3 relates to, 6 links to)

Activity

Ascending order - Click to sort in descending order

Tom Skrainar created issue - 2017-07-14 23:07

Jesse Glick made changes - 2017-07-25 18:11

Field	Original Value	New Value
Labels	Pipeline performance	performance

Tom Skrainar made changes - 2017-07-25 23:15

Attachment

JENKINS-45553_20170725.tgz [ 39026 ]

Tom Skrainar made changes - 2017-07-25 23:17

Attachment

JENKINS-45553_20170725.tgz [ 39027 ]

Tom Skrainar made changes - 2017-07-25 23:20

Attachment

~~JENKINS-45553~~_20170725.tgz [ 39026 ]

Jesse Glick made changes - 2017-07-27 21:41

Component/s		core [ 15593 ]
Component/s		workflow-durable-task-step-plugin [ 21715 ]
Component/s		workflow-support-plugin [ 21719 ]
Component/s	pipeline [ 21692 ]
Labels	performance	performance pipeline

Jesse Glick made changes - 2017-07-27 21:46

Status

Open [ 1 ]

In Progress [ 3 ]

Jesse Glick made changes - 2017-07-27 21:46

Assignee

Jesse Glick [ jglick ]

Jesse Glick made changes - 2017-07-27 22:03

Link

This issue relates to ~~JENKINS-38223~~ [ ~~JENKINS-38223~~ ]

Jesse Glick made changes - 2017-07-27 22:03

Link

This issue relates to ~~JENKINS-34542~~ [ ~~JENKINS-34542~~ ]

Jesse Glick made changes - 2017-07-27 22:05

Remote Link

This issue links to "workflow-support PR 38 (Web Link)" [ 17350 ]

Jesse Glick made changes - 2017-07-27 22:09

Remote Link

This issue links to "workflow-durable-task-step PR 44 (Web Link)" [ 17351 ]

Jesse Glick made changes - 2017-07-27 22:14

Remote Link

This issue links to "workflow-durable-task-step PR 45 (Web Link)" [ 17352 ]

Jesse Glick made changes - 2017-07-27 22:15

Link

This issue relates to ~~JENKINS-26132~~ [ ~~JENKINS-26132~~ ]

Jesse Glick made changes - 2017-07-27 22:16

Labels

performance pipeline

lts-candidate performance pipeline

Jesse Glick made changes - 2017-07-27 22:18

Link

This issue relates to ~~JENKINS-42556~~ [ ~~JENKINS-42556~~ ]

Jesse Glick made changes - 2017-07-27 22:19

Remote Link

This issue links to "core PR 2947 (Web Link)" [ 17353 ]

Jesse Glick made changes - 2017-07-27 22:26

Remote Link

This issue links to "core PR 2948 (Web Link)" [ 17354 ]

Jesse Glick made changes - 2017-07-27 22:49

Remote Link

This issue links to "workflow-durable-task-step PR 46 (Web Link)" [ 17355 ]

Jesse Glick made changes - 2017-07-27 22:50

Status

In Progress [ 3 ]

In Review [ 10005 ]

Jesse Glick made changes - 2017-07-27 23:23

Link

This issue relates to ~~JENKINS-40934~~ [ ~~JENKINS-40934~~ ]

Andrew Bayer made changes - 2017-08-01 02:19

Link

This issue is duplicated by ~~JENKINS-45876~~ [ ~~JENKINS-45876~~ ]

Jesse Glick made changes - 2017-08-03 19:16

Link

This issue depends on ~~JENKINS-36547~~ [ ~~JENKINS-36547~~ ]

Jesse Glick made changes - 2017-08-03 19:19

Link

This issue depends on ~~JENKINS-38381~~ [ ~~JENKINS-38381~~ ]

Jesse Glick made changes - 2017-08-26 22:19

Resolution		Fixed [ 1 ]
Status	In Review [ 10005 ]	Resolved [ 5 ]

Oliver Gondža made changes - 2017-09-01 13:29

Labels

lts-candidate performance pipeline

performance pipeline

Sam Van Oort made changes - 2017-09-26 22:23

Status

Resolved [ 5 ]

Closed [ 6 ]

People

Assignee:: Jesse Glick

Reporter:: Tom Skrainar

Votes:: 4 Vote for this issue

Watchers:: 13 Start watching this issue

Dates

Created:: 2017-07-14 23:07

Updated:: 2018-02-07 16:50

Resolved:: 2017-08-26 22:19