[JENKINS-30055] Simple script with "long" loop created unloadable flow

Type: Bug
Resolution: Fixed
Priority: Major
Component/s: pipeline
Labels:
- performance
- robustness

Similar Issues:
Powered by SuggestiMate

Show

Create a workflow with a simple script containing a loop with e.g. 1000 iterations:

node {
  	stage 'Build'
	for (int i = 0; i < 1000; i++) {
		echo "hello: ${i}"
	}
}

Run the job a few times. Restart jenkins and try loading the job, you should see an error like:

*** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message transform method call failed at ../../../src/share/instrument/JPLISAgent.c line: 844
Aug 20, 2015 1:09:35 PM org.jenkinsci.plugins.workflow.job.WorkflowRun onLoad
WARNING: null
java.io.IOException: failed to load flow node from /Users/tfennelly/projects/jenkins-plugins/cloudbees-workflow-plugin/aggregator/work/jobs/WWW/builds/5/workflow/1008.xml
	at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage$PersistenceContext.loadInner(SimpleXStreamFlowNodeStorage.java:227)
	at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage$PersistenceContext.loadOuter(SimpleXStreamFlowNodeStorage.java:213)
	at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage$PersistenceContext.access$100(SimpleXStreamFlowNodeStorage.java:185)
	at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.getNode(SimpleXStreamFlowNodeStorage.java:87)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:397)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:413)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:380)
	at hudson.model.RunMap.retrieve(RunMap.java:223)
	at hudson.model.RunMap.retrieve(RunMap.java:57)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:465)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:448)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:356)
	at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:226)
	at org.jenkinsci.plugins.workflow.job.WorkflowJob.getBuildByNumber(WorkflowJob.java:199)
	at org.jenkinsci.plugins.workflow.job.WorkflowJob.getBuildByNumber(WorkflowJob.java:92)
	at jenkins.model.PeepholePermalink.resolve(PeepholePermalink.java:95)
	at hudson.model.Job.getLastSuccessfulBuild(Job.java:888)
	at jenkins.tasks.SimpleBuildStep$LastBuildActionFactory.createFor(SimpleBuildStep.java:111)
	at jenkins.tasks.SimpleBuildStep$LastBuildActionFactory.createFor(SimpleBuildStep.java:101)
	at hudson.model.Actionable.createFor(Actionable.java:107)
	at hudson.model.Actionable.getAllActions(Actionable.java:98)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
	at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
	at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
	at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
	at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
	at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
	at org.apache.commons.jexl.parser.ASTElvisNode.value(ASTElvisNode.java:37)
	at org.apache.commons.jexl.parser.ASTExpression.value(ASTExpression.java:54)
	at org.apache.commons.jexl.parser.ASTElvisNode.value(ASTElvisNode.java:40)
	at org.apache.commons.jexl.parser.ASTExpression.value(ASTExpression.java:54)
	at org.apache.commons.jexl.parser.ASTExpressionExpression.value(ASTExpressionExpression.java:56)
	at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)

depends on

JENKINS-27057 CME in WorkflowRun.logsToCopy

Resolved

is related to

JENKINS-31360 Permgen leak with SimpleXStreamFlowNodeStorage

Resolved

JENKINS-30651 File descriptor leak when invoking for-loop

Resolved

JENKINS-30896 Unoptimized node storage

Closed

links to

PR 213

Jesse Glick added a comment - 2015-08-21 15:29

Smells like a race condition: perhaps a reference to a new flow node is saved before the node itself is.

Jesse Glick added a comment - 2015-08-21 15:29 Smells like a race condition: perhaps a reference to a new flow node is saved before the node itself is.

Jesse Glick added a comment - 2015-09-25 22:00

Rather a file descriptor leak; the listener in DefaultStepContext gets notified asynchronously, so not fast enough to close the log files from completed steps.

Also a failure to ever remove the listener results in a quadratic slowdown. With that fixed, the insertion of

sleep unit: 'MILLISECONDS', time: 1

inside the loop lets it work, but without that extra pause the notifications just pile up undelivered and the error occurs.

Jesse Glick added a comment - 2015-09-25 22:00 Rather a file descriptor leak; the listener in DefaultStepContext gets notified asynchronously, so not fast enough to close the log files from completed steps. Also a failure to ever remove the listener results in a quadratic slowdown. With that fixed, the insertion of sleep unit: 'MILLISECONDS' , time: 1 inside the loop lets it work, but without that extra pause the notifications just pile up undelivered and the error occurs.

SCM/JIRA link daemon added a comment - 2016-01-11 18:36

Code changed in jenkins
User: Jesse Glick
Path:
aggregator/src/test/java/org/jenkinsci/plugins/workflow/ScalabilityTest.java
support/src/main/java/org/jenkinsci/plugins/workflow/support/storage/SimpleXStreamFlowNodeStorage.java
http://jenkins-ci.org/commit/workflow-plugin/d3dc35a0f511610bd5ca56ea44317c4200c4953a
Log:
Reproduced the originally reported exception from ~~JENKINS-30055~~, perhaps independent of ~~JENKINS-30651~~.

SCM/JIRA link daemon added a comment - 2016-01-11 18:36 Code changed in jenkins User: Jesse Glick Path: aggregator/src/test/java/org/jenkinsci/plugins/workflow/ScalabilityTest.java support/src/main/java/org/jenkinsci/plugins/workflow/support/storage/SimpleXStreamFlowNodeStorage.java http://jenkins-ci.org/commit/workflow-plugin/d3dc35a0f511610bd5ca56ea44317c4200c4953a Log: Reproduced the originally reported exception from JENKINS-30055 , perhaps independent of JENKINS-30651 .

SCM/JIRA link daemon added a comment - 2016-01-11 18:36

Code changed in jenkins
User: Jesse Glick
Path:
CHANGES.md
aggregator/src/test/java/org/jenkinsci/plugins/workflow/ScalabilityTest.java
api/src/main/java/org/jenkinsci/plugins/workflow/flow/FlowExecution.java
api/src/main/java/org/jenkinsci/plugins/workflow/flow/GraphListener.java
api/src/main/java/org/jenkinsci/plugins/workflow/graph/BlockEndNode.java
api/src/main/java/org/jenkinsci/plugins/workflow/graph/FlowNode.java
api/src/main/java/org/jenkinsci/plugins/workflow/graph/ForkNode.java
api/src/main/java/org/jenkinsci/plugins/workflow/graph/JoinNode.java
cps/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java
cps/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java
cps/src/main/java/org/jenkinsci/plugins/workflow/cps/FlowHead.java
job/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java
support/src/main/java/org/jenkinsci/plugins/workflow/support/DefaultStepContext.java
support/src/main/java/org/jenkinsci/plugins/workflow/support/storage/SimpleXStreamFlowNodeStorage.java
http://jenkins-ci.org/commit/workflow-plugin/0fdf534bce127223058749643753e2e9206fd448
Log:
Merge pull request #213 from jglick/many-steps-~~JENKINS-30055~~

~~JENKINS-30055~~ Handle a large number of steps in quick succession

Compare: https://github.com/jenkinsci/workflow-plugin/compare/97ce55c78e77...0fdf534bce12

SCM/JIRA link daemon added a comment - 2016-01-11 18:36 Code changed in jenkins User: Jesse Glick Path: CHANGES.md aggregator/src/test/java/org/jenkinsci/plugins/workflow/ScalabilityTest.java api/src/main/java/org/jenkinsci/plugins/workflow/flow/FlowExecution.java api/src/main/java/org/jenkinsci/plugins/workflow/flow/GraphListener.java api/src/main/java/org/jenkinsci/plugins/workflow/graph/BlockEndNode.java api/src/main/java/org/jenkinsci/plugins/workflow/graph/FlowNode.java api/src/main/java/org/jenkinsci/plugins/workflow/graph/ForkNode.java api/src/main/java/org/jenkinsci/plugins/workflow/graph/JoinNode.java cps/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java cps/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java cps/src/main/java/org/jenkinsci/plugins/workflow/cps/FlowHead.java job/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java support/src/main/java/org/jenkinsci/plugins/workflow/support/DefaultStepContext.java support/src/main/java/org/jenkinsci/plugins/workflow/support/storage/SimpleXStreamFlowNodeStorage.java http://jenkins-ci.org/commit/workflow-plugin/0fdf534bce127223058749643753e2e9206fd448 Log: Merge pull request #213 from jglick/many-steps- JENKINS-30055 JENKINS-30055 Handle a large number of steps in quick succession Compare: https://github.com/jenkinsci/workflow-plugin/compare/97ce55c78e77...0fdf534bce12

SCM/JIRA link daemon added a comment - 2016-04-05 19:52

Code changed in jenkins
User: Jesse Glick
Path:
api/src/main/java/org/jenkinsci/plugins/workflow/flow/FlowExecution.java
http://jenkins-ci.org/commit/workflow-api-plugin/f697e0cfd884f7106cb7066364b2eb372206c6a7
Log:
[FIXED JENKINS-30055] The listener which closes flow graph log files must receive events immediately, and unregister itself.
Originally-Committed-As: 1084a9bd2495989bb78e013c08c8348542e7955f

SCM/JIRA link daemon added a comment - 2016-04-05 19:52 Code changed in jenkins User: Jesse Glick Path: api/src/main/java/org/jenkinsci/plugins/workflow/flow/FlowExecution.java http://jenkins-ci.org/commit/workflow-api-plugin/f697e0cfd884f7106cb7066364b2eb372206c6a7 Log: [FIXED JENKINS-30055] The listener which closes flow graph log files must receive events immediately, and unregister itself. Originally-Committed-As: 1084a9bd2495989bb78e013c08c8348542e7955f

SCM/JIRA link daemon added a comment - 2016-04-05 20:03

Code changed in jenkins
User: Jesse Glick
Path:
aggregator/src/test/java/org/jenkinsci/plugins/workflow/ScalabilityTest.java
support/src/main/java/org/jenkinsci/plugins/workflow/support/DefaultStepContext.java
http://jenkins-ci.org/commit/workflow-support-plugin/8379c0d2a7c4e36de305db68bc54acc2418b7c47
Log:
[FIXED JENKINS-30055] The listener which closes flow graph log files must receive events immediately, and unregister itself.
Originally-Committed-As: 1084a9bd2495989bb78e013c08c8348542e7955f

SCM/JIRA link daemon added a comment - 2016-04-05 20:03 Code changed in jenkins User: Jesse Glick Path: aggregator/src/test/java/org/jenkinsci/plugins/workflow/ScalabilityTest.java support/src/main/java/org/jenkinsci/plugins/workflow/support/DefaultStepContext.java http://jenkins-ci.org/commit/workflow-support-plugin/8379c0d2a7c4e36de305db68bc54acc2418b7c47 Log: [FIXED JENKINS-30055] The listener which closes flow graph log files must receive events immediately, and unregister itself. Originally-Committed-As: 1084a9bd2495989bb78e013c08c8348542e7955f

SCM/JIRA link daemon added a comment - 2016-04-05 20:03

Code changed in jenkins
User: Jesse Glick
Path:
aggregator/src/test/java/org/jenkinsci/plugins/workflow/ScalabilityTest.java
support/src/main/java/org/jenkinsci/plugins/workflow/support/storage/SimpleXStreamFlowNodeStorage.java
http://jenkins-ci.org/commit/workflow-support-plugin/ad18aafe913d801eeb88594d89b176df024495b4
Log:
Reproduced the originally reported exception from ~~JENKINS-30055~~, perhaps independent of ~~JENKINS-30651~~.

Originally-Committed-As: d3dc35a0f511610bd5ca56ea44317c4200c4953a

SCM/JIRA link daemon added a comment - 2016-04-05 20:03 Code changed in jenkins User: Jesse Glick Path: aggregator/src/test/java/org/jenkinsci/plugins/workflow/ScalabilityTest.java support/src/main/java/org/jenkinsci/plugins/workflow/support/storage/SimpleXStreamFlowNodeStorage.java http://jenkins-ci.org/commit/workflow-support-plugin/ad18aafe913d801eeb88594d89b176df024495b4 Log: Reproduced the originally reported exception from JENKINS-30055 , perhaps independent of JENKINS-30651 . Originally-Committed-As: d3dc35a0f511610bd5ca56ea44317c4200c4953a

SCM/JIRA link daemon added a comment - 2016-04-05 20:30

Code changed in jenkins
User: Jesse Glick
Path:
cps/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java
cps/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java
cps/src/main/java/org/jenkinsci/plugins/workflow/cps/FlowHead.java
http://jenkins-ci.org/commit/workflow-cps-plugin/630edfb4313db63d01f6eb135cbce7081f496fae
Log:
[FIXED JENKINS-30055] The listener which closes flow graph log files must receive events immediately, and unregister itself.
Originally-Committed-As: 1084a9bd2495989bb78e013c08c8348542e7955f

SCM/JIRA link daemon added a comment - 2016-04-05 20:30 Code changed in jenkins User: Jesse Glick Path: cps/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java cps/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java cps/src/main/java/org/jenkinsci/plugins/workflow/cps/FlowHead.java http://jenkins-ci.org/commit/workflow-cps-plugin/630edfb4313db63d01f6eb135cbce7081f496fae Log: [FIXED JENKINS-30055] The listener which closes flow graph log files must receive events immediately, and unregister itself. Originally-Committed-As: 1084a9bd2495989bb78e013c08c8348542e7955f

SCM/JIRA link daemon added a comment - 2016-04-05 20:31

Code changed in jenkins
User: Jesse Glick
Path:
job/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java
http://jenkins-ci.org/commit/workflow-job-plugin/35a9115af586c5c0f3283e9cbebcab62c647c7ee
Log:
[FIXED JENKINS-30055] The listener which closes flow graph log files must receive events immediately, and unregister itself.
Originally-Committed-As: 1084a9bd2495989bb78e013c08c8348542e7955f

SCM/JIRA link daemon added a comment - 2016-04-05 20:31 Code changed in jenkins User: Jesse Glick Path: job/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java http://jenkins-ci.org/commit/workflow-job-plugin/35a9115af586c5c0f3283e9cbebcab62c647c7ee Log: [FIXED JENKINS-30055] The listener which closes flow graph log files must receive events immediately, and unregister itself. Originally-Committed-As: 1084a9bd2495989bb78e013c08c8348542e7955f

Assignee:: Jesse Glick

Reporter:: Tom FENNELLY

Votes:: 2 Vote for this issue

Watchers:: 7 Start watching this issue

Created:: 2015-08-20 12:49

Updated:: 2016-08-26 21:55

Resolved:: 2016-01-11 18:36

Details

Description

Attachments

Issue Links

Activity

Collapse comment: Jesse Glick added a comment - 2015-08-21 15:29

Expand comment: Jesse Glick added a comment - 2015-08-21 15:29

Collapse comment: Jesse Glick added a comment - 2015-09-25 22:00

Expand comment: Jesse Glick added a comment - 2015-09-25 22:00

Collapse comment: SCM/JIRA link daemon added a comment - 2016-01-11 18:36

Expand comment: SCM/JIRA link daemon added a comment - 2016-01-11 18:36

Collapse comment: SCM/JIRA link daemon added a comment - 2016-01-11 18:36

Expand comment: SCM/JIRA link daemon added a comment - 2016-01-11 18:36

Collapse comment: SCM/JIRA link daemon added a comment - 2016-01-11 18:36

Expand comment: SCM/JIRA link daemon added a comment - 2016-01-11 18:36

Collapse comment: SCM/JIRA link daemon added a comment - 2016-04-05 19:52

Expand comment: SCM/JIRA link daemon added a comment - 2016-04-05 19:52

Collapse comment: SCM/JIRA link daemon added a comment - 2016-04-05 20:03

Expand comment: SCM/JIRA link daemon added a comment - 2016-04-05 20:03

Collapse comment: SCM/JIRA link daemon added a comment - 2016-04-05 20:03

Expand comment: SCM/JIRA link daemon added a comment - 2016-04-05 20:03

Collapse comment: SCM/JIRA link daemon added a comment - 2016-04-05 20:30

Expand comment: SCM/JIRA link daemon added a comment - 2016-04-05 20:30

Collapse comment: SCM/JIRA link daemon added a comment - 2016-04-05 20:31

Expand comment: SCM/JIRA link daemon added a comment - 2016-04-05 20:31

People

Dates