[JENKINS-47201] ConcurrentModificationException when serializing MultiSCMRevisionState#revisionStates

Type: Bug
Resolution: Fixed
Priority: Major
Component/s: workflow-scm-step-plugin
Labels:
None
Environment:
Jenkins 2.73.1
workflow-scm-step-plugin 2.6

Similar Issues:
Powered by SuggestiMate

Show
Released As:
Pipeline SCM Step 2.7

Since updating to Jenkins 2.73.1 and adding another build slave, we're often seeing exceptions when one of the parallel pipeline steps finishes.

All the other parallel running steps hang after that exception is thrown, effectively blocking all executors of our CI systems until the jobs are manually cancelled.

https://github.com/dlang/ci/blob/71199e5f0d5ee5a12aaa0985ee89e23b9e59fa9f/pipeline.groovy

https://ci.dlang.io/job/dlang-org/job/dmd/job/master/364/consoleFull

java.util.ConcurrentModificationException
	at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
	at java.util.HashMap$EntryIterator.next(HashMap.java:1471)
	at java.util.HashMap$EntryIterator.next(HashMap.java:1469)
	at com.thoughtworks.xstream.converters.collections.MapConverter.marshal(MapConverter.java:75)
	at 
...
Caused: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState#revisionStates for class org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
	at 
...
Caused: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
	at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
	at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
	at hudson.XmlFile.write(XmlFile.java:171)
	at hudson.model.Run.save(Run.java:1933)
	at hudson.BulkChange.commit(BulkChange.java:98)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1221)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:407)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Maybe a race condition on MultiSCMRevisionState's revisionStates Map?

Seems to be similar to ~~JENKINS-34313~~, but that seemed to primarily report a different issue.

duplicates

JENKINS-34313 SAXParseException from parallel SCM checkout

Resolved

relates to

JENKINS-43176 MercurialChangeLogParser fails in parallel checkouts

Open

links to

https://github.com/jenkinsci/workflow-scm-step-plugin/pull/21

jenkinsci/workflow-scm-step-plugin#27

Martin Nowak created issue - 2017-09-28 14:48

Martin Nowak made changes - 2017-10-07 12:54

Priority

Original: Minor [ 4 ]

New: Major [ 3 ]

Martin Nowak made changes - 2017-10-07 12:56

Description

Original: Since updating to Jenkins 2.73.1 and adding another build slave, we're often seeing exceptions when one of the parallel pipeline steps finishes.

https://github.com/dlang/ci/blob/71199e5f0d5ee5a12aaa0985ee89e23b9e59fa9f/pipeline.groovy

https://ci.dlang.io/job/dlang-org/job/dmd/job/master/364/consoleFull
{noformat}
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
at java.util.HashMap$EntryIterator.next(HashMap.java:1471)
at java.util.HashMap$EntryIterator.next(HashMap.java:1469)
at com.thoughtworks.xstream.converters.collections.MapConverter.marshal(MapConverter.java:75)
at
...
Caused: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState#revisionStates for class org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState
at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
at
...
Caused: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun
at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
at hudson.XmlFile.write(XmlFile.java:171)
at hudson.model.Run.save(Run.java:1933)
at hudson.BulkChange.commit(BulkChange.java:98)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1221)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:407)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
Maybe a race condition on MultiSCMRevisionState's [revisionStates|https://github.com/jenkinsci/workflow-scm-step-plugin/blob/fb41e91dd095c7b446f946ecaa059f7f74bbd2ce/src/main/java/org/jenkinsci/plugins/workflow/steps/scm/MultiSCMRevisionState.java#L41] Map?

Seems to be similar to ~~JENKINS-34313~~, but that seemed to primarily report a different issue.

New: Since updating to Jenkins 2.73.1 and adding another build slave, we're often seeing exceptions when one of the parallel pipeline steps finishes.

All the other parallel running steps hang after that exception is thrown, effectively blocking all executors of our CI systems until the jobs are manually cancelled.

[https://github.com/dlang/ci/blob/71199e5f0d5ee5a12aaa0985ee89e23b9e59fa9f/pipeline.groovy]

[https://ci.dlang.io/job/dlang-org/job/dmd/job/master/364/consoleFull]
{noformat}
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
at java.util.HashMap$EntryIterator.next(HashMap.java:1471)
at java.util.HashMap$EntryIterator.next(HashMap.java:1469)
at com.thoughtworks.xstream.converters.collections.MapConverter.marshal(MapConverter.java:75)
at
...
Caused: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState#revisionStates for class org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState
at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
at
...
Caused: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun
at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
at hudson.XmlFile.write(XmlFile.java:171)
at hudson.model.Run.save(Run.java:1933)
at hudson.BulkChange.commit(BulkChange.java:98)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1221)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:407)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
Maybe a race condition on MultiSCMRevisionState's [revisionStates|https://github.com/jenkinsci/workflow-scm-step-plugin/blob/fb41e91dd095c7b446f946ecaa059f7f74bbd2ce/src/main/java/org/jenkinsci/plugins/workflow/steps/scm/MultiSCMRevisionState.java#L41] Map?

Seems to be similar to ~~JENKINS-34313~~, but that seemed to primarily report a different issue.

Martin Nowak made changes - 2017-10-13 13:21

Remote Link

New: This issue links to "https://github.com/jenkinsci/workflow-scm-step-plugin/pull/21 (Web Link)" [ 17880 ]

Jesse Glick made changes - 2017-11-16 17:25

Link

New: This issue duplicates ~~JENKINS-34313~~ [ ~~JENKINS-34313~~ ]

Jesse Glick made changes - 2017-11-16 17:25

Resolution		New: Duplicate [ 3 ]
Status	Original: Open [ 1 ]	New: Resolved [ 5 ]

Jesse Glick made changes - 2017-11-16 17:26

Resolution	Original: Duplicate [ 3 ]
Status	Original: Resolved [ 5 ]	New: Reopened [ 4 ]

Jesse Glick made changes - 2017-11-16 17:27

Link

New: This issue relates to JENKINS-43176 [ JENKINS-43176 ]

Jesse Glick made changes - 2018-03-06 14:27

Assignee

New: Martin Nowak [ dawg ]

Jesse Glick made changes - 2018-03-06 14:27

Status

Original: Reopened [ 4 ]

New: Open [ 1 ]

Jesse Glick made changes - 2018-03-06 14:27

Status

Original: Open [ 1 ]

New: In Progress [ 3 ]

Assignee:: Martin Nowak

Reporter:: Martin Nowak

Votes:: 4 Vote for this issue

Watchers:: 9 Start watching this issue

Created:: 2017-09-28 14:48

Updated:: 2018-10-01 20:24

Resolved:: 2018-10-01 20:24

Jenkins

Details

Description

Attachments

Issue Links

Activity

People

Dates