-
Bug
-
Resolution: Unresolved
-
Minor
-
Powered by SuggestiMate -
workflow-cps-global-lib 2.18
This keeps up reported as Flaked tests as it seems to clash with some other test (related activity) and gets retried correctly. (Using synchronized mocks makes no difference).
Reproducible by https://github.com/jenkinsci/resource-disposer-plugin/commit/bc5f00008cc31a680e103dd7bd804bd64b0a4668 and mvn clean package. Not by mvn clean package -Dtest=AsyncResourceDisposerTest#combined of course.
Caused by: java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:953) at java.util.LinkedList$ListItr.next(LinkedList.java:886) at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:73) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:223) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:210) ... 128 more
- relates to
-
JENKINS-19561 Unsafe & inefficient concurrency in XStream
-
- Fixed but Unreleased
-
-
JENKINS-42720 ConcurrentModificationException when serializing LockableResourcesManager
-
- Closed
-
- links to
[JENKINS-41037] Unable to serialize lists because of ConcurrentModificationException
Issue seen:
java.util.ConcurrentModificationException
15:00:02 at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1042)
15:00:02 at java.base/java.util.ArrayList$Itr.next(ArrayList.java:996)
15:00:02 at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:73)
15:00:02 at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
15:00:02 at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
15:00:02 at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
15:00:02 at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:263)
15:00:02 at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:250)
15:00:02 Caused: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.workflow.libs.LibrariesAction#libraries for class org.jenkinsci.plugins.workflow.libs.LibrariesAction
15:00:02 at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:254)
15:00:02 at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:222)
15:00:02 at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
15:00:02 at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:208)
15:00:02 at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:149)
15:00:02 at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
15:00:02 at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
15:00:02 at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
15:00:02 at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88)
15:00:02 at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64)
15:00:02 at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74)
15:00:02 at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
15:00:02 at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
15:00:02 at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
15:00:02 at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:263)
15:00:02 at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:250)
15:00:02 Caused: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun
15:00:02 at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:254)
15:00:02 at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:222)
15:00:02 at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
15:00:02 at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:208)
15:00:02 at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:149)
15:00:02 at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
15:00:02 at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
15:00:02 at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
15:00:02 at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
15:00:02 at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
15:00:02 at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
15:00:02 at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
15:00:02 at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
15:00:02 at hudson.util.XStream2.toXMLUTF8(XStream2.java:313)
15:00:02 at org.jenkinsci.plugins.workflow.support.PipelineIOUtils.writeByXStream(PipelineIOUtils.java:34)
15:00:02 at org.jenkinsci.plugins.workflow.job.WorkflowRun.save(WorkflowRun.java:1143)
15:00:02 at hudson.BulkChange.commit(BulkChange.java:98)
15:00:02 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1475)
15:00:02 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458)
15:00:02 at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:37)
15:00:02 at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
15:00:02 at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
15:00:02 at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
15:00:02 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
15:00:02 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
15:00:02 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
15:00:02 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
15:00:02 at java.base/java.lang.Thread.run(Thread.java:834)
I see exactly the same issue;
[2020-02-23T12:22:16.287Z] java.util.ConcurrentModificationException [2020-02-23T12:22:16.287Z] at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909) [2020-02-23T12:22:16.287Z] at java.util.ArrayList$Itr.next(ArrayList.java:859) [2020-02-23T12:22:16.287Z] at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:73) [2020-02-23T12:22:16.287Z] at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) [2020-02-23T12:22:16.287Z] at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) [2020-02-23T12:22:16.287Z] at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) [2020-02-23T12:22:16.287Z] at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:263) [2020-02-23T12:22:16.287Z] at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:250) [2020-02-23T12:22:16.287Z] Caused: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.workflow.libs.LibrariesAction#libraries for class org.jenkinsci.plugins.workflow.libs.LibrariesAction [2020-02-23T12:22:16.288Z] at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:254) [2020-02-23T12:22:16.288Z] at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:222) [2020-02-23T12:22:16.288Z] at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) [2020-02-23T12:22:16.288Z] at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:208) [2020-02-23T12:22:16.288Z] at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:149) [2020-02-23T12:22:16.288Z] at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) [2020-02-23T12:22:16.288Z] at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) [2020-02-23T12:22:16.288Z] at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) [2020-02-23T12:22:16.288Z] at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88) [2020-02-23T12:22:16.288Z] at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64) [2020-02-23T12:22:16.288Z] at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74) [2020-02-23T12:22:16.288Z] at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) [2020-02-23T12:22:16.288Z] at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) [2020-02-23T12:22:16.288Z] at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) [2020-02-23T12:22:16.288Z] at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:263) [2020-02-23T12:22:16.288Z] at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:250) [2020-02-23T12:22:16.288Z] Caused: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun [2020-02-23T12:22:16.288Z] at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:254) [2020-02-23T12:22:16.288Z] at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:222) [2020-02-23T12:22:16.288Z] at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) [2020-02-23T12:22:16.288Z] at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:208) [2020-02-23T12:22:16.288Z] at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:149) [2020-02-23T12:22:16.288Z] at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) [2020-02-23T12:22:16.288Z] at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) [2020-02-23T12:22:16.288Z] at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) [2020-02-23T12:22:16.288Z] at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82) [2020-02-23T12:22:16.288Z] at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37) [2020-02-23T12:22:16.288Z] at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026) [2020-02-23T12:22:16.288Z] at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015) [2020-02-23T12:22:16.288Z] at com.thoughtworks.xstream.XStream.toXML(XStream.java:988) [2020-02-23T12:22:16.288Z] at hudson.util.XStream2.toXMLUTF8(XStream2.java:313) [2020-02-23T12:22:16.288Z] at org.jenkinsci.plugins.workflow.support.PipelineIOUtils.writeByXStream(PipelineIOUtils.java:34) [2020-02-23T12:22:16.288Z] at org.jenkinsci.plugins.workflow.job.WorkflowRun.save(WorkflowRun.java:1140) [2020-02-23T12:22:16.288Z] at hudson.BulkChange.commit(BulkChange.java:98) [2020-02-23T12:22:16.288Z] at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1475) [2020-02-23T12:22:16.288Z] at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458) [2020-02-23T12:22:16.288Z] at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:37) [2020-02-23T12:22:16.288Z] at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131) [2020-02-23T12:22:16.288Z] at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) [2020-02-23T12:22:16.288Z] at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) [2020-02-23T12:22:16.288Z] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [2020-02-23T12:22:16.288Z] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [2020-02-23T12:22:16.288Z] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [2020-02-23T12:22:16.288Z] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [2020-02-23T12:22:16.288Z] at java.lang.Thread.run(Thread.java:748)
I am seeing the same issue when loading libraries dynamically using
library()
It doesn't seem to matter if the library statement is the first line in the pipeline, nor inside a node.
I have noticed however that this will almost always happen on build #1 of a multibranch pipeline
it does on occasion occur on other builds but less frequently
[Pipeline] Start of Pipeline [Pipeline] library Loading library libs@libs [Pipeline] End of Pipeline java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909) at java.util.ArrayList$Itr.next(ArrayList.java:859) at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:73) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:263) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:250) Caused: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.workflow.libs.LibrariesAction#libraries for class org.jenkinsci.plugins.workflow.libs.LibrariesAction at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:254) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:222) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:208) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:149) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88) at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64) at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:263) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:250) Caused: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:254) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:222) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:208) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:149) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015) at com.thoughtworks.xstream.XStream.toXML(XStream.java:988) at hudson.util.XStream2.toXMLUTF8(XStream2.java:313) at org.jenkinsci.plugins.workflow.support.PipelineIOUtils.writeByXStream(PipelineIOUtils.java:34) at org.jenkinsci.plugins.workflow.job.WorkflowRun.save(WorkflowRun.java:1142) at hudson.BulkChange.commit(BulkChange.java:98) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1476) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Finished: FAILURE
Under discussion in https://github.com/x-stream/xstream/pull/222 but probably this will need to be split into distinct issues, one for each collection which is being concurrently modified during save, and each case fixed independently.
workflow-cps-global-lib-plugin #100 should solve the most-reported issues. Looks like there are a couple more in other plugins.
jenkinsci/resource-disposer-plugin#8 fixes this issue in Resource Disposer's test suite. At the time of this writing, the only remaining known instance of this issue is in Cloud Statistics, as described in this comment.
is there any plan to release a new version of workflow-cps-global-lib-plugin with the fix?
is there any plan to release a new version of workflow-cps-global-lib-plugin with the fix?
That was done in 2.18.
We ran into this issue on the first build of a branch pipeline job. Same exact stack trace as Bahram Parsapour ran into.