-
Bug
-
Resolution: Fixed
-
Minor
-
None
-
Jenkins v2.47
latest pipeline plugins
latest lockable-resources-plugin v1.11.1
-
-
lockable-resources 2.1
Not fully serializable. This occurred in one of our multi-branch pipeline builds, all previous/subsequent builds went fine.
java.lang.RuntimeException: Failed to serialize org.jenkins.plugins.lockableresources.LockableResourcesManager#queuedContexts for class org.jenkins.plugins.lockableresources.LockableResourcesManager at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015) at com.thoughtworks.xstream.XStream.toXML(XStream.java:988) at hudson.XmlFile.write(XmlFile.java:170) at hudson.model.Descriptor.save(Descriptor.java:871) at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlockNames(LockableResourcesManager.java:361) at org.jenkins.plugins.lockableresources.LockStepExecution$Callback.finished(LockStepExecution.java:84) at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onSuccess(BodyExecutionCallback.java:114) at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:362) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:60) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82) at sun.reflect.GeneratedMethodAccessor274.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.dispatch(CollectionLiteralBlock.java:55) at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.item(CollectionLiteralBlock.java:45) at sun.reflect.GeneratedMethodAccessor273.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21) at com.cloudbees.groovy.cps.Next.step(Next.java:74) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:165) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:328) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:80) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:240) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:228) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) at java.util.ArrayList$Itr.next(ArrayList.java:851) at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:73) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252) ... 53 more
our Jenkinsfiel looks basically like this:
try { timestamps { node { stage('Prepare workspace') { properties(...) // set up git connection properties checkout scm String jdkHome = "${tool name: 'jdk8', type: 'hudson.model.JDK'}" env.PATH = "${jdkHome}/bin:${env.PATH}" } mainBuildStages() milestone(label: 'MainBuild-Done') } stage('Integration testing') { lock(resource: 'integrationTestsLock', inversePrecedence: true) { integrationTestsStage() milestone(label:'IntegrationTests-Done') } } } } catch (err) { // email notifications etc throw err } def mainBuildStages() { // run main build using mvn } def integrationTestsStage() { def pipelineRunBranches = [:] pipelineRunBranches['itests-withEmptyDatabase'] = { node { // unstash and run itests using mvn } } pipelineRunBranches['itests-withPopulatedDatabase'] = { node { // unstash and run other itests using mvn } } parallel pipelineRunBranches }
The error occurred at the very end, just before the pipeline was done:
[....] [Pipeline] // parallel [Pipeline] milestone (IntegrationTests-Done) 16:25:48 Trying to pass milestone 1 [Pipeline] } [Pipeline] // lock [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timestamps --> here the exception occurred
- relates to
-
-
- In Progress
-
-
-
- Reopened
-
[JENKINS-42720] ConcurrentModificationException when serializing LockableResourcesManager
We have been experiencing this bug too....not sure why it's tagged as Minor, since it causes an exception that stops the pipeline job. I would consider it to be Critical.
I am not a Java programmer, but from the callstack I see that the problem is that on LockableResourcesManager.queueContext, we tried to save the locks to an xml file, but the serialization fails since another thread is modifying the queueContexts private member. I only see 2 spots where that member is being modified, on queueContext and on unqueueContext. I can see that those functions are not set to be "synchronized"....so shouldn't they be and that would fix the issue? I can work on a PR, but wanted to put my notes here to see if someone else can comment on the validity of my rationale....abayer?
Alejandro del Castillo
added a comment - - edited We have been experiencing this bug too....not sure why it's tagged as Minor, since it causes an exception that stops the pipeline job. I would consider it to be Critical.
I am not a Java programmer, but from the callstack I see that the problem is that on LockableResourcesManager.queueContext, we tried to save the locks to an xml file, but the serialization fails since another thread is modifying the queueContexts private member. I only see 2 spots where that member is being modified, on queueContext and on unqueueContext. I can see that those functions are not set to be "synchronized"....so shouldn't they be and that would fix the issue? I can work on a PR, but wanted to put my notes here to see if someone else can comment on the validity of my rationale.... abayer ? Alejandro del Castillo
added a comment - Got another call stack that, while being different seems to point to the same issue. I haven't noticed this failures, since the pipeline instance is set to be superseeded....but when I looked a the log directly, found the exception. The failure is on unqueueContext.....what I am suspecting is that if you have a pipeline instance that is about to try to grab the lock (call queueContext) while another one is being superseeded, then you may hit the concurrency problem. I asume the save() method takes a while, so that's the window where any other queue/unqueue operation will fail.
2017-05-12 08:48:36 -0600 [L2] is locked, waiting...
Superseded by NXG-Pipeline#1560
[Pipeline] // lock
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] End of Pipeline
java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.remove(Unknown Source)
at org.jenkins.plugins.lockableresources.LockableResourcesManager.unqueueContext(LockableResourcesManager.java:565)
at org.jenkins.plugins.lockableresources.LockStepExecution.stop(LockStepExecution.java:95)
at org.jenkinsci.plugins.workflow.cps.CpsThread.stop(CpsThread.java:287)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$6.onSuccess(CpsFlowExecution.java:872)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$6.onSuccess(CpsFlowExecution.java:861)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4$1.run(CpsFlowExecution.java:658)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Alejandro del Castillo
added a comment - Got another call stack that, while being different seems to point to the same issue. I haven't noticed this failures, since the pipeline instance is set to be superseeded....but when I looked a the log directly, found the exception. The failure is on unqueueContext.....what I am suspecting is that if you have a pipeline instance that is about to try to grab the lock (call queueContext) while another one is being superseeded, then you may hit the concurrency problem. I asume the save() method takes a while, so that's the window where any other queue/unqueue operation will fail.
2017-05-12 08:48:36 -0600 [L2] is locked, waiting...
Superseded by NXG-Pipeline#1560
[Pipeline] // lock
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] End of Pipeline
java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.remove(Unknown Source)
at org.jenkins.plugins.lockableresources.LockableResourcesManager.unqueueContext(LockableResourcesManager.java:565)
at org.jenkins.plugins.lockableresources.LockStepExecution.stop(LockStepExecution.java:95)
at org.jenkinsci.plugins.workflow.cps.CpsThread.stop(CpsThread.java:287)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$6.onSuccess(CpsFlowExecution.java:872)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$6.onSuccess(CpsFlowExecution.java:861)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4$1.run(CpsFlowExecution.java:658)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source) slightly different stacktrace today in the same pipeline (with no notable changes since when this issue was created) where the original exception occurred:
[...] [Pipeline] } [Pipeline] // timestamps [Pipeline] End of Pipeline java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) at java.util.ArrayList$Itr.next(ArrayList.java:851) at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:73) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252) Caused: java.lang.RuntimeException: Failed to serialize org.jenkins.plugins.lockableresources.LockableResourcesManager#queuedContexts for class org.jenkins.plugins.lockableresources.LockableResourcesManager at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015) at com.thoughtworks.xstream.XStream.toXML(XStream.java:988) at hudson.XmlFile.write(XmlFile.java:171) at hudson.model.Descriptor.save(Descriptor.java:872) at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlockNames(LockableResourcesManager.java:422) at org.jenkins.plugins.lockableresources.LockStepExecution$Callback.finished(LockStepExecution.java:84) at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onSuccess(BodyExecutionCallback.java:114) at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:362) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:60) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82) at sun.reflect.GeneratedMethodAccessor303.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.dispatch(CollectionLiteralBlock.java:55) at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.item(CollectionLiteralBlock.java:45) at sun.reflect.GeneratedMethodAccessor302.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21) at com.cloudbees.groovy.cps.Next.step(Next.java:83) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:173) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:162) at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122) at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:162) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:35) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:32) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:32) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:330) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:242) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:230) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Finished: FAILURE
Our current jenkins env: Jenkins v 2.64, Lockable resources plugin v2.0, latest versions of all other plugins.
Alejandro del Castillo
added a comment - yeah, I believe it might be the same issue. I created a PR [1] with my attempt to fix the problem. I haven't been able to reproduce it with the fix, but I was never able to reproduce it reliably. You may want to try the fix, then would love to hear if it indeed fixes your instance of the problem.
[1] https://github.com/jenkinsci/lockable-resources-plugin/pull/57
Alejandro del Castillo
added a comment - yeah, I believe it might be the same issue. I created a PR [1] with my attempt to fix the problem. I haven't been able to reproduce it with the fix, but I was never able to reproduce it reliably. You may want to try the fix, then would love to hear if it indeed fixes your instance of the problem.
[1] https://github.com/jenkinsci/lockable-resources-plugin/pull/57 
Constantin Bugneac
added a comment - experiencing the same issue, are there any workarounds meanwhile ?
Constantin Bugneac
added a comment - experiencing the same issue, are there any workarounds meanwhile ? 
Tobias Gruetzmacher
added a comment - I suspect this was about the same issue as JENKINS-44597, which was fixed in release 2.1 - If you see similar issues, please open a new issue with new evidence.
Tobias Gruetzmacher
added a comment - I suspect this was about the same issue as JENKINS-44597 , which was fixed in release 2.1 - If you see similar issues, please open a new issue with new evidence. Yes thanks, I confirm that this is solved, I haven't seen this issue in the last months. So closing this issue.
We've hit this issue as well. In common with stefanthurnherr we have, after our main build stage, a parallel set of steps as the final set of steps. This could be relevant in identifying the cause if there is a race or other concurrency issue occuring.