Details
-
Bug
-
Status: Resolved (View Workflow)
-
Major
-
Resolution: Fixed
-
None
-
Jenkins ver. 1.651.1
Pipeline plug-ins: ver. 1.15
Pipeline: SCM Step plug-in: ver. 1.15
SCM API Plugin: ver 1.1
Subversion Plug-in: ver. 2.5.6
-
-
workflow-scm-step 2.8
Description
In out pipeline we at first build multiple upstream Maven projects and then build the main project. Upstream projects are build in parallel:
parallel(
projA: {
projects.buildDependency('projA')
},
projB: {
projects.buildDependency('projB')
}
...
)
projects.buildDependency() just checks-out each project from SVN and starts Maven build. The problem is that we often (in about 20% of builds) have errors caused by concurrent / parallel SVN check-outs. Sometimes it is ConcurrentModificationException and sometimes "SAXParseException: changelog11.xml; lineNumber: 1; columnNumber: 1; Premature end of file". See stacktraces below.
As a temporary workaround we added
sleep time: random.nextInt(1000), unit: 'MILLISECONDS'
just before the checkout step to introduce time shifts between check-outs. We used random numbers because we were not sure if some local or env counter variable would be thread-safe in case of parallel steps.
Now error happens only if time difference between check-outs is narrow, e.g. 5 milliseconds if 2 generated random numbers are close.
Would it be possible to make Pipeline plug-in more resilient in case of parallel check-outs?
Caused by: org.jenkinsci.plugins.workflow.cps.steps.ParallelStepException: Parallel step projA failed ... 26 more Caused by: hudson.util.IOException2: Failed to parse /var/lib/jenkins/jobs/ha-pipeline/builds/426/changelog11.xml at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:638) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1200(WorkflowRun.java:111) at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:812) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:123) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:49) at hudson.security.ACL.impersonate(ACL.java:213) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:47) ... 5 more Caused by: org.xml.sax.SAXParseException; systemId: file:/var/lib/jenkins/jobs/ha-pipeline/builds/426/changelog11.xml; lineNumber: 1; columnNumber: 1; Premature end of file. at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76) ... 15 more
org.jenkinsci.plugins.workflow.cps.steps.ParallelStepException: Parallel step projB failed at org.jenkinsci.plugins.workflow.cps.steps.ParallelStep$ResultHandler$Callback.checkAllDone(ParallelStep.java:153) at org.jenkinsci.plugins.workflow.cps.steps.ParallelStep$ResultHandler$Callback.onSuccess(ParallelStep.java:110) at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:311) at com.cloudbees.groovy.cps.impl.LocalVariableBlock$LocalVariable.get(LocalVariableBlock.java:33) at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30) at com.cloudbees.groovy.cps.impl.LocalVariableBlock.evalLValue(LocalVariableBlock.java:22) at com.cloudbees.groovy.cps.LValueBlock$BlockImpl.eval(LValueBlock.java:55) at com.cloudbees.groovy.cps.LValueBlock.eval(LValueBlock.java:16) at com.cloudbees.groovy.cps.Next.step(Next.java:58) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:106) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:164) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:277) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:77) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:186) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:184) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) Caused by: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:215) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:183) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:168) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:109) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015) at com.thoughtworks.xstream.XStream.toXML(XStream.java:988) at hudson.XmlFile.write(XmlFile.java:178) at hudson.model.Run.save(Run.java:1896) at hudson.util.PersistedList.onModified(PersistedList.java:173) at hudson.util.PersistedList._onModified(PersistedList.java:181) at hudson.util.PersistedList.add(PersistedList.java:72) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:651) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1200(WorkflowRun.java:111) at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:812) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:123) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:49) at hudson.security.ACL.impersonate(ACL.java:213) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState#revisionStates for class org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:215) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:183) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:168) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:109) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88) at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64) at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:224) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:211) ... 31 more Caused by: java.util.ConcurrentModificationException at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922) at java.util.HashMap$EntryIterator.next(HashMap.java:962) at java.util.HashMap$EntryIterator.next(HashMap.java:960) at com.thoughtworks.xstream.converters.collections.MapConverter.marshal(MapConverter.java:75) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:224) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:211) ... 46 more
Attachments
Issue Links
- is duplicated by
-
-
- Open
-
-
-
- Fixed but Unreleased
-
-
-
- Resolved
-
-
-
- Closed
-
-
JENKINS-56254 Parallel Changelogs when multiple scms are used are not persisted between restarts
-
- Resolved
-
- relates to
-
-
- Fixed but Unreleased
-
- links to
(1 relates to, 2 links to)
Activity
Andre Morales
added a comment - I was having the same issue with a pipeline of 23 parallel checkouts. It was failing randomly.
I have installed plugin workflow-scm-step-2.8-rc326.7286a182d923.hpi and everything goes well now. Since installation, all 23 agents running in parallel are checking out correctly.
Andre Morales
added a comment - I was having the same issue with a pipeline of 23 parallel checkouts. It was failing randomly.
I have installed plugin workflow-scm-step-2.8-rc326.7286a182d923.hpi and everything goes well now. Since installation, all 23 agents running in parallel are checking out correctly.
Is there a chance to get a new release of the plugin with this fix included?
That would be great as this issue is really problematic for larger parallel pipelines.
Torsten Werner
added a comment - Please, release the fix. The issue is really a problem for us. A fix for this issue was just released in Pipeline SCM Step Plugin version 2.8.
Hi dnusbaum,
I'm having the same issue for. I'll use workflow-scm-step-2.8-rc326.7286a182d923.hpi to confirm if it's fixing the "ConcurrentModificationException". I'll test it against 100+ parallelized repositories (in groups of 12 concurrent extractions)
Thanks in advance