Details
-
Bug
-
Status: Resolved (View Workflow)
-
Major
-
Resolution: Fixed
-
None
-
Jenkins ver. 1.651.1
Pipeline plug-ins: ver. 1.15
Pipeline: SCM Step plug-in: ver. 1.15
SCM API Plugin: ver 1.1
Subversion Plug-in: ver. 2.5.6
-
-
workflow-scm-step 2.8
Description
In out pipeline we at first build multiple upstream Maven projects and then build the main project. Upstream projects are build in parallel:
parallel(
projA: {
projects.buildDependency('projA')
},
projB: {
projects.buildDependency('projB')
}
...
)
projects.buildDependency() just checks-out each project from SVN and starts Maven build. The problem is that we often (in about 20% of builds) have errors caused by concurrent / parallel SVN check-outs. Sometimes it is ConcurrentModificationException and sometimes "SAXParseException: changelog11.xml; lineNumber: 1; columnNumber: 1; Premature end of file". See stacktraces below.
As a temporary workaround we added
sleep time: random.nextInt(1000), unit: 'MILLISECONDS'
just before the checkout step to introduce time shifts between check-outs. We used random numbers because we were not sure if some local or env counter variable would be thread-safe in case of parallel steps.
Now error happens only if time difference between check-outs is narrow, e.g. 5 milliseconds if 2 generated random numbers are close.
Would it be possible to make Pipeline plug-in more resilient in case of parallel check-outs?
Caused by: org.jenkinsci.plugins.workflow.cps.steps.ParallelStepException: Parallel step projA failed ... 26 more Caused by: hudson.util.IOException2: Failed to parse /var/lib/jenkins/jobs/ha-pipeline/builds/426/changelog11.xml at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:638) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1200(WorkflowRun.java:111) at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:812) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:123) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:49) at hudson.security.ACL.impersonate(ACL.java:213) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:47) ... 5 more Caused by: org.xml.sax.SAXParseException; systemId: file:/var/lib/jenkins/jobs/ha-pipeline/builds/426/changelog11.xml; lineNumber: 1; columnNumber: 1; Premature end of file. at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76) ... 15 more
org.jenkinsci.plugins.workflow.cps.steps.ParallelStepException: Parallel step projB failed at org.jenkinsci.plugins.workflow.cps.steps.ParallelStep$ResultHandler$Callback.checkAllDone(ParallelStep.java:153) at org.jenkinsci.plugins.workflow.cps.steps.ParallelStep$ResultHandler$Callback.onSuccess(ParallelStep.java:110) at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:311) at com.cloudbees.groovy.cps.impl.LocalVariableBlock$LocalVariable.get(LocalVariableBlock.java:33) at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30) at com.cloudbees.groovy.cps.impl.LocalVariableBlock.evalLValue(LocalVariableBlock.java:22) at com.cloudbees.groovy.cps.LValueBlock$BlockImpl.eval(LValueBlock.java:55) at com.cloudbees.groovy.cps.LValueBlock.eval(LValueBlock.java:16) at com.cloudbees.groovy.cps.Next.step(Next.java:58) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:106) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:164) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:277) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:77) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:186) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:184) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) Caused by: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:215) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:183) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:168) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:109) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015) at com.thoughtworks.xstream.XStream.toXML(XStream.java:988) at hudson.XmlFile.write(XmlFile.java:178) at hudson.model.Run.save(Run.java:1896) at hudson.util.PersistedList.onModified(PersistedList.java:173) at hudson.util.PersistedList._onModified(PersistedList.java:181) at hudson.util.PersistedList.add(PersistedList.java:72) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:651) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1200(WorkflowRun.java:111) at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:812) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:123) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:49) at hudson.security.ACL.impersonate(ACL.java:213) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState#revisionStates for class org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:215) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:183) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:168) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:109) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88) at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64) at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:224) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:211) ... 31 more Caused by: java.util.ConcurrentModificationException at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922) at java.util.HashMap$EntryIterator.next(HashMap.java:962) at java.util.HashMap$EntryIterator.next(HashMap.java:960) at com.thoughtworks.xstream.converters.collections.MapConverter.marshal(MapConverter.java:75) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:224) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:211) ... 46 more
Attachments
Issue Links
- is duplicated by
-
-
- Open
-
-
-
- Fixed but Unreleased
-
-
-
- Resolved
-
-
-
- Closed
-
-
JENKINS-56254 Parallel Changelogs when multiple scms are used are not persisted between restarts
-
- Resolved
-
- relates to
-
-
- Fixed but Unreleased
-
- links to
(1 relates to, 2 links to)
Activity
I believe we are seeing the same problem, using Git.
The below error happens when running tests in parallel, on different nodes, and each node triggering `checkout(scm)` using Jenkins Pipeline workflow.
[test:1] > git rev-parse --is-inside-work-tree # timeout=10 [test:1] Fetching changes from 2 remote Git repositories [test:1] > git config remote.origin.url https://github.com/org/our-repo.git # timeout=10 [test:1] Fetching upstream changes from https://github.com/org/our-repo.git [test:1] > git --version # timeout=10 [test:1] using GIT_ASKPASS to set credentials github login credentials for "ci" account [test:1] > git fetch --tags --progress https://github.com/org/our-repo.git +refs/heads/*:refs/remotes/origin/* [test:1] > git config remote.origin1.url https://github.com/org/our-repo.git # timeout=10 [test:1] ERROR: Error fetching remote repo 'origin1' [test:1] hudson.plugins.git.GitException: Failed to fetch from https://github.com/org/our-repo.git [test:1] at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:797) [test:1] at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1051) [test:1] at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1082) [test:1] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109) [test:1] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83) [test:1] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73) [test:1] at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:52) [test:1] at hudson.security.ACL.impersonate(ACL.java:221) [test:1] at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:49) [test:1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [test:1] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [test:1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [test:1] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [test:1] at java.lang.Thread.run(Thread.java:745) [test:1] Caused by: hudson.plugins.git.GitException: Command "git config remote.origin1.url https://github.com/org/our-repo.git" returned status code 255: [test:1] stdout: [test:1] stderr: error: could not lock config file .git/config: File exists [test:1] [test:1] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1752) [test:1] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1725) [test:1] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1721) [test:1] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1391) [test:1] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1403) [test:1] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1107) [test:1] at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160) [test:1] at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) [test:1] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [test:1] at java.lang.reflect.Method.invoke(Method.java:498) [test:1] at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:887) [test:1] at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:862) [test:1] at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:821) [test:1] at hudson.remoting.UserRequest.perform(UserRequest.java:153) [test:1] at hudson.remoting.UserRequest.perform(UserRequest.java:50) [test:1] at hudson.remoting.Request$2.run(Request.java:332) [test:1] at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) [test:1] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [test:1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [test:1] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [test:1] at hudson.remoting.Engine$1$1.run(Engine.java:88) [test:1] at java.lang.Thread.run(Thread.java:745) [test:1] at ......remote call to kubernetes-f2462ed7839d4e6a833775d67a2b867d-3b5c4c72b81c(Native Method) [test:1] at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1433) [test:1] at hudson.remoting.UserResponse.retrieve(UserRequest.java:253) [test:1] at hudson.remoting.Channel.call(Channel.java:797) [test:1] at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:252) [test:1] at com.sun.proxy.$Proxy108.setRemoteUrl(Unknown Source) [test:1] at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:294) [test:1] at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:785) [test:1] ... 13 more [test:1] Failed in branch test:1
jeanmertz probably an unrelated issue which should be filed separately with detailed steps to reproduce from scratch. Offhand it sounds like you are actually using the same workspace in different branches, but I do not really recognize the error, which is coming from the command-line git tool.
I see the same error at the end of a large parallel execution (44 parallel steps).
This pipeline generates a log file of up to 2 MB.
A single steps looks like this:
node ('windows %some label%') { checkout svn lock ('execution for this node') { lock ('specific part of this execution') { withCredentials execute batch script } } handle errors of batch script }
Error after last parallel step has finished:
[Pipeline] [*lastParallelPart*] // node [Pipeline] [*lastParallelPart*] } [Pipeline] // parallel [Pipeline] End of Pipeline hudson.util.IOException2: Failed to parse /opt/ccde/jenkins_home/jobs/*job*/builds/268/changelog1.xml at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:746) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1500(WorkflowRun.java:125) at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:936) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:123) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:260) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Suppressed: hudson.util.IOException2: Failed to parse /opt/ccde/jenkins_home/jobs/*job*/builds/268/changelog1.xml ... 16 more Caused by: org.xml.sax.SAXParseException; systemId: file:/opt/ccde/jenkins_home/jobs/*job*/builds/268/changelog1.xml; lineNumber: 1; columnNumber: 1; Premature end of file. at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76) ... 15 more Caused by: org.xml.sax.SAXParseException; systemId: file:/opt/ccde/jenkins_home/jobs/*job*/builds/268/changelog1.xml; lineNumber: 1; columnNumber: 1; Premature end of file. at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76) ... 15 more Finished: FAILURE
I added a random delay (0 - 2 seconds) before the svn checkout and at the end of each parallel step.
I also added a Lock (Lockable Resources Plugin) at the end of a parallel step to prevent a parallel finish of multiple parallel steps.
It would be very nice, if this could be fixed.
Aaron Dietz
added a comment - - edited I see the same error at the end of a large parallel execution (44 parallel steps).
This pipeline generates a log file of up to 2 MB.
A single steps looks like this:
node ( 'windows %some label%' ) {
checkout svn
lock ( 'execution for this node' ) {
lock ( 'specific part of this execution' ) {
withCredentials execute batch script
}
}
handle errors of batch script
}
Error after last parallel step has finished:
[Pipeline] [*lastParallelPart*] // node
[Pipeline] [*lastParallelPart*] }
[Pipeline] // parallel
[Pipeline] End of Pipeline
hudson.util.IOException2: Failed to parse /opt/ccde/jenkins_home/jobs /*job*/ builds/268/changelog1.xml
at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80)
at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:746)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1500(WorkflowRun.java:125)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:936)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:123)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
at hudson.security.ACL.impersonate(ACL.java:260)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang. Thread .run( Thread .java:748)
Suppressed: hudson.util.IOException2: Failed to parse /opt/ccde/jenkins_home/jobs /*job*/ builds/268/changelog1.xml
... 16 more
Caused by: org.xml.sax.SAXParseException; systemId: file:/opt/ccde/jenkins_home/jobs /*job*/ builds/268/changelog1.xml; lineNumber: 1; columnNumber: 1; Premature end of file.
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
at org.apache.commons.digester.Digester.parse(Digester.java:1871)
at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76)
... 15 more
Caused by: org.xml.sax.SAXParseException; systemId: file:/opt/ccde/jenkins_home/jobs /*job*/ builds/268/changelog1.xml; lineNumber: 1; columnNumber: 1; Premature end of file.
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
at org.apache.commons.digester.Digester.parse(Digester.java:1871)
at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76)
... 15 more
Finished: FAILURE
I added a random delay (0 - 2 seconds) before the svn checkout and at the end of each parallel step.
I also added a Lock (Lockable Resources Plugin) at the end of a parallel step to prevent a parallel finish of multiple parallel steps.
It would be very nice, if this could be fixed. Just ran into this today when we switched from VM hosted Jenkins to docker container:
org.xml.sax.SAXParseException; Premature end of file. at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76) Also: org.xml.sax.SAXParseException; Premature end of file. at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76) Caused: hudson.util.IOException2: Failed to parse /var/jenkins_home/jobs/*job*/branches/*branch*/builds/2/changelog1.xml Caused: hudson.util.IOException2: Failed to parse /var/jenkins_home/jobs/*job*/branches/*branch*/builds/2/changelog1.xml at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:1031) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1400(WorkflowRun.java:144) at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:1239) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:127) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:290) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Finished: FAILURE
Before today, we had never seen this. Been using Jenkins out of a VM (Ubuntu 12) for 4 years.
We are getting this so frequently now. It has to be about 50% of our builds are failing with:
Failed to parse /var/jenkins_home/jobs/*job*/branches/*branch*/builds/103/changelog1.xml
This is killing us today. So man occurrences. More than 50% of our builds. The recommended "sleep time" work-around did not appear to help when implemented on some branch build testing. We are move it to trunk because it has gotten so bad today, but I'm not hopeful.
Aaron Dietz
added a comment - @Shannon Kerr
Have you also tried the lock method from my approach above?
Acquire a singleton lock "execution end" (or a nicer name 
) at the end of the execution and release it instandly.
This way no execution can end in parallel.
Next escalation step is to add a minimal delay before you release the lock.
Definitely not pretty, but fixes it for me
Aaron Dietz
added a comment - @Shannon Kerr
Have you also tried the lock method from my approach above?
Acquire a singleton lock "execution end" (or a nicer name ) at the end of the execution and release it instandly.
This way no execution can end in parallel.
Next escalation step is to add a minimal delay before you release the lock.
Definitely not pretty, but fixes it for me mixermachine I have not yet. Will definitely try this today. The team I support is really suffering with this issue. I stopped updating Jenkins and our plugins over a month ago (as we approach our product release), but this issue seems to be increasing in frequency, which I cannot explain since I've frozen everything.
mixermachine can you share your implementation of the post checkout lock? I've not used this plugin up to now (lock). Thank you.
edit: Perhaps your implementation is what you wrote in your original comment above. I'm using scripted pipeline at this point and right now I have:
Random random = new Random() sleep time: random.nextInt(2000), unit: 'MILLISECONDS' checkout(.....) lock ("SVN CO end for ${env.NODE_NAME}") { echo "Post-svnco lock" } sleep time: random.nextInt(2000), unit: 'MILLISECONDS'
Is that sufficient?
Aaron Dietz
added a comment - Just try your luck with the lock at first.
lock ("end_lock") { echo "end lock" }
This is the least time consuming way.
If that does not work:
lock ("end_lock") { echo "end lock" sleep time: (50 + random.nextInt(400)) // hopefully enough time }
It might be enough to have one lock for each node (as you did show), but it could also be necessary to synchronize all jobs as some files are written back to the master.
I hope this helps
Greetings from Germany
Aaron
Aaron Dietz
added a comment - kerrhome
Just try your luck with the lock at first.
lock ( "end_lock" ) {
echo "end lock"
}
This is the least time consuming way.
If that does not work:
lock ( "end_lock" ) {
echo "end lock"
sleep time: (50 + random.nextInt(400)) // hopefully enough time
}
It might be enough to have one lock for each node (as you did show), but it could also be necessary to synchronize all jobs as some files are written back to the master.
I hope this helps
Greetings from Germany
Aaron mixermachine Thank you. I've been testing with this all day today and so far so good. Hopefully your solution works until this ticket gets a proper resolution. Thank you again.
Aaron Dietz
added a comment - kerrhome No problem, happy to help mixermachine well, I've had two occurrences of the issue since the new lock implementation. Just to be clear about what we're doing, we are running at least 3 svn co's in parallel via a call to our groovy script. These checkouts are happening on 3 different machines (one Linux physical machine for docker builds and 2 Windows VMs), all 3 checkouts in parallel. I assume when others are doing checkouts that they too are running parallel checkouts on different machines, but thought I'd better clarify.
Are there any other changes you (or others) recommend we try to stop these issues?
David Aldrich
added a comment - Please show me how to implement a lock in a declarative pipeline script.
David Aldrich
added a comment - Please show me how to implement a lock in a declarative pipeline script. Aaron Dietz
added a comment - kerrhome sorry for the late answer, I'm currently busy with my bachelor thesis.
Mh I can only guess...
I think preventing paralle CO is not an option ? (then you could simply create a lock for the CO command)
A very wild guess is to check what hard drives your (main?) server uses. Maybe you can upgrade to a SSD setup so the machine spends less time to write the files out? (not optimal fix for a software problem, I know)
The fastest thing you can currently try is to place locks and delays on variouse places.
I might look into this further after I submited my bachelor thesis (time budget is currently packed).
Hy David
I have used the "normal" jenkins pipeline scripts till now and don't know of a declarative pipeline script (maybe we talk about the same? Groovy scripting language that defines a Jenkins Pipeline)
If we talk about the same, my approach from a few posts up should work for you (hopefully, as this is a temporary fix).
Aaron Dietz
added a comment - kerrhome sorry for the late answer, I'm currently busy with my bachelor thesis.
Mh I can only guess...
I think preventing paralle CO is not an option ? (then you could simply create a lock for the CO command)
A very wild guess is to check what hard drives your (main?) server uses. Maybe you can upgrade to a SSD setup so the machine spends less time to write the files out? (not optimal fix for a software problem, I know)
The fastest thing you can currently try is to place locks and delays on variouse places.
I might look into this further after I submited my bachelor thesis (time budget is currently packed).
davida2009
Hy David
I have used the "normal" jenkins pipeline scripts till now and don't know of a declarative pipeline script (maybe we talk about the same? Groovy scripting language that defines a Jenkins Pipeline)
If we talk about the same, my approach from a few posts up should work for you (hopefully, as this is a temporary fix). AFAIK workflow-scm-step 2.7 should have resolved this (JENKINS-47201), unless you are talking about some unrelated issue.
Aaron Dietz
added a comment - Aaron Dietz
added a comment - jglick that are some good news, thanks for the implementation.
kerrhome fixed for you? jglick I guess you mean Pipeline: SCM Step 2.7? I updated to that plugin in October 2018, but have seen the problem since then.
David Aldrich
added a comment - - edited jglick I guess you mean Pipeline: SCM Step 2.7? I updated to that plugin in October 2018, but have seen the problem since then. the problem
Meaning a ConcurrentModificationException serializing MultiSCMRevisionState#revisionStates, or some other problem?
David Aldrich
added a comment - @jglick Sorry, I don't have a record of the exception now, but it did mention org.xml.sax.SAXParseException.
I seem to have fixed it by adding a lock at the end of each parallel stage.
David Aldrich
added a comment - @jglick Sorry, I don't have a record of the exception now, but it did mention org.xml.sax.SAXParseException.
I seem to have fixed it by adding a lock at the end of each parallel stage. David Aldrich
added a comment - @jglick The exception occurred again for us this weekend:
org.xml.sax.SAXParseException; systemId: file:/var/lib/jenkins/jobs/Zodiac_gcc_all_versions/builds/123/changelog1.xml; lineNumber: 1; columnNumber: 1; Premature end of file. at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76) Caused: hudson.util.IOException2: Failed to parse /var/lib/jenkins/jobs/Zodiac_gcc_all_versions/builds/123/changelog1.xml at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:835) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1100(WorkflowRun.java:133) at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:1100) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:136) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:51) at hudson.security.ACL.impersonate(ACL.java:290) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:48) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
We are running Jenkins 2.150.1.
David Aldrich
added a comment - @jglick The exception occurred again for us this weekend:
org.xml.sax.SAXParseException; systemId: file:/ var /lib/jenkins/jobs/Zodiac_gcc_all_versions/builds/123/changelog1.xml; lineNumber: 1; columnNumber: 1; Premature end of file.
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
at org.apache.commons.digester.Digester.parse(Digester.java:1871)
at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76)
Caused: hudson.util.IOException2: Failed to parse / var /lib/jenkins/jobs/Zodiac_gcc_all_versions/builds/123/changelog1.xml
at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80)
at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:835)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1100(WorkflowRun.java:133)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:1100)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:136)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:51)
at hudson.security.ACL.impersonate(ACL.java:290)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:48)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang. Thread .run( Thread .java:748)
We are running Jenkins 2.150.1. Studying the fix of JENKINS-47201 by dnusbaum I suspect it was incomplete, as this block is not synchronized as a whole, and so two concurrent branches could both pick the same changelog file.
This is probably reported against the Subversion plugin since that is the most widely used SCM plugin to actually treat changelog.xml as an XML file. Despite the name, the format is up to the SCM plugin to interpret, and the Git plugin writes and reads a completely non-XML format. (The Mercurial plugin also uses XML but is probably too little used for people to hit this and report it.)
Studying the fix of
JENKINS-47201by Devin Nusbaum I suspect it was incomplete, as this block is not synchronized as a whole, and so two concurrent branches could both pick the same changelog file.
Looks like it, I think we could reorder some statements in the method so that we could lock Run when looking for the file without needing to lock prev as well, although maybe that code could be refactored to use the temp file APIs or something to be more robust.
See https://github.com/jenkinsci/workflow-scm-step-plugin/pull/32.
David Aldrich
added a comment - The SAXParseException is now occuring repeatedly (every build) for us.
David Aldrich
added a comment - The SAXParseException is now occuring repeatedly (every build) for us. davida2009 I've attached an incremental build of the Pipeline SCM Step Plugin that contains PR #32. Feel free to try it out (preferably in a test environment first) to let us know if it addresses the problem. I am not sure when I will have time to test the code further myself and merge/release that change, so if you are blocked I would recommend testing that build.
David Aldrich
added a comment - Thanks Devin, I will give it a try. dnusbaum I have been running workflow-scm-step-2.8-rc326.7286a182d923.hpi for 3 weeks and have not seen a single instance of the SAXParseException. It has also been running fine with no resource locks for 3 days. So it looks as though PR #32 fixes the problem.
Please will you consider releasing a formal update of the plugin?
David Aldrich
added a comment - - edited dnusbaum I have been running workflow-scm-step-2.8-rc326.7286a182d923.hpi for 3 weeks and have not seen a single instance of the SAXParseException. It has also been running fine with no resource locks for 3 days. So it looks as though PR #32 fixes the problem.
Please will you consider releasing a formal update of the plugin? Hi dnusbaum,
I'm having the same issue for. I'll use workflow-scm-step-2.8-rc326.7286a182d923.hpi to confirm if it's fixing the "ConcurrentModificationException". I'll test it against 100+ parallelized repositories (in groups of 12 concurrent extractions)
Thanks in advance
Benjamin Boyer
added a comment - - edited Hi dnusbaum ,
I'm having the same issue for. I'll use workflow-scm-step-2.8-rc326.7286a182d923.hpi to confirm if it's fixing the "ConcurrentModificationException". I'll test it against 100+ parallelized repositories (in groups of 12 concurrent extractions)
Thanks in advance 
Andre Morales
added a comment - I was having the same issue with a pipeline of 23 parallel checkouts. It was failing randomly.
I have installed plugin workflow-scm-step-2.8-rc326.7286a182d923.hpi and everything goes well now. Since installation, all 23 agents running in parallel are checking out correctly.
Andre Morales
added a comment - I was having the same issue with a pipeline of 23 parallel checkouts. It was failing randomly.
I have installed plugin workflow-scm-step-2.8-rc326.7286a182d923.hpi and everything goes well now. Since installation, all 23 agents running in parallel are checking out correctly.
Is there a chance to get a new release of the plugin with this fix included?
That would be great as this issue is really problematic for larger parallel pipelines.
Torsten Werner
added a comment - Please, release the fix. The issue is really a problem for us. A fix for this issue was just released in Pipeline SCM Step Plugin version 2.8.
The SAXParseException seems like a bug in the Subversion plugin.
The ConcurrentModificationException is
JENKINS-19561.