-
Bug
-
Resolution: Unresolved
-
Major
-
Operating System: Centos 7.2, 64-bit
JDK: 1.8.0
Jenkins: 2.48
Mercurial Plugin: 1.59
Running Jenkins directly
No reverse proxy
installed via yum
issue occurs on the master node
Using parallel nodes and checkout sometimes causes the following error:
hudson.util.IOException2: Failed to parse /var/lib/jenkins/jobs/parallel-test/builds/28/changelog0.xml: '<?xml version="1.0" encoding="UTF-8"?> <changesets> ' at hudson.plugins.mercurial.MercurialChangeLogParser.parse(MercurialChangeLogParser.java:55) at hudson.plugins.mercurial.MercurialChangeLogParser.parse(MercurialChangeLogParser.java:26) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:746) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1500(WorkflowRun.java:125) at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:936) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:123) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:260) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Suppressed: hudson.util.IOException2: Failed to parse /var/lib/jenkins/jobs/parallel-test/builds/28/changelog0.xml: '<?xml version="1.0" encoding="UTF-8"?> <changesets> ' ... 16 more Caused by: org.xml.sax.SAXParseException; systemId: file:/var/lib/jenkins/jobs/parallel-test/builds/28/changelog0.xml; lineNumber: 3; columnNumber: 1; XML document structures must start and end within the same entity. at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.plugins.mercurial.MercurialChangeLogParser.parse(MercurialChangeLogParser.java:51) ... 15 more Caused by: org.xml.sax.SAXParseException; systemId: file:/var/lib/jenkins/jobs/parallel-test/builds/28/changelog0.xml; lineNumber: 3; columnNumber: 1; XML document structures must start and end within the same entity. at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.plugins.mercurial.MercurialChangeLogParser.parse(MercurialChangeLogParser.java:51) ... 15 more Finished: FAILURE
Here is the code that I used to generate that error:
#!groovy try { parallel ( 0: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 1: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 2: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 3: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 4: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 5: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 6: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 7: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 8: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 9: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } } ) } catch (e) { throw e }
However this error happens randomly. In the attachments, I did two identical runs, one passes, the other has multiple failed nodes.
Also I was able to reproduce this error with just two parallel nodes.
My current workaround is to not use parallel nodes, but then jobs run much slower. Does anyone have an alternative workaround or solution? Or how do I even get started trying to debug this?
- relates to
-
-
- Resolved
-
[JENKINS-43176] MercurialChangeLogParser fails in parallel checkouts
| Description |
Original:
Using parallel nodes and checkout sometimes causes the following error: hudson.util.IOException2: Failed to parse /var/lib/jenkins/jobs/parallel-test/builds/28/changelog0.xml: '<?xml version="1.0" encoding="UTF-8"?> <changesets> ' at hudson.plugins.mercurial.MercurialChangeLogParser.parse(MercurialChangeLogParser.java:55) at hudson.plugins.mercurial.MercurialChangeLogParser.parse(MercurialChangeLogParser.java:26) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:746) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1500(WorkflowRun.java:125) at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:936) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:123) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:260) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Suppressed: hudson.util.IOException2: Failed to parse /var/lib/jenkins/jobs/parallel-test/builds/28/changelog0.xml: '<?xml version="1.0" encoding="UTF-8"?> <changesets> ' ... 16 more Caused by: org.xml.sax.SAXParseException; systemId: file:/var/lib/jenkins/jobs/parallel-test/builds/28/changelog0.xml; lineNumber: 3; columnNumber: 1; XML document structures must start and end within the same entity. at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.plugins.mercurial.MercurialChangeLogParser.parse(MercurialChangeLogParser.java:51) ... 15 more Caused by: org.xml.sax.SAXParseException; systemId: file:/var/lib/jenkins/jobs/parallel-test/builds/28/changelog0.xml; lineNumber: 3; columnNumber: 1; XML document structures must start and end within the same entity. at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.plugins.mercurial.MercurialChangeLogParser.parse(MercurialChangeLogParser.java:51) ... 15 more Finished: FAILURE Here is the code that I used to generate that error: {code:java} #!groovy try { parallel ( 0: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 1: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 2: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 3: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 4: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 5: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 6: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 7: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 8: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 9: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } } ) node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } } catch (e) { throw e } {code} However this error happens randomly. In the attachments, I did two identical runs, one passes, the other has multiple failed nodes. Also I was able to reproduce this error with just two parallel nodes. My current workaround is to not use parallel nodes, but then jobs run much slower. Does anyone have an alternative workaround or solution? Or how do I even get started trying to debug this? |
New:
Using parallel nodes and checkout sometimes causes the following error: {code:java} hudson.util.IOException2: Failed to parse /var/lib/jenkins/jobs/parallel-test/builds/28/changelog0.xml: '<?xml version="1.0" encoding="UTF-8"?> <changesets> ' at hudson.plugins.mercurial.MercurialChangeLogParser.parse(MercurialChangeLogParser.java:55) at hudson.plugins.mercurial.MercurialChangeLogParser.parse(MercurialChangeLogParser.java:26) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:746) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1500(WorkflowRun.java:125) at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:936) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:123) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:260) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Suppressed: hudson.util.IOException2: Failed to parse /var/lib/jenkins/jobs/parallel-test/builds/28/changelog0.xml: '<?xml version="1.0" encoding="UTF-8"?> <changesets> ' ... 16 more Caused by: org.xml.sax.SAXParseException; systemId: file:/var/lib/jenkins/jobs/parallel-test/builds/28/changelog0.xml; lineNumber: 3; columnNumber: 1; XML document structures must start and end within the same entity. at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.plugins.mercurial.MercurialChangeLogParser.parse(MercurialChangeLogParser.java:51) ... 15 more Caused by: org.xml.sax.SAXParseException; systemId: file:/var/lib/jenkins/jobs/parallel-test/builds/28/changelog0.xml; lineNumber: 3; columnNumber: 1; XML document structures must start and end within the same entity. at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.plugins.mercurial.MercurialChangeLogParser.parse(MercurialChangeLogParser.java:51) ... 15 more Finished: FAILURE{code} Here is the code that I used to generate that error: {code:java} #!groovy try { parallel ( 0: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 1: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 2: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 3: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 4: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 5: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 6: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 7: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 8: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 9: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } } ) node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } } catch (e) { throw e } {code} However this error happens randomly. In the attachments, I did two identical runs, one passes, the other has multiple failed nodes. Also I was able to reproduce this error with just two parallel nodes. My current workaround is to not use parallel nodes, but then jobs run much slower. Does anyone have an alternative workaround or solution? Or how do I even get started trying to debug this? |
| Description |
Original:
Using parallel nodes and checkout sometimes causes the following error: {code:java} hudson.util.IOException2: Failed to parse /var/lib/jenkins/jobs/parallel-test/builds/28/changelog0.xml: '<?xml version="1.0" encoding="UTF-8"?> <changesets> ' at hudson.plugins.mercurial.MercurialChangeLogParser.parse(MercurialChangeLogParser.java:55) at hudson.plugins.mercurial.MercurialChangeLogParser.parse(MercurialChangeLogParser.java:26) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:746) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1500(WorkflowRun.java:125) at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:936) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:123) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:260) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Suppressed: hudson.util.IOException2: Failed to parse /var/lib/jenkins/jobs/parallel-test/builds/28/changelog0.xml: '<?xml version="1.0" encoding="UTF-8"?> <changesets> ' ... 16 more Caused by: org.xml.sax.SAXParseException; systemId: file:/var/lib/jenkins/jobs/parallel-test/builds/28/changelog0.xml; lineNumber: 3; columnNumber: 1; XML document structures must start and end within the same entity. at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.plugins.mercurial.MercurialChangeLogParser.parse(MercurialChangeLogParser.java:51) ... 15 more Caused by: org.xml.sax.SAXParseException; systemId: file:/var/lib/jenkins/jobs/parallel-test/builds/28/changelog0.xml; lineNumber: 3; columnNumber: 1; XML document structures must start and end within the same entity. at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.plugins.mercurial.MercurialChangeLogParser.parse(MercurialChangeLogParser.java:51) ... 15 more Finished: FAILURE{code} Here is the code that I used to generate that error: {code:java} #!groovy try { parallel ( 0: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 1: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 2: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 3: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 4: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 5: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 6: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 7: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 8: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 9: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } } ) node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } } catch (e) { throw e } {code} However this error happens randomly. In the attachments, I did two identical runs, one passes, the other has multiple failed nodes. Also I was able to reproduce this error with just two parallel nodes. My current workaround is to not use parallel nodes, but then jobs run much slower. Does anyone have an alternative workaround or solution? Or how do I even get started trying to debug this? |
New:
Using parallel nodes and checkout sometimes causes the following error: {code:java} hudson.util.IOException2: Failed to parse /var/lib/jenkins/jobs/parallel-test/builds/28/changelog0.xml: '<?xml version="1.0" encoding="UTF-8"?> <changesets> ' at hudson.plugins.mercurial.MercurialChangeLogParser.parse(MercurialChangeLogParser.java:55) at hudson.plugins.mercurial.MercurialChangeLogParser.parse(MercurialChangeLogParser.java:26) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:746) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1500(WorkflowRun.java:125) at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:936) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:123) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:260) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Suppressed: hudson.util.IOException2: Failed to parse /var/lib/jenkins/jobs/parallel-test/builds/28/changelog0.xml: '<?xml version="1.0" encoding="UTF-8"?> <changesets> ' ... 16 more Caused by: org.xml.sax.SAXParseException; systemId: file:/var/lib/jenkins/jobs/parallel-test/builds/28/changelog0.xml; lineNumber: 3; columnNumber: 1; XML document structures must start and end within the same entity. at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.plugins.mercurial.MercurialChangeLogParser.parse(MercurialChangeLogParser.java:51) ... 15 more Caused by: org.xml.sax.SAXParseException; systemId: file:/var/lib/jenkins/jobs/parallel-test/builds/28/changelog0.xml; lineNumber: 3; columnNumber: 1; XML document structures must start and end within the same entity. at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.plugins.mercurial.MercurialChangeLogParser.parse(MercurialChangeLogParser.java:51) ... 15 more Finished: FAILURE{code} Here is the code that I used to generate that error: {code:java} #!groovy try { parallel ( 0: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 1: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 2: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 3: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 4: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 5: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 6: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 7: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 8: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } }, 9: { node { checkout($class: 'MercurialSCM', source: 'https://bitbucket.org/vicyap/jenkins-parallel-test', clean: true) } } ) } catch (e) { throw e } {code} However this error happens randomly. In the attachments, I did two identical runs, one passes, the other has multiple failed nodes. Also I was able to reproduce this error with just two parallel nodes. My current workaround is to not use parallel nodes, but then jobs run much slower. Does anyone have an alternative workaround or solution? Or how do I even get started trying to debug this? |
| Priority | Original: Minor [ 4 ] | New: Major [ 3 ] |
Adam Paul
added a comment - We are seeing this too and it's failing ~1/3 of our jobs (nearly all of which do parallel checkouts as we build our product on two architectures).
Adam Paul
added a comment - We are seeing this too and it's failing ~1/3 of our jobs (nearly all of which do parallel checkouts as we build our product on two architectures). 
Olivier Sechet
added a comment - Same here. I had to add a sleep with different delay in each branch before the checkout to avoid random failing jobs.
Olivier Sechet
added a comment - Same here. I had to add a sleep with different delay in each branch before the checkout to avoid random failing jobs. Olivier Sechet
added a comment - The problem starting to be blocking for us, we tried to understand what is happening. It seems all the parallel branches write to the same changelog.xml file, at the same time. It leads to a file that cannot be read at some point. Most of the time, the file looks as in the provided log:
'<?xml version="1.0" encoding="UTF-8"?> <changesets> '
But once or twice we got `NUL` (0x00 in hex) characters inside:
'<?xml version="1.0" encoding="UTF-8"?> <changesets> \0\0\0\0\0\0\0\0\0\author='...' rev='9230' date='1499864372.0-7200'><msg>...
Olivier Sechet
added a comment - The problem starting to be blocking for us, we tried to understand what is happening. It seems all the parallel branches write to the same changelog.xml file, at the same time. It leads to a file that cannot be read at some point. Most of the time, the file looks as in the provided log:
'<?xml version= "1.0" encoding= "UTF-8" ?>
<changesets>
'
But once or twice we got `NUL` (0x00 in hex) characters inside:
'<?xml version= "1.0" encoding= "UTF-8" ?>
<changesets>
\0\0\0\0\0\0\0\0\0\author= '...' rev= '9230' date= '1499864372.0-7200' ><msg>...
I suspect this is a race condition in SCMStep.checkout. Should try to atomically create the file before moving on to another choice. Or should use some other means of uniquifying files per build.
| Component/s | New: workflow-scm-step-plugin [ 21717 ] | |
| Component/s | Original: mercurial-plugin [ 15502 ] |
Definitely looks like a bug. No idea about cause offhand. Could be a problem in mercurial, or could be a race condition in workflow-scm-step.