Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34313

SAXParseException from parallel SCM checkout

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Major
    • Resolution: Fixed
    • None
    • Jenkins ver. 1.651.1
      Pipeline plug-ins: ver. 1.15
      Pipeline: SCM Step plug-in: ver. 1.15
      SCM API Plugin: ver 1.1
      Subversion Plug-in: ver. 2.5.6
    • workflow-scm-step 2.8

    Description

      In out pipeline we at first build multiple upstream Maven projects and then build the main project. Upstream projects are build in parallel:

      parallel(
          projA: {
              projects.buildDependency('projA')
          },
          projB: {
              projects.buildDependency('projB')
          }
      ...
      )
      

      projects.buildDependency() just checks-out each project from SVN and starts Maven build. The problem is that we often (in about 20% of builds) have errors caused by concurrent / parallel SVN check-outs. Sometimes it is ConcurrentModificationException and sometimes "SAXParseException: changelog11.xml; lineNumber: 1; columnNumber: 1; Premature end of file". See stacktraces below.

      As a temporary workaround we added

      sleep time: random.nextInt(1000), unit: 'MILLISECONDS'
      

      just before the checkout step to introduce time shifts between check-outs. We used random numbers because we were not sure if some local or env counter variable would be thread-safe in case of parallel steps.

      Now error happens only if time difference between check-outs is narrow, e.g. 5 milliseconds if 2 generated random numbers are close.

      Would it be possible to make Pipeline plug-in more resilient in case of parallel check-outs?

      Caused by: org.jenkinsci.plugins.workflow.cps.steps.ParallelStepException: Parallel step projA failed
      	... 26 more
      Caused by: hudson.util.IOException2: Failed to parse /var/lib/jenkins/jobs/ha-pipeline/builds/426/changelog11.xml
      	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80)
      	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:638)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1200(WorkflowRun.java:111)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:812)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:123)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
      	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:49)
      	at hudson.security.ACL.impersonate(ACL.java:213)
      	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:47)
      	... 5 more
      Caused by: org.xml.sax.SAXParseException; systemId: file:/var/lib/jenkins/jobs/ha-pipeline/builds/426/changelog11.xml; lineNumber: 1; columnNumber: 1; Premature end of file.
      	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
      	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
      	at org.apache.commons.digester.Digester.parse(Digester.java:1871)
      	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76)
      	... 15 more
      
      org.jenkinsci.plugins.workflow.cps.steps.ParallelStepException: Parallel step projB failed
      	at org.jenkinsci.plugins.workflow.cps.steps.ParallelStep$ResultHandler$Callback.checkAllDone(ParallelStep.java:153)
      	at org.jenkinsci.plugins.workflow.cps.steps.ParallelStep$ResultHandler$Callback.onSuccess(ParallelStep.java:110)
      	at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:311)
      	at com.cloudbees.groovy.cps.impl.LocalVariableBlock$LocalVariable.get(LocalVariableBlock.java:33)
      	at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
      	at com.cloudbees.groovy.cps.impl.LocalVariableBlock.evalLValue(LocalVariableBlock.java:22)
      	at com.cloudbees.groovy.cps.LValueBlock$BlockImpl.eval(LValueBlock.java:55)
      	at com.cloudbees.groovy.cps.LValueBlock.eval(LValueBlock.java:16)
      	at com.cloudbees.groovy.cps.Next.step(Next.java:58)
      	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:106)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
      	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:164)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:277)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:77)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:186)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:184)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      Caused by: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:215)
      	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:183)
      	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
      	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:168)
      	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:109)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
      	at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
      	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
      	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
      	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
      	at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
      	at hudson.XmlFile.write(XmlFile.java:178)
      	at hudson.model.Run.save(Run.java:1896)
      	at hudson.util.PersistedList.onModified(PersistedList.java:173)
      	at hudson.util.PersistedList._onModified(PersistedList.java:181)
      	at hudson.util.PersistedList.add(PersistedList.java:72)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:651)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1200(WorkflowRun.java:111)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:812)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:123)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
      	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:49)
      	at hudson.security.ACL.impersonate(ACL.java:213)
      	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:47)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState#revisionStates for class org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:215)
      	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:183)
      	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
      	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:168)
      	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:109)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88)
      	at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64)
      	at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
      	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:224)
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:211)
      	... 31 more
      Caused by: java.util.ConcurrentModificationException
      	at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922)
      	at java.util.HashMap$EntryIterator.next(HashMap.java:962)
      	at java.util.HashMap$EntryIterator.next(HashMap.java:960)
      	at com.thoughtworks.xstream.converters.collections.MapConverter.marshal(MapConverter.java:75)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
      	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:224)
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:211)
      	... 46 more
      

      Attachments

        1. stable.PNG
          203 kB
          Andre Morales
        2. workflow-scm-step-2.8-rc326.7286a182d923.hpi
          15 kB
          Devin Nusbaum

        Issue Links

          Activity

            benjboyer Benjamin Boyer added a comment - - edited

            Hi dnusbaum,

            I'm having the same issue for. I'll use workflow-scm-step-2.8-rc326.7286a182d923.hpi to confirm if it's fixing the "ConcurrentModificationException". I'll test it against 100+ parallelized repositories (in groups of 12 concurrent extractions)

            Thanks in advance

            benjboyer Benjamin Boyer added a comment - - edited Hi dnusbaum , I'm having the same issue for. I'll use workflow-scm-step-2.8-rc326.7286a182d923.hpi to confirm if it's fixing the "ConcurrentModificationException". I'll test it against 100+ parallelized repositories (in groups of 12 concurrent extractions) Thanks in advance
            skojenkins Andre Morales added a comment -

            I was having the same issue with a pipeline of 23 parallel  checkouts. It was failing randomly. 

            I have installed plugin workflow-scm-step-2.8-rc326.7286a182d923.hpi and everything goes well now.  Since installation, all 23 agents running in parallel are checking out correctly.

             

            skojenkins Andre Morales added a comment - I was having the same issue with a pipeline of 23 parallel  checkouts. It was failing randomly.  I have installed plugin  workflow-scm-step-2.8-rc326.7286a182d923.hpi and everything goes well now.  Since installation, all 23 agents running in parallel are checking out correctly.  

            Is there a chance to get a new release of the plugin with this fix included?
            That would be great as this issue is really problematic for larger parallel pipelines.

            thefifo Wolfgang Issovits added a comment - Is there a chance to get a new release of the plugin with this fix included? That would be great as this issue is really problematic for larger parallel pipelines.

            Please, release the fix. The issue is really a problem for us.

            twerner Torsten Werner added a comment - Please, release the fix. The issue is really a problem for us.
            dnusbaum Devin Nusbaum added a comment -

            A fix for this issue was just released in Pipeline SCM Step Plugin version 2.8.

            dnusbaum Devin Nusbaum added a comment - A fix for this issue was just released in Pipeline SCM Step Plugin version 2.8.

            People

              dnusbaum Devin Nusbaum
              gitt Slawa Giterman
              Votes:
              18 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: