Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34313

SAXParseException from parallel SCM checkout

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Major
    • Resolution: Fixed
    • None
    • Jenkins ver. 1.651.1
      Pipeline plug-ins: ver. 1.15
      Pipeline: SCM Step plug-in: ver. 1.15
      SCM API Plugin: ver 1.1
      Subversion Plug-in: ver. 2.5.6
    • workflow-scm-step 2.8

    Description

      In out pipeline we at first build multiple upstream Maven projects and then build the main project. Upstream projects are build in parallel:

      parallel(
          projA: {
              projects.buildDependency('projA')
          },
          projB: {
              projects.buildDependency('projB')
          }
      ...
      )
      

      projects.buildDependency() just checks-out each project from SVN and starts Maven build. The problem is that we often (in about 20% of builds) have errors caused by concurrent / parallel SVN check-outs. Sometimes it is ConcurrentModificationException and sometimes "SAXParseException: changelog11.xml; lineNumber: 1; columnNumber: 1; Premature end of file". See stacktraces below.

      As a temporary workaround we added

      sleep time: random.nextInt(1000), unit: 'MILLISECONDS'
      

      just before the checkout step to introduce time shifts between check-outs. We used random numbers because we were not sure if some local or env counter variable would be thread-safe in case of parallel steps.

      Now error happens only if time difference between check-outs is narrow, e.g. 5 milliseconds if 2 generated random numbers are close.

      Would it be possible to make Pipeline plug-in more resilient in case of parallel check-outs?

      Caused by: org.jenkinsci.plugins.workflow.cps.steps.ParallelStepException: Parallel step projA failed
      	... 26 more
      Caused by: hudson.util.IOException2: Failed to parse /var/lib/jenkins/jobs/ha-pipeline/builds/426/changelog11.xml
      	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80)
      	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:638)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1200(WorkflowRun.java:111)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:812)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:123)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
      	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:49)
      	at hudson.security.ACL.impersonate(ACL.java:213)
      	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:47)
      	... 5 more
      Caused by: org.xml.sax.SAXParseException; systemId: file:/var/lib/jenkins/jobs/ha-pipeline/builds/426/changelog11.xml; lineNumber: 1; columnNumber: 1; Premature end of file.
      	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
      	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
      	at org.apache.commons.digester.Digester.parse(Digester.java:1871)
      	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76)
      	... 15 more
      
      org.jenkinsci.plugins.workflow.cps.steps.ParallelStepException: Parallel step projB failed
      	at org.jenkinsci.plugins.workflow.cps.steps.ParallelStep$ResultHandler$Callback.checkAllDone(ParallelStep.java:153)
      	at org.jenkinsci.plugins.workflow.cps.steps.ParallelStep$ResultHandler$Callback.onSuccess(ParallelStep.java:110)
      	at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:311)
      	at com.cloudbees.groovy.cps.impl.LocalVariableBlock$LocalVariable.get(LocalVariableBlock.java:33)
      	at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
      	at com.cloudbees.groovy.cps.impl.LocalVariableBlock.evalLValue(LocalVariableBlock.java:22)
      	at com.cloudbees.groovy.cps.LValueBlock$BlockImpl.eval(LValueBlock.java:55)
      	at com.cloudbees.groovy.cps.LValueBlock.eval(LValueBlock.java:16)
      	at com.cloudbees.groovy.cps.Next.step(Next.java:58)
      	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:106)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
      	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:164)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:277)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:77)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:186)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:184)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      Caused by: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:215)
      	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:183)
      	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
      	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:168)
      	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:109)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
      	at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
      	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
      	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
      	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
      	at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
      	at hudson.XmlFile.write(XmlFile.java:178)
      	at hudson.model.Run.save(Run.java:1896)
      	at hudson.util.PersistedList.onModified(PersistedList.java:173)
      	at hudson.util.PersistedList._onModified(PersistedList.java:181)
      	at hudson.util.PersistedList.add(PersistedList.java:72)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:651)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1200(WorkflowRun.java:111)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:812)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:123)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
      	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:49)
      	at hudson.security.ACL.impersonate(ACL.java:213)
      	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:47)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState#revisionStates for class org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:215)
      	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:183)
      	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
      	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:168)
      	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:109)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88)
      	at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64)
      	at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
      	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:224)
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:211)
      	... 31 more
      Caused by: java.util.ConcurrentModificationException
      	at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922)
      	at java.util.HashMap$EntryIterator.next(HashMap.java:962)
      	at java.util.HashMap$EntryIterator.next(HashMap.java:960)
      	at com.thoughtworks.xstream.converters.collections.MapConverter.marshal(MapConverter.java:75)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
      	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:224)
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:211)
      	... 46 more
      

      Attachments

        Issue Links

          Activity

            gitt Slawa Giterman created issue -
            rtyler R. Tyler Croy made changes -
            Field Original Value New Value
            Workflow JNJira [ 170393 ] JNJira + In-Review [ 183882 ]
            jglick Jesse Glick added a comment -

            The SAXParseException seems like a bug in the Subversion plugin.

            The ConcurrentModificationException is JENKINS-19561.

            jglick Jesse Glick added a comment - The SAXParseException seems like a bug in the Subversion plugin. The ConcurrentModificationException is JENKINS-19561 .
            jglick Jesse Glick made changes -
            Component/s subversion-plugin [ 15485 ]
            Component/s workflow-plugin [ 18820 ]
            Assignee Jesse Glick [ jglick ]
            Labels workflow
            abayer Andrew Bayer made changes -
            Labels workflow pipeline workflow
            abayer Andrew Bayer made changes -
            Labels pipeline workflow pipeline
            jeanmertz Jean Mertz added a comment -

            I believe we are seeing the same problem, using Git.

            The below error happens when running tests in parallel, on different nodes, and each node triggering `checkout(scm)` using Jenkins Pipeline workflow.

            [test:1]  > git rev-parse --is-inside-work-tree # timeout=10
            [test:1] Fetching changes from 2 remote Git repositories
            [test:1]  > git config remote.origin.url https://github.com/org/our-repo.git # timeout=10
            [test:1] Fetching upstream changes from https://github.com/org/our-repo.git
            [test:1]  > git --version # timeout=10
            [test:1] using GIT_ASKPASS to set credentials github login credentials for "ci" account
            [test:1]  > git fetch --tags --progress https://github.com/org/our-repo.git +refs/heads/*:refs/remotes/origin/*
            [test:1]  > git config remote.origin1.url https://github.com/org/our-repo.git # timeout=10
            [test:1] ERROR: Error fetching remote repo 'origin1'
            [test:1] hudson.plugins.git.GitException: Failed to fetch from https://github.com/org/our-repo.git
            [test:1]   at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:797)
            [test:1]   at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1051)
            [test:1]   at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1082)
            [test:1]   at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109)
            [test:1]   at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
            [test:1]   at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
            [test:1]   at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:52)
            [test:1]   at hudson.security.ACL.impersonate(ACL.java:221)
            [test:1]   at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:49)
            [test:1]   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            [test:1]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            [test:1]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            [test:1]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            [test:1]   at java.lang.Thread.run(Thread.java:745)
            [test:1] Caused by: hudson.plugins.git.GitException: Command "git config remote.origin1.url https://github.com/org/our-repo.git" returned status code 255:
            [test:1] stdout: 
            [test:1] stderr: error: could not lock config file .git/config: File exists
            [test:1] 
            [test:1]   at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1752)
            [test:1]   at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1725)
            [test:1]   at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1721)
            [test:1]   at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1391)
            [test:1]   at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1403)
            [test:1]   at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1107)
            [test:1]   at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
            [test:1]   at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
            [test:1]   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            [test:1]   at java.lang.reflect.Method.invoke(Method.java:498)
            [test:1]   at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:887)
            [test:1]   at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:862)
            [test:1]   at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:821)
            [test:1]   at hudson.remoting.UserRequest.perform(UserRequest.java:153)
            [test:1]   at hudson.remoting.UserRequest.perform(UserRequest.java:50)
            [test:1]   at hudson.remoting.Request$2.run(Request.java:332)
            [test:1]   at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
            [test:1]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            [test:1]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            [test:1]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            [test:1]   at hudson.remoting.Engine$1$1.run(Engine.java:88)
            [test:1]   at java.lang.Thread.run(Thread.java:745)
            [test:1]   at ......remote call to kubernetes-f2462ed7839d4e6a833775d67a2b867d-3b5c4c72b81c(Native Method)
            [test:1]   at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1433)
            [test:1]   at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
            [test:1]   at hudson.remoting.Channel.call(Channel.java:797)
            [test:1]   at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:252)
            [test:1]   at com.sun.proxy.$Proxy108.setRemoteUrl(Unknown Source)
            [test:1]   at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:294)
            [test:1]   at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:785)
            [test:1]   ... 13 more
            [test:1] Failed in branch test:1
            
            jeanmertz Jean Mertz added a comment - I believe we are seeing the same problem, using Git. The below error happens when running tests in parallel, on different nodes, and each node triggering `checkout(scm)` using Jenkins Pipeline workflow. [test:1] > git rev-parse --is-inside-work-tree # timeout=10 [test:1] Fetching changes from 2 remote Git repositories [test:1] > git config remote.origin.url https://github.com/org/our-repo.git # timeout=10 [test:1] Fetching upstream changes from https://github.com/org/our-repo.git [test:1] > git --version # timeout=10 [test:1] using GIT_ASKPASS to set credentials github login credentials for "ci" account [test:1] > git fetch --tags --progress https://github.com/org/our-repo.git +refs/heads/*:refs/remotes/origin/* [test:1] > git config remote.origin1.url https://github.com/org/our-repo.git # timeout=10 [test:1] ERROR: Error fetching remote repo 'origin1' [test:1] hudson.plugins.git.GitException: Failed to fetch from https://github.com/org/our-repo.git [test:1] at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:797) [test:1] at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1051) [test:1] at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1082) [test:1] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109) [test:1] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83) [test:1] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73) [test:1] at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:52) [test:1] at hudson.security.ACL.impersonate(ACL.java:221) [test:1] at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:49) [test:1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [test:1] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [test:1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [test:1] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [test:1] at java.lang.Thread.run(Thread.java:745) [test:1] Caused by: hudson.plugins.git.GitException: Command "git config remote.origin1.url https://github.com/org/our-repo.git" returned status code 255: [test:1] stdout: [test:1] stderr: error: could not lock config file .git/config: File exists [test:1] [test:1] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1752) [test:1] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1725) [test:1] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1721) [test:1] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1391) [test:1] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1403) [test:1] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1107) [test:1] at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160) [test:1] at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) [test:1] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [test:1] at java.lang.reflect.Method.invoke(Method.java:498) [test:1] at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:887) [test:1] at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:862) [test:1] at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:821) [test:1] at hudson.remoting.UserRequest.perform(UserRequest.java:153) [test:1] at hudson.remoting.UserRequest.perform(UserRequest.java:50) [test:1] at hudson.remoting.Request$2.run(Request.java:332) [test:1] at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) [test:1] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [test:1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [test:1] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [test:1] at hudson.remoting.Engine$1$1.run(Engine.java:88) [test:1] at java.lang.Thread.run(Thread.java:745) [test:1] at ......remote call to kubernetes-f2462ed7839d4e6a833775d67a2b867d-3b5c4c72b81c(Native Method) [test:1] at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1433) [test:1] at hudson.remoting.UserResponse.retrieve(UserRequest.java:253) [test:1] at hudson.remoting.Channel.call(Channel.java:797) [test:1] at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:252) [test:1] at com.sun.proxy.$Proxy108.setRemoteUrl(Unknown Source) [test:1] at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:294) [test:1] at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:785) [test:1] ... 13 more [test:1] Failed in branch test:1
            jglick Jesse Glick added a comment -

            jeanmertz probably an unrelated issue which should be filed separately with detailed steps to reproduce from scratch. Offhand it sounds like you are actually using the same workspace in different branches, but I do not really recognize the error, which is coming from the command-line git tool.

            jglick Jesse Glick added a comment - jeanmertz probably an unrelated issue which should be filed separately with detailed steps to reproduce from scratch. Offhand it sounds like you are actually using the same workspace in different branches, but I do not really recognize the error, which is coming from the command-line git tool.
            mixermachine Aaron Dietz added a comment - - edited

            I see the same error at the end of a large parallel execution (44 parallel steps).

            This pipeline generates a log file of up to 2 MB.

            A single steps looks like this:

             

            node ('windows %some label%') {
             checkout svn
            
             lock ('execution for this node') {
              lock ('specific part of this execution') {
                withCredentials execute batch script
               }  
             }
            
             handle errors of batch script
            }
            

             

            Error after last parallel step has finished: 

            [Pipeline] [*lastParallelPart*] // node
            [Pipeline] [*lastParallelPart*] }
            [Pipeline] // parallel
            [Pipeline] End of Pipeline
            hudson.util.IOException2: Failed to parse /opt/ccde/jenkins_home/jobs/*job*/builds/268/changelog1.xml
            	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80)
            	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:746)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1500(WorkflowRun.java:125)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:936)
            	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:123)
            	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
            	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
            	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
            	at hudson.security.ACL.impersonate(ACL.java:260)
            	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
            	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            	at java.lang.Thread.run(Thread.java:748)
            	Suppressed: hudson.util.IOException2: Failed to parse /opt/ccde/jenkins_home/jobs/*job*/builds/268/changelog1.xml
            		... 16 more
            	Caused by: org.xml.sax.SAXParseException; systemId: file:/opt/ccde/jenkins_home/jobs/*job*/builds/268/changelog1.xml; lineNumber: 1; columnNumber: 1; Premature end of file.
            		at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
            		at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
            		at org.apache.commons.digester.Digester.parse(Digester.java:1871)
            		at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76)
            		... 15 more
            Caused by: org.xml.sax.SAXParseException; systemId: file:/opt/ccde/jenkins_home/jobs/*job*/builds/268/changelog1.xml; lineNumber: 1; columnNumber: 1; Premature end of file.
            	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
            	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
            	at org.apache.commons.digester.Digester.parse(Digester.java:1871)
            	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76)
            	... 15 more
            Finished: FAILURE
            

             

             

            I added a random delay (0 - 2 seconds) before the svn checkout and at the end of each parallel step.

            I also added a Lock (Lockable Resources Plugin) at the end of a parallel step to prevent a parallel finish of multiple parallel steps.

            It would be very nice, if this could be fixed. 

            mixermachine Aaron Dietz added a comment - - edited I see the same error at the end of a large parallel execution (44 parallel steps). This pipeline generates a log file of up to 2 MB. A single steps looks like this:   node ( 'windows %some label%' ) {  checkout svn lock ( 'execution for this node' ) { lock ( 'specific part of this execution' ) { withCredentials execute batch script } } handle errors of batch script }   Error after last parallel step has finished:  [Pipeline] [*lastParallelPart*] // node [Pipeline] [*lastParallelPart*] } [Pipeline] // parallel [Pipeline] End of Pipeline hudson.util.IOException2: Failed to parse /opt/ccde/jenkins_home/jobs /*job*/ builds/268/changelog1.xml at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:746) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1500(WorkflowRun.java:125) at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:936) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:123) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:260) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:748) Suppressed: hudson.util.IOException2: Failed to parse /opt/ccde/jenkins_home/jobs /*job*/ builds/268/changelog1.xml ... 16 more Caused by: org.xml.sax.SAXParseException; systemId: file:/opt/ccde/jenkins_home/jobs /*job*/ builds/268/changelog1.xml; lineNumber: 1; columnNumber: 1; Premature end of file. at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76) ... 15 more Caused by: org.xml.sax.SAXParseException; systemId: file:/opt/ccde/jenkins_home/jobs /*job*/ builds/268/changelog1.xml; lineNumber: 1; columnNumber: 1; Premature end of file. at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76) ... 15 more Finished: FAILURE     I added a random delay (0 - 2 seconds) before the svn checkout and at the end of each parallel step. I also added a Lock (Lockable Resources Plugin) at the end of a parallel step to prevent a parallel finish of multiple parallel steps. It would be very nice, if this could be fixed. 
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-47201 [ JENKINS-47201 ]
            kerrhome Shannon Kerr added a comment -

            Just ran into this today when we switched from VM hosted Jenkins to docker container:

            org.xml.sax.SAXParseException; Premature end of file.
             at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
             at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
             at org.apache.commons.digester.Digester.parse(Digester.java:1871)
             at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76)
            Also: org.xml.sax.SAXParseException; Premature end of file.
             at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
             at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
             at org.apache.commons.digester.Digester.parse(Digester.java:1871)
             at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76)
             Caused: hudson.util.IOException2: Failed to parse /var/jenkins_home/jobs/*job*/branches/*branch*/builds/2/changelog1.xml
            Caused: hudson.util.IOException2: Failed to parse /var/jenkins_home/jobs/*job*/branches/*branch*/builds/2/changelog1.xml
             at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80)
             at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:1031)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1400(WorkflowRun.java:144)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:1239)
             at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:127)
             at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
             at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
             at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
             at hudson.security.ACL.impersonate(ACL.java:290)
             at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
             at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
             at java.util.concurrent.FutureTask.run(FutureTask.java:266)
             at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
             at java.lang.Thread.run(Thread.java:748)
            Finished: FAILURE

             

            Before today, we had never seen this.  Been using Jenkins out of a VM (Ubuntu 12) for 4 years.

             

            kerrhome Shannon Kerr added a comment - Just ran into this today when we switched from VM hosted Jenkins to docker container: org.xml.sax.SAXParseException; Premature end of file. at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76) Also: org.xml.sax.SAXParseException; Premature end of file. at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76) Caused: hudson.util.IOException2: Failed to parse / var /jenkins_home/jobs /*job*/ branches /*branch*/ builds/2/changelog1.xml Caused: hudson.util.IOException2: Failed to parse / var /jenkins_home/jobs /*job*/ branches /*branch*/ builds/2/changelog1.xml at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:1031) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1400(WorkflowRun.java:144) at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:1239) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:127) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:290) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748) Finished: FAILURE   Before today, we had never seen this.  Been using Jenkins out of a VM (Ubuntu 12) for 4 years.  
            kerrhome Shannon Kerr added a comment -

            We are getting this so frequently now.  It has to be about 50% of our builds are failing with:

            Failed to parse /var/jenkins_home/jobs/*job*/branches/*branch*/builds/103/changelog1.xml
            kerrhome Shannon Kerr added a comment - We are getting this so frequently now.  It has to be about 50% of our builds are failing with: Failed to parse / var /jenkins_home/jobs /*job*/ branches /*branch*/ builds/103/changelog1.xml
            kerrhome Shannon Kerr added a comment -

            This is killing us today.  So man occurrences.  More than 50% of our builds. The recommended "sleep time" work-around did not appear to help when implemented on some branch build testing.  We are move it to trunk because it has gotten so bad today, but I'm not hopeful.

             

            kerrhome Shannon Kerr added a comment - This is killing us today.  So man occurrences.  More than 50% of our builds. The recommended "sleep time" work-around did not appear to help when implemented on some branch build testing.  We are move it to trunk because it has gotten so bad today, but I'm not hopeful.  
            mixermachine Aaron Dietz added a comment -

            @Shannon Kerr

            Have you also tried the lock method from my approach above?
            Acquire a singleton lock "execution end" (or a nicer name ) at the end of the execution and release it instandly.
            This way no execution can end in parallel.
            Next escalation step is to add a minimal delay before you release the lock.

            Definitely not pretty, but fixes it for me

            mixermachine Aaron Dietz added a comment - @Shannon Kerr Have you also tried the lock method from my approach above? Acquire a singleton lock "execution end" (or a nicer name ) at the end of the execution and release it instandly. This way no execution can end in parallel. Next escalation step is to add a minimal delay before you release the lock. Definitely not pretty, but fixes it for me
            kerrhome Shannon Kerr added a comment -

            mixermachine I have not yet. Will definitely try this today. The team I support is really suffering with this issue.  I stopped updating Jenkins and our plugins over a month ago (as we approach our product release), but this issue seems to be increasing in frequency, which I cannot explain since I've frozen everything.

             

            kerrhome Shannon Kerr added a comment - mixermachine I have not yet. Will definitely try this today. The team I support is really suffering with this issue.  I stopped updating Jenkins and our plugins over a month ago (as we approach our product release), but this issue seems to be increasing in frequency, which I cannot explain since I've frozen everything.  
            kerrhome Shannon Kerr added a comment - - edited

            mixermachine can you share your implementation of the post checkout lock?  I've not used this plugin up to now (lock).  Thank you.

             

            edit: Perhaps your implementation is what you wrote in your original comment above.  I'm using scripted pipeline at this point and right now I have:

             

            Random random = new Random()
            sleep time: random.nextInt(2000), unit: 'MILLISECONDS'
            checkout(.....)
            lock ("SVN CO end for ${env.NODE_NAME}") {
                echo "Post-svnco lock"
            }
            sleep time: random.nextInt(2000), unit: 'MILLISECONDS'
            

            Is that sufficient?

             

            kerrhome Shannon Kerr added a comment - - edited mixermachine can you share your implementation of the post checkout lock?  I've not used this plugin up to now (lock).  Thank you.   edit: Perhaps your implementation is what you wrote in your original comment above.  I'm using scripted pipeline at this point and right now I have:   Random random = new Random() sleep time: random.nextInt(2000), unit: 'MILLISECONDS' checkout(.....) lock ( "SVN CO end for ${env.NODE_NAME}" ) {     echo "Post-svnco lock" } sleep time: random.nextInt(2000), unit: 'MILLISECONDS' Is that sufficient?  
            mixermachine Aaron Dietz added a comment -

            kerrhome

            Just try your luck with the lock at first.

            lock ("end_lock") {
                echo "end lock"
            }
            

            This is the least time consuming way.

             

            If that does not work:

            lock ("end_lock") {
                echo "end lock"
                sleep time: (50 + random.nextInt(400)) // hopefully enough time
            }
            

            It might be enough to have one lock for each node (as you did show), but it could also be necessary to synchronize all jobs as some files are written back to the master.

             

            I hope this helps

             

            Greetings from Germany

            Aaron

            mixermachine Aaron Dietz added a comment - kerrhome Just try your luck with the lock at first. lock ( "end_lock" ) { echo "end lock" } This is the least time consuming way.   If that does not work: lock ( "end_lock" ) { echo "end lock" sleep time: (50 + random.nextInt(400)) // hopefully enough time } It might be enough to have one lock for each node (as you did show), but it could also be necessary to synchronize all jobs as some files are written back to the master.   I hope this helps   Greetings from Germany Aaron
            kerrhome Shannon Kerr added a comment -

            mixermachine Thank you.  I've been testing with this all day today and so far so good.  Hopefully your solution works until this ticket gets a proper resolution. Thank you again.

            kerrhome Shannon Kerr added a comment - mixermachine Thank you.  I've been testing with this all day today and so far so good.  Hopefully your solution works until this ticket gets a proper resolution. Thank you again.
            mixermachine Aaron Dietz added a comment -

            kerrhome No problem, happy to help

            mixermachine Aaron Dietz added a comment - kerrhome No problem, happy to help
            kerrhome Shannon Kerr added a comment -

            mixermachine well, I've had two occurrences of the issue since the new lock implementation.  Just to be clear about what we're doing, we are running at least 3 svn co's in parallel via a call to our groovy script.  These checkouts are happening on 3 different machines (one Linux physical machine for docker builds and 2 Windows VMs), all 3 checkouts in parallel.  I assume when others are doing checkouts that they too are running parallel checkouts on different machines, but thought I'd better clarify.

            Are there any other changes you (or others) recommend we try to stop these issues?

            kerrhome Shannon Kerr added a comment - mixermachine well, I've had two occurrences of the issue since the new lock implementation.  Just to be clear about what we're doing, we are running at least 3 svn co's in parallel via a call to our groovy script.  These checkouts are happening on 3 different machines (one Linux physical machine for docker builds and 2 Windows VMs), all 3 checkouts in parallel.  I assume when others are doing checkouts that they too are running parallel checkouts on different machines, but thought I'd better clarify. Are there any other changes you (or others) recommend we try to stop these issues?
            davida2009 David Aldrich added a comment -

            Please show me how to implement a lock in a declarative pipeline script.

            davida2009 David Aldrich added a comment - Please show me how to implement a lock in a declarative pipeline script.
            mixermachine Aaron Dietz added a comment -

            kerrhome sorry for the late answer, I'm currently busy with my bachelor thesis.

            Mh I can only guess...

            I think preventing paralle CO is not an option ? (then you could simply create a lock for the CO command)

            A very wild guess is to check what hard drives your (main?) server uses. Maybe you can upgrade to a SSD setup so the machine spends less time to write the files out? (not optimal fix for a software problem, I know)

            The fastest thing you can currently try is to place locks and delays on variouse places.

            I might look into this further after I submited my bachelor thesis (time budget is currently packed).

             

            davida2009

            Hy David

            I have used the "normal" jenkins pipeline scripts till now and don't know of a declarative pipeline script (maybe we talk about the same? Groovy scripting language that defines a Jenkins Pipeline)

            If we talk about the same, my approach from a few posts up should work for you (hopefully, as this is a temporary fix).

            mixermachine Aaron Dietz added a comment - kerrhome sorry for the late answer, I'm currently busy with my bachelor thesis. Mh I can only guess... I think preventing paralle CO is not an option ? (then you could simply create a lock for the CO command) A very wild guess is to check what hard drives your (main?) server uses. Maybe you can upgrade to a SSD setup so the machine spends less time to write the files out? (not optimal fix for a software problem, I know) The fastest thing you can currently try is to place locks and delays on variouse places. I might look into this further after I submited my bachelor thesis (time budget is currently packed).   davida2009 Hy David I have used the "normal" jenkins pipeline scripts till now and don't know of a declarative pipeline script (maybe we talk about the same? Groovy scripting language that defines a Jenkins Pipeline) If we talk about the same, my approach from a few posts up should work for you (hopefully, as this is a temporary fix).
            jglick Jesse Glick added a comment -

            AFAIK workflow-scm-step 2.7 should have resolved this (JENKINS-47201), unless you are talking about some unrelated issue.

            jglick Jesse Glick added a comment - AFAIK workflow-scm-step 2.7 should have resolved this ( JENKINS-47201 ), unless you are talking about some unrelated issue.
            mixermachine Aaron Dietz added a comment -

            jglick that are some good news, thanks for the implementation.

            kerrhome fixed for you?

            mixermachine Aaron Dietz added a comment - jglick that are some good news, thanks for the implementation. kerrhome fixed for you?
            davida2009 David Aldrich added a comment - - edited

            jglick I guess you mean Pipeline: SCM Step 2.7? I updated to that plugin in October 2018, but have seen the problem since then.

            davida2009 David Aldrich added a comment - - edited jglick I guess you mean Pipeline: SCM Step 2.7? I updated to that plugin in October 2018, but have seen the problem since then.
            jglick Jesse Glick added a comment -

            the problem

            Meaning a ConcurrentModificationException serializing MultiSCMRevisionState#revisionStates, or some other problem?

            jglick Jesse Glick added a comment - the problem Meaning a ConcurrentModificationException serializing MultiSCMRevisionState#revisionStates , or some other problem?
            davida2009 David Aldrich added a comment -

            @jglick Sorry, I don't have a record of the exception now, but it did mention org.xml.sax.SAXParseException.

            I seem to have fixed it by adding a lock at the end of each parallel stage.

            davida2009 David Aldrich added a comment - @jglick Sorry, I don't have a record of the exception now, but it did mention org.xml.sax.SAXParseException. I seem to have fixed it by adding a lock at the end of each parallel stage.
            abayer Andrew Bayer made changes -
            Link This issue relates to JENKINS-53140 [ JENKINS-53140 ]
            davida2009 David Aldrich added a comment -

            @jglick The exception occurred again for us this weekend:

            org.xml.sax.SAXParseException; systemId: file:/var/lib/jenkins/jobs/Zodiac_gcc_all_versions/builds/123/changelog1.xml; lineNumber: 1; columnNumber: 1; Premature end of file.
            	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
            	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
            	at org.apache.commons.digester.Digester.parse(Digester.java:1871)
            	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76)
            Caused: hudson.util.IOException2: Failed to parse /var/lib/jenkins/jobs/Zodiac_gcc_all_versions/builds/123/changelog1.xml
            	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80)
            	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:835)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1100(WorkflowRun.java:133)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:1100)
            	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:136)
            	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
            	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
            	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:51)
            	at hudson.security.ACL.impersonate(ACL.java:290)
            	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:48)
            	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            	at java.lang.Thread.run(Thread.java:748)
            

            We are running Jenkins 2.150.1.

            davida2009 David Aldrich added a comment - @jglick The exception occurred again for us this weekend: org.xml.sax.SAXParseException; systemId: file:/ var /lib/jenkins/jobs/Zodiac_gcc_all_versions/builds/123/changelog1.xml; lineNumber: 1; columnNumber: 1; Premature end of file. at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76) Caused: hudson.util.IOException2: Failed to parse / var /lib/jenkins/jobs/Zodiac_gcc_all_versions/builds/123/changelog1.xml at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:835) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1100(WorkflowRun.java:133) at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:1100) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:136) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:51) at hudson.security.ACL.impersonate(ACL.java:290) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:48) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748) We are running Jenkins 2.150.1.
            jglick Jesse Glick added a comment -

            Studying the fix of JENKINS-47201 by dnusbaum I suspect it was incomplete, as this block is not synchronized as a whole, and so two concurrent branches could both pick the same changelog file.

            This is probably reported against the Subversion plugin since that is the most widely used SCM plugin to actually treat changelog.xml as an XML file. Despite the name, the format is up to the SCM plugin to interpret, and the Git plugin writes and reads a completely non-XML format. (The Mercurial plugin also uses XML but is probably too little used for people to hit this and report it.)

            jglick Jesse Glick added a comment - Studying the fix of JENKINS-47201 by dnusbaum I suspect it was incomplete, as this block is not synchronized as a whole, and so two concurrent branches could both pick the same changelog file. This is probably reported against the Subversion plugin since that is the most widely used SCM plugin to actually treat changelog.xml as an XML file. Despite the name, the format is up to the SCM plugin to interpret, and the Git plugin writes and reads a completely non-XML format. (The Mercurial plugin also uses XML but is probably too little used for people to hit this and report it.)
            jglick Jesse Glick made changes -
            Component/s workflow-scm-step-plugin [ 21717 ]
            Component/s subversion-plugin [ 15485 ]
            Labels pipeline
            jglick Jesse Glick made changes -
            Summary Exceptions by parallel SCM checkout SAXParseException from parallel SCM checkout
            dnusbaum Devin Nusbaum added a comment - - edited

            Studying the fix of JENKINS-47201 by Devin Nusbaum I suspect it was incomplete, as this block is not synchronized as a whole, and so two concurrent branches could both pick the same changelog file.

            Looks like it, I think we could reorder some statements in the method so that we could lock Run when looking for the file without needing to lock prev as well, although maybe that code could be refactored to use the temp file APIs or something to be more robust.

            See https://github.com/jenkinsci/workflow-scm-step-plugin/pull/32.

            dnusbaum Devin Nusbaum added a comment - - edited Studying the fix of JENKINS-47201 by Devin Nusbaum I suspect it was incomplete, as this block is not synchronized as a whole, and so two concurrent branches could both pick the same changelog file. Looks like it, I think we could reorder some statements in the method so that we could lock Run when looking for the file without needing to lock prev as well, although maybe that code could be refactored to use the temp file APIs or something to be more robust. See https://github.com/jenkinsci/workflow-scm-step-plugin/pull/32 .
            dnusbaum Devin Nusbaum made changes -
            Remote Link This issue links to "jenkinsci/workflow-scm-step-plugin#32 (Web Link)" [ 22230 ]
            dnusbaum Devin Nusbaum made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            dnusbaum Devin Nusbaum made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            dnusbaum Devin Nusbaum made changes -
            Assignee Devin Nusbaum [ dnusbaum ]
            dnusbaum Devin Nusbaum made changes -
            Remote Link This issue links to "jenkinsci/workflow-scm-step-plugin#31 (Web Link)" [ 22231 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-39968 [ JENKINS-39968 ]
            davida2009 David Aldrich added a comment -

            The SAXParseException is now occuring repeatedly (every build) for us.

            davida2009 David Aldrich added a comment - The SAXParseException is now occuring repeatedly (every build) for us.
            dnusbaum Devin Nusbaum made changes -
            dnusbaum Devin Nusbaum added a comment -

            davida2009 I've attached an incremental build of the Pipeline SCM Step Plugin that contains PR #32. Feel free to try it out (preferably in a test environment first) to let us know if it addresses the problem. I am not sure when I will have time to test the code further myself and merge/release that change, so if you are blocked I would recommend testing that build.

            workflow-scm-step-2.8-rc326.7286a182d923.hpi

            dnusbaum Devin Nusbaum added a comment - davida2009 I've attached an incremental build of the Pipeline SCM Step Plugin that contains PR #32 . Feel free to try it out (preferably in a test environment first) to let us know if it addresses the problem. I am not sure when I will have time to test the code further myself and merge/release that change, so if you are blocked I would recommend testing that build. workflow-scm-step-2.8-rc326.7286a182d923.hpi
            davida2009 David Aldrich added a comment -

            Thanks Devin, I will give it a try.

            davida2009 David Aldrich added a comment - Thanks Devin, I will give it a try.
            davida2009 David Aldrich added a comment - - edited

            dnusbaum I have been running workflow-scm-step-2.8-rc326.7286a182d923.hpi for 3 weeks and have not seen a single instance of the SAXParseException. It has also been running fine with no resource locks for 3 days. So it looks as though PR #32 fixes the problem.

            Please will you consider releasing a formal update of the plugin?

            davida2009 David Aldrich added a comment - - edited dnusbaum I have been running workflow-scm-step-2.8-rc326.7286a182d923.hpi for 3 weeks and have not seen a single instance of the SAXParseException. It has also been running fine with no resource locks for 3 days. So it looks as though PR #32 fixes the problem. Please will you consider releasing a formal update of the plugin?
            dnusbaum Devin Nusbaum made changes -
            Link This issue is duplicated by JENKINS-56254 [ JENKINS-56254 ]
            benjboyer Benjamin Boyer made changes -
            Assignee Devin Nusbaum [ dnusbaum ] Benjamin Boyer [ benjboyer ]
            benjboyer Benjamin Boyer made changes -
            Assignee Benjamin Boyer [ benjboyer ] Devin Nusbaum [ dnusbaum ]
            benjboyer Benjamin Boyer added a comment - - edited

            Hi dnusbaum,

            I'm having the same issue for. I'll use workflow-scm-step-2.8-rc326.7286a182d923.hpi to confirm if it's fixing the "ConcurrentModificationException". I'll test it against 100+ parallelized repositories (in groups of 12 concurrent extractions)

            Thanks in advance

            benjboyer Benjamin Boyer added a comment - - edited Hi dnusbaum , I'm having the same issue for. I'll use workflow-scm-step-2.8-rc326.7286a182d923.hpi to confirm if it's fixing the "ConcurrentModificationException". I'll test it against 100+ parallelized repositories (in groups of 12 concurrent extractions) Thanks in advance
            skojenkins Andre Morales made changes -
            Attachment stable.PNG [ 46653 ]
            skojenkins Andre Morales added a comment -

            I was having the same issue with a pipeline of 23 parallel  checkouts. It was failing randomly. 

            I have installed plugin workflow-scm-step-2.8-rc326.7286a182d923.hpi and everything goes well now.  Since installation, all 23 agents running in parallel are checking out correctly.

             

            skojenkins Andre Morales added a comment - I was having the same issue with a pipeline of 23 parallel  checkouts. It was failing randomly.  I have installed plugin  workflow-scm-step-2.8-rc326.7286a182d923.hpi and everything goes well now.  Since installation, all 23 agents running in parallel are checking out correctly.  

            Is there a chance to get a new release of the plugin with this fix included?
            That would be great as this issue is really problematic for larger parallel pipelines.

            thefifo Wolfgang Issovits added a comment - Is there a chance to get a new release of the plugin with this fix included? That would be great as this issue is really problematic for larger parallel pipelines.

            Please, release the fix. The issue is really a problem for us.

            twerner Torsten Werner added a comment - Please, release the fix. The issue is really a problem for us.
            dnusbaum Devin Nusbaum added a comment -

            A fix for this issue was just released in Pipeline SCM Step Plugin version 2.8.

            dnusbaum Devin Nusbaum added a comment - A fix for this issue was just released in Pipeline SCM Step Plugin version 2.8.
            dnusbaum Devin Nusbaum made changes -
            Released As workflow-scm-step 2.8
            Resolution Fixed [ 1 ]
            Status In Review [ 10005 ] Resolved [ 5 ]
            duael Emmanuel Ruhnau made changes -
            Link This issue is duplicated by JENKINS-53030 [ JENKINS-53030 ]
            solvingj jerry wiltse made changes -
            Link This issue is duplicated by JENKINS-53140 [ JENKINS-53140 ]

            People

              dnusbaum Devin Nusbaum
              gitt Slawa Giterman
              Votes:
              18 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: