Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34313

SAXParseException from parallel SCM checkout

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Labels:
      None
    • Environment:
      Jenkins ver. 1.651.1
      Pipeline plug-ins: ver. 1.15
      Pipeline: SCM Step plug-in: ver. 1.15
      SCM API Plugin: ver 1.1
      Subversion Plug-in: ver. 2.5.6
    • Similar Issues:
    • Released As:
      workflow-scm-step 2.8

      Description

      In out pipeline we at first build multiple upstream Maven projects and then build the main project. Upstream projects are build in parallel:

      parallel(
          projA: {
              projects.buildDependency('projA')
          },
          projB: {
              projects.buildDependency('projB')
          }
      ...
      )
      

      projects.buildDependency() just checks-out each project from SVN and starts Maven build. The problem is that we often (in about 20% of builds) have errors caused by concurrent / parallel SVN check-outs. Sometimes it is ConcurrentModificationException and sometimes "SAXParseException: changelog11.xml; lineNumber: 1; columnNumber: 1; Premature end of file". See stacktraces below.

      As a temporary workaround we added

      sleep time: random.nextInt(1000), unit: 'MILLISECONDS'
      

      just before the checkout step to introduce time shifts between check-outs. We used random numbers because we were not sure if some local or env counter variable would be thread-safe in case of parallel steps.

      Now error happens only if time difference between check-outs is narrow, e.g. 5 milliseconds if 2 generated random numbers are close.

      Would it be possible to make Pipeline plug-in more resilient in case of parallel check-outs?

      Caused by: org.jenkinsci.plugins.workflow.cps.steps.ParallelStepException: Parallel step projA failed
      	... 26 more
      Caused by: hudson.util.IOException2: Failed to parse /var/lib/jenkins/jobs/ha-pipeline/builds/426/changelog11.xml
      	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80)
      	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:638)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1200(WorkflowRun.java:111)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:812)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:123)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
      	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:49)
      	at hudson.security.ACL.impersonate(ACL.java:213)
      	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:47)
      	... 5 more
      Caused by: org.xml.sax.SAXParseException; systemId: file:/var/lib/jenkins/jobs/ha-pipeline/builds/426/changelog11.xml; lineNumber: 1; columnNumber: 1; Premature end of file.
      	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
      	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
      	at org.apache.commons.digester.Digester.parse(Digester.java:1871)
      	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76)
      	... 15 more
      
      org.jenkinsci.plugins.workflow.cps.steps.ParallelStepException: Parallel step projB failed
      	at org.jenkinsci.plugins.workflow.cps.steps.ParallelStep$ResultHandler$Callback.checkAllDone(ParallelStep.java:153)
      	at org.jenkinsci.plugins.workflow.cps.steps.ParallelStep$ResultHandler$Callback.onSuccess(ParallelStep.java:110)
      	at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:311)
      	at com.cloudbees.groovy.cps.impl.LocalVariableBlock$LocalVariable.get(LocalVariableBlock.java:33)
      	at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
      	at com.cloudbees.groovy.cps.impl.LocalVariableBlock.evalLValue(LocalVariableBlock.java:22)
      	at com.cloudbees.groovy.cps.LValueBlock$BlockImpl.eval(LValueBlock.java:55)
      	at com.cloudbees.groovy.cps.LValueBlock.eval(LValueBlock.java:16)
      	at com.cloudbees.groovy.cps.Next.step(Next.java:58)
      	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:106)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
      	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:164)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:277)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:77)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:186)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:184)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      Caused by: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:215)
      	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:183)
      	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
      	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:168)
      	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:109)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
      	at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
      	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
      	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
      	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
      	at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
      	at hudson.XmlFile.write(XmlFile.java:178)
      	at hudson.model.Run.save(Run.java:1896)
      	at hudson.util.PersistedList.onModified(PersistedList.java:173)
      	at hudson.util.PersistedList._onModified(PersistedList.java:181)
      	at hudson.util.PersistedList.add(PersistedList.java:72)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:651)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1200(WorkflowRun.java:111)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:812)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:123)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
      	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:49)
      	at hudson.security.ACL.impersonate(ACL.java:213)
      	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:47)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState#revisionStates for class org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:215)
      	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:183)
      	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
      	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:168)
      	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:109)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88)
      	at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64)
      	at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
      	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:224)
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:211)
      	... 31 more
      Caused by: java.util.ConcurrentModificationException
      	at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922)
      	at java.util.HashMap$EntryIterator.next(HashMap.java:962)
      	at java.util.HashMap$EntryIterator.next(HashMap.java:960)
      	at com.thoughtworks.xstream.converters.collections.MapConverter.marshal(MapConverter.java:75)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
      	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:224)
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:211)
      	... 46 more
      

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            The SAXParseException seems like a bug in the Subversion plugin.

            The ConcurrentModificationException is JENKINS-19561.

            Show
            jglick Jesse Glick added a comment - The SAXParseException seems like a bug in the Subversion plugin. The ConcurrentModificationException is JENKINS-19561 .
            Hide
            jeanmertz Jean Mertz added a comment -

            I believe we are seeing the same problem, using Git.

            The below error happens when running tests in parallel, on different nodes, and each node triggering `checkout(scm)` using Jenkins Pipeline workflow.

            [test:1]  > git rev-parse --is-inside-work-tree # timeout=10
            [test:1] Fetching changes from 2 remote Git repositories
            [test:1]  > git config remote.origin.url https://github.com/org/our-repo.git # timeout=10
            [test:1] Fetching upstream changes from https://github.com/org/our-repo.git
            [test:1]  > git --version # timeout=10
            [test:1] using GIT_ASKPASS to set credentials github login credentials for "ci" account
            [test:1]  > git fetch --tags --progress https://github.com/org/our-repo.git +refs/heads/*:refs/remotes/origin/*
            [test:1]  > git config remote.origin1.url https://github.com/org/our-repo.git # timeout=10
            [test:1] ERROR: Error fetching remote repo 'origin1'
            [test:1] hudson.plugins.git.GitException: Failed to fetch from https://github.com/org/our-repo.git
            [test:1]   at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:797)
            [test:1]   at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1051)
            [test:1]   at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1082)
            [test:1]   at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109)
            [test:1]   at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
            [test:1]   at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
            [test:1]   at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:52)
            [test:1]   at hudson.security.ACL.impersonate(ACL.java:221)
            [test:1]   at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:49)
            [test:1]   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            [test:1]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            [test:1]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            [test:1]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            [test:1]   at java.lang.Thread.run(Thread.java:745)
            [test:1] Caused by: hudson.plugins.git.GitException: Command "git config remote.origin1.url https://github.com/org/our-repo.git" returned status code 255:
            [test:1] stdout: 
            [test:1] stderr: error: could not lock config file .git/config: File exists
            [test:1] 
            [test:1]   at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1752)
            [test:1]   at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1725)
            [test:1]   at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1721)
            [test:1]   at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1391)
            [test:1]   at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1403)
            [test:1]   at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1107)
            [test:1]   at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
            [test:1]   at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
            [test:1]   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            [test:1]   at java.lang.reflect.Method.invoke(Method.java:498)
            [test:1]   at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:887)
            [test:1]   at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:862)
            [test:1]   at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:821)
            [test:1]   at hudson.remoting.UserRequest.perform(UserRequest.java:153)
            [test:1]   at hudson.remoting.UserRequest.perform(UserRequest.java:50)
            [test:1]   at hudson.remoting.Request$2.run(Request.java:332)
            [test:1]   at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
            [test:1]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            [test:1]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            [test:1]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            [test:1]   at hudson.remoting.Engine$1$1.run(Engine.java:88)
            [test:1]   at java.lang.Thread.run(Thread.java:745)
            [test:1]   at ......remote call to kubernetes-f2462ed7839d4e6a833775d67a2b867d-3b5c4c72b81c(Native Method)
            [test:1]   at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1433)
            [test:1]   at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
            [test:1]   at hudson.remoting.Channel.call(Channel.java:797)
            [test:1]   at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:252)
            [test:1]   at com.sun.proxy.$Proxy108.setRemoteUrl(Unknown Source)
            [test:1]   at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:294)
            [test:1]   at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:785)
            [test:1]   ... 13 more
            [test:1] Failed in branch test:1
            
            Show
            jeanmertz Jean Mertz added a comment - I believe we are seeing the same problem, using Git. The below error happens when running tests in parallel, on different nodes, and each node triggering `checkout(scm)` using Jenkins Pipeline workflow. [test:1] > git rev-parse --is-inside-work-tree # timeout=10 [test:1] Fetching changes from 2 remote Git repositories [test:1] > git config remote.origin.url https://github.com/org/our-repo.git # timeout=10 [test:1] Fetching upstream changes from https://github.com/org/our-repo.git [test:1] > git --version # timeout=10 [test:1] using GIT_ASKPASS to set credentials github login credentials for "ci" account [test:1] > git fetch --tags --progress https://github.com/org/our-repo.git +refs/heads/*:refs/remotes/origin/* [test:1] > git config remote.origin1.url https://github.com/org/our-repo.git # timeout=10 [test:1] ERROR: Error fetching remote repo 'origin1' [test:1] hudson.plugins.git.GitException: Failed to fetch from https://github.com/org/our-repo.git [test:1] at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:797) [test:1] at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1051) [test:1] at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1082) [test:1] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109) [test:1] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83) [test:1] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73) [test:1] at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:52) [test:1] at hudson.security.ACL.impersonate(ACL.java:221) [test:1] at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:49) [test:1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [test:1] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [test:1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [test:1] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [test:1] at java.lang.Thread.run(Thread.java:745) [test:1] Caused by: hudson.plugins.git.GitException: Command "git config remote.origin1.url https://github.com/org/our-repo.git" returned status code 255: [test:1] stdout: [test:1] stderr: error: could not lock config file .git/config: File exists [test:1] [test:1] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1752) [test:1] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1725) [test:1] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1721) [test:1] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1391) [test:1] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1403) [test:1] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1107) [test:1] at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160) [test:1] at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) [test:1] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [test:1] at java.lang.reflect.Method.invoke(Method.java:498) [test:1] at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:887) [test:1] at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:862) [test:1] at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:821) [test:1] at hudson.remoting.UserRequest.perform(UserRequest.java:153) [test:1] at hudson.remoting.UserRequest.perform(UserRequest.java:50) [test:1] at hudson.remoting.Request$2.run(Request.java:332) [test:1] at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) [test:1] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [test:1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [test:1] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [test:1] at hudson.remoting.Engine$1$1.run(Engine.java:88) [test:1] at java.lang.Thread.run(Thread.java:745) [test:1] at ......remote call to kubernetes-f2462ed7839d4e6a833775d67a2b867d-3b5c4c72b81c(Native Method) [test:1] at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1433) [test:1] at hudson.remoting.UserResponse.retrieve(UserRequest.java:253) [test:1] at hudson.remoting.Channel.call(Channel.java:797) [test:1] at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:252) [test:1] at com.sun.proxy.$Proxy108.setRemoteUrl(Unknown Source) [test:1] at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:294) [test:1] at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:785) [test:1] ... 13 more [test:1] Failed in branch test:1
            Hide
            jglick Jesse Glick added a comment -

            Jean Mertz probably an unrelated issue which should be filed separately with detailed steps to reproduce from scratch. Offhand it sounds like you are actually using the same workspace in different branches, but I do not really recognize the error, which is coming from the command-line git tool.

            Show
            jglick Jesse Glick added a comment - Jean Mertz probably an unrelated issue which should be filed separately with detailed steps to reproduce from scratch. Offhand it sounds like you are actually using the same workspace in different branches, but I do not really recognize the error, which is coming from the command-line git tool.
            Hide
            mixermachine Aaron Dietz added a comment - - edited

            I see the same error at the end of a large parallel execution (44 parallel steps).

            This pipeline generates a log file of up to 2 MB.

            A single steps looks like this:

             

            node ('windows %some label%') {
             checkout svn
            
             lock ('execution for this node') {
              lock ('specific part of this execution') {
                withCredentials execute batch script
               }  
             }
            
             handle errors of batch script
            }
            

             

            Error after last parallel step has finished: 

            [Pipeline] [*lastParallelPart*] // node
            [Pipeline] [*lastParallelPart*] }
            [Pipeline] // parallel
            [Pipeline] End of Pipeline
            hudson.util.IOException2: Failed to parse /opt/ccde/jenkins_home/jobs/*job*/builds/268/changelog1.xml
            	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80)
            	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:746)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1500(WorkflowRun.java:125)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:936)
            	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:123)
            	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
            	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
            	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
            	at hudson.security.ACL.impersonate(ACL.java:260)
            	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
            	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            	at java.lang.Thread.run(Thread.java:748)
            	Suppressed: hudson.util.IOException2: Failed to parse /opt/ccde/jenkins_home/jobs/*job*/builds/268/changelog1.xml
            		... 16 more
            	Caused by: org.xml.sax.SAXParseException; systemId: file:/opt/ccde/jenkins_home/jobs/*job*/builds/268/changelog1.xml; lineNumber: 1; columnNumber: 1; Premature end of file.
            		at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
            		at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
            		at org.apache.commons.digester.Digester.parse(Digester.java:1871)
            		at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76)
            		... 15 more
            Caused by: org.xml.sax.SAXParseException; systemId: file:/opt/ccde/jenkins_home/jobs/*job*/builds/268/changelog1.xml; lineNumber: 1; columnNumber: 1; Premature end of file.
            	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
            	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
            	at org.apache.commons.digester.Digester.parse(Digester.java:1871)
            	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76)
            	... 15 more
            Finished: FAILURE
            

             

             

            I added a random delay (0 - 2 seconds) before the svn checkout and at the end of each parallel step.

            I also added a Lock (Lockable Resources Plugin) at the end of a parallel step to prevent a parallel finish of multiple parallel steps.

            It would be very nice, if this could be fixed. 

            Show
            mixermachine Aaron Dietz added a comment - - edited I see the same error at the end of a large parallel execution (44 parallel steps). This pipeline generates a log file of up to 2 MB. A single steps looks like this:   node ( 'windows %some label%' ) {  checkout svn lock ( 'execution for this node' ) { lock ( 'specific part of this execution' ) { withCredentials execute batch script } } handle errors of batch script }   Error after last parallel step has finished:  [Pipeline] [*lastParallelPart*] // node [Pipeline] [*lastParallelPart*] } [Pipeline] // parallel [Pipeline] End of Pipeline hudson.util.IOException2: Failed to parse /opt/ccde/jenkins_home/jobs /*job*/ builds/268/changelog1.xml at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:746) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1500(WorkflowRun.java:125) at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:936) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:123) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:260) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:748) Suppressed: hudson.util.IOException2: Failed to parse /opt/ccde/jenkins_home/jobs /*job*/ builds/268/changelog1.xml ... 16 more Caused by: org.xml.sax.SAXParseException; systemId: file:/opt/ccde/jenkins_home/jobs /*job*/ builds/268/changelog1.xml; lineNumber: 1; columnNumber: 1; Premature end of file. at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76) ... 15 more Caused by: org.xml.sax.SAXParseException; systemId: file:/opt/ccde/jenkins_home/jobs /*job*/ builds/268/changelog1.xml; lineNumber: 1; columnNumber: 1; Premature end of file. at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76) ... 15 more Finished: FAILURE     I added a random delay (0 - 2 seconds) before the svn checkout and at the end of each parallel step. I also added a Lock (Lockable Resources Plugin) at the end of a parallel step to prevent a parallel finish of multiple parallel steps. It would be very nice, if this could be fixed. 
            Hide
            kerrhome Shannon Kerr added a comment -

            Just ran into this today when we switched from VM hosted Jenkins to docker container:

            org.xml.sax.SAXParseException; Premature end of file.
             at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
             at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
             at org.apache.commons.digester.Digester.parse(Digester.java:1871)
             at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76)
            Also: org.xml.sax.SAXParseException; Premature end of file.
             at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
             at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
             at org.apache.commons.digester.Digester.parse(Digester.java:1871)
             at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76)
             Caused: hudson.util.IOException2: Failed to parse /var/jenkins_home/jobs/*job*/branches/*branch*/builds/2/changelog1.xml
            Caused: hudson.util.IOException2: Failed to parse /var/jenkins_home/jobs/*job*/branches/*branch*/builds/2/changelog1.xml
             at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80)
             at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:1031)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1400(WorkflowRun.java:144)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:1239)
             at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:127)
             at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
             at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
             at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
             at hudson.security.ACL.impersonate(ACL.java:290)
             at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
             at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
             at java.util.concurrent.FutureTask.run(FutureTask.java:266)
             at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
             at java.lang.Thread.run(Thread.java:748)
            Finished: FAILURE

             

            Before today, we had never seen this.  Been using Jenkins out of a VM (Ubuntu 12) for 4 years.

             

            Show
            kerrhome Shannon Kerr added a comment - Just ran into this today when we switched from VM hosted Jenkins to docker container: org.xml.sax.SAXParseException; Premature end of file. at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76) Also: org.xml.sax.SAXParseException; Premature end of file. at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76) Caused: hudson.util.IOException2: Failed to parse / var /jenkins_home/jobs /*job*/ branches /*branch*/ builds/2/changelog1.xml Caused: hudson.util.IOException2: Failed to parse / var /jenkins_home/jobs /*job*/ branches /*branch*/ builds/2/changelog1.xml at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:1031) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1400(WorkflowRun.java:144) at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:1239) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:127) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:290) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748) Finished: FAILURE   Before today, we had never seen this.  Been using Jenkins out of a VM (Ubuntu 12) for 4 years.  
            Hide
            kerrhome Shannon Kerr added a comment -

            We are getting this so frequently now.  It has to be about 50% of our builds are failing with:

            Failed to parse /var/jenkins_home/jobs/*job*/branches/*branch*/builds/103/changelog1.xml
            Show
            kerrhome Shannon Kerr added a comment - We are getting this so frequently now.  It has to be about 50% of our builds are failing with: Failed to parse / var /jenkins_home/jobs /*job*/ branches /*branch*/ builds/103/changelog1.xml
            Hide
            kerrhome Shannon Kerr added a comment -

            This is killing us today.  So man occurrences.  More than 50% of our builds. The recommended "sleep time" work-around did not appear to help when implemented on some branch build testing.  We are move it to trunk because it has gotten so bad today, but I'm not hopeful.

             

            Show
            kerrhome Shannon Kerr added a comment - This is killing us today.  So man occurrences.  More than 50% of our builds. The recommended "sleep time" work-around did not appear to help when implemented on some branch build testing.  We are move it to trunk because it has gotten so bad today, but I'm not hopeful.  
            Hide
            mixermachine Aaron Dietz added a comment -

            @Shannon Kerr

            Have you also tried the lock method from my approach above?
            Acquire a singleton lock "execution end" (or a nicer name ) at the end of the execution and release it instandly.
            This way no execution can end in parallel.
            Next escalation step is to add a minimal delay before you release the lock.

            Definitely not pretty, but fixes it for me

            Show
            mixermachine Aaron Dietz added a comment - @Shannon Kerr Have you also tried the lock method from my approach above? Acquire a singleton lock "execution end" (or a nicer name ) at the end of the execution and release it instandly. This way no execution can end in parallel. Next escalation step is to add a minimal delay before you release the lock. Definitely not pretty, but fixes it for me
            Hide
            kerrhome Shannon Kerr added a comment -

            Aaron Dietz I have not yet. Will definitely try this today. The team I support is really suffering with this issue.  I stopped updating Jenkins and our plugins over a month ago (as we approach our product release), but this issue seems to be increasing in frequency, which I cannot explain since I've frozen everything.

             

            Show
            kerrhome Shannon Kerr added a comment - Aaron Dietz I have not yet. Will definitely try this today. The team I support is really suffering with this issue.  I stopped updating Jenkins and our plugins over a month ago (as we approach our product release), but this issue seems to be increasing in frequency, which I cannot explain since I've frozen everything.  
            Hide
            kerrhome Shannon Kerr added a comment - - edited

            Aaron Dietz can you share your implementation of the post checkout lock?  I've not used this plugin up to now (lock).  Thank you.

             

            edit: Perhaps your implementation is what you wrote in your original comment above.  I'm using scripted pipeline at this point and right now I have:

             

            Random random = new Random()
            sleep time: random.nextInt(2000), unit: 'MILLISECONDS'
            checkout(.....)
            lock ("SVN CO end for ${env.NODE_NAME}") {
                echo "Post-svnco lock"
            }
            sleep time: random.nextInt(2000), unit: 'MILLISECONDS'
            

            Is that sufficient?

             

            Show
            kerrhome Shannon Kerr added a comment - - edited Aaron Dietz can you share your implementation of the post checkout lock?  I've not used this plugin up to now (lock).  Thank you.   edit: Perhaps your implementation is what you wrote in your original comment above.  I'm using scripted pipeline at this point and right now I have:   Random random = new Random() sleep time: random.nextInt(2000), unit: 'MILLISECONDS' checkout(.....) lock ( "SVN CO end for ${env.NODE_NAME}" ) {     echo "Post-svnco lock" } sleep time: random.nextInt(2000), unit: 'MILLISECONDS' Is that sufficient?  
            Hide
            mixermachine Aaron Dietz added a comment -

            Shannon Kerr

            Just try your luck with the lock at first.

            lock ("end_lock") {
                echo "end lock"
            }
            

            This is the least time consuming way.

             

            If that does not work:

            lock ("end_lock") {
                echo "end lock"
                sleep time: (50 + random.nextInt(400)) // hopefully enough time
            }
            

            It might be enough to have one lock for each node (as you did show), but it could also be necessary to synchronize all jobs as some files are written back to the master.

             

            I hope this helps

             

            Greetings from Germany

            Aaron

            Show
            mixermachine Aaron Dietz added a comment - Shannon Kerr Just try your luck with the lock at first. lock ( "end_lock" ) { echo "end lock" } This is the least time consuming way.   If that does not work: lock ( "end_lock" ) { echo "end lock" sleep time: (50 + random.nextInt(400)) // hopefully enough time } It might be enough to have one lock for each node (as you did show), but it could also be necessary to synchronize all jobs as some files are written back to the master.   I hope this helps   Greetings from Germany Aaron
            Hide
            kerrhome Shannon Kerr added a comment -

            Aaron Dietz Thank you.  I've been testing with this all day today and so far so good.  Hopefully your solution works until this ticket gets a proper resolution. Thank you again.

            Show
            kerrhome Shannon Kerr added a comment - Aaron Dietz Thank you.  I've been testing with this all day today and so far so good.  Hopefully your solution works until this ticket gets a proper resolution. Thank you again.
            Hide
            mixermachine Aaron Dietz added a comment -

            Shannon Kerr No problem, happy to help

            Show
            mixermachine Aaron Dietz added a comment - Shannon Kerr No problem, happy to help
            Hide
            kerrhome Shannon Kerr added a comment -

            Aaron Dietz well, I've had two occurrences of the issue since the new lock implementation.  Just to be clear about what we're doing, we are running at least 3 svn co's in parallel via a call to our groovy script.  These checkouts are happening on 3 different machines (one Linux physical machine for docker builds and 2 Windows VMs), all 3 checkouts in parallel.  I assume when others are doing checkouts that they too are running parallel checkouts on different machines, but thought I'd better clarify.

            Are there any other changes you (or others) recommend we try to stop these issues?

            Show
            kerrhome Shannon Kerr added a comment - Aaron Dietz well, I've had two occurrences of the issue since the new lock implementation.  Just to be clear about what we're doing, we are running at least 3 svn co's in parallel via a call to our groovy script.  These checkouts are happening on 3 different machines (one Linux physical machine for docker builds and 2 Windows VMs), all 3 checkouts in parallel.  I assume when others are doing checkouts that they too are running parallel checkouts on different machines, but thought I'd better clarify. Are there any other changes you (or others) recommend we try to stop these issues?
            Hide
            davida2009 David Aldrich added a comment -

            Please show me how to implement a lock in a declarative pipeline script.

            Show
            davida2009 David Aldrich added a comment - Please show me how to implement a lock in a declarative pipeline script.
            Hide
            mixermachine Aaron Dietz added a comment -

            Shannon Kerr sorry for the late answer, I'm currently busy with my bachelor thesis.

            Mh I can only guess...

            I think preventing paralle CO is not an option ? (then you could simply create a lock for the CO command)

            A very wild guess is to check what hard drives your (main?) server uses. Maybe you can upgrade to a SSD setup so the machine spends less time to write the files out? (not optimal fix for a software problem, I know)

            The fastest thing you can currently try is to place locks and delays on variouse places.

            I might look into this further after I submited my bachelor thesis (time budget is currently packed).

             

            David Aldrich

            Hy David

            I have used the "normal" jenkins pipeline scripts till now and don't know of a declarative pipeline script (maybe we talk about the same? Groovy scripting language that defines a Jenkins Pipeline)

            If we talk about the same, my approach from a few posts up should work for you (hopefully, as this is a temporary fix).

            Show
            mixermachine Aaron Dietz added a comment - Shannon Kerr sorry for the late answer, I'm currently busy with my bachelor thesis. Mh I can only guess... I think preventing paralle CO is not an option ? (then you could simply create a lock for the CO command) A very wild guess is to check what hard drives your (main?) server uses. Maybe you can upgrade to a SSD setup so the machine spends less time to write the files out? (not optimal fix for a software problem, I know) The fastest thing you can currently try is to place locks and delays on variouse places. I might look into this further after I submited my bachelor thesis (time budget is currently packed).   David Aldrich Hy David I have used the "normal" jenkins pipeline scripts till now and don't know of a declarative pipeline script (maybe we talk about the same? Groovy scripting language that defines a Jenkins Pipeline) If we talk about the same, my approach from a few posts up should work for you (hopefully, as this is a temporary fix).
            Hide
            jglick Jesse Glick added a comment -

            AFAIK workflow-scm-step 2.7 should have resolved this (JENKINS-47201), unless you are talking about some unrelated issue.

            Show
            jglick Jesse Glick added a comment - AFAIK workflow-scm-step 2.7 should have resolved this ( JENKINS-47201 ), unless you are talking about some unrelated issue.
            Hide
            mixermachine Aaron Dietz added a comment -

            Jesse Glick that are some good news, thanks for the implementation.

            Shannon Kerr fixed for you?

            Show
            mixermachine Aaron Dietz added a comment - Jesse Glick that are some good news, thanks for the implementation. Shannon Kerr fixed for you?
            Hide
            davida2009 David Aldrich added a comment - - edited

            Jesse Glick I guess you mean Pipeline: SCM Step 2.7? I updated to that plugin in October 2018, but have seen the problem since then.

            Show
            davida2009 David Aldrich added a comment - - edited Jesse Glick I guess you mean Pipeline: SCM Step 2.7? I updated to that plugin in October 2018, but have seen the problem since then.
            Hide
            jglick Jesse Glick added a comment -

            the problem

            Meaning a ConcurrentModificationException serializing MultiSCMRevisionState#revisionStates, or some other problem?

            Show
            jglick Jesse Glick added a comment - the problem Meaning a ConcurrentModificationException serializing MultiSCMRevisionState#revisionStates , or some other problem?
            Hide
            davida2009 David Aldrich added a comment -

            @jglick Sorry, I don't have a record of the exception now, but it did mention org.xml.sax.SAXParseException.

            I seem to have fixed it by adding a lock at the end of each parallel stage.

            Show
            davida2009 David Aldrich added a comment - @jglick Sorry, I don't have a record of the exception now, but it did mention org.xml.sax.SAXParseException. I seem to have fixed it by adding a lock at the end of each parallel stage.
            Hide
            davida2009 David Aldrich added a comment -

            @jglick The exception occurred again for us this weekend:

            org.xml.sax.SAXParseException; systemId: file:/var/lib/jenkins/jobs/Zodiac_gcc_all_versions/builds/123/changelog1.xml; lineNumber: 1; columnNumber: 1; Premature end of file.
            	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
            	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
            	at org.apache.commons.digester.Digester.parse(Digester.java:1871)
            	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76)
            Caused: hudson.util.IOException2: Failed to parse /var/lib/jenkins/jobs/Zodiac_gcc_all_versions/builds/123/changelog1.xml
            	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80)
            	at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:835)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1100(WorkflowRun.java:133)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:1100)
            	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:136)
            	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
            	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
            	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:51)
            	at hudson.security.ACL.impersonate(ACL.java:290)
            	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:48)
            	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            	at java.lang.Thread.run(Thread.java:748)
            

            We are running Jenkins 2.150.1.

            Show
            davida2009 David Aldrich added a comment - @jglick The exception occurred again for us this weekend: org.xml.sax.SAXParseException; systemId: file:/ var /lib/jenkins/jobs/Zodiac_gcc_all_versions/builds/123/changelog1.xml; lineNumber: 1; columnNumber: 1; Premature end of file. at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76) Caused: hudson.util.IOException2: Failed to parse / var /lib/jenkins/jobs/Zodiac_gcc_all_versions/builds/123/changelog1.xml at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:80) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:835) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1100(WorkflowRun.java:133) at org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:1100) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:136) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:51) at hudson.security.ACL.impersonate(ACL.java:290) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:48) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748) We are running Jenkins 2.150.1.
            Hide
            jglick Jesse Glick added a comment -

            Studying the fix of JENKINS-47201 by Devin Nusbaum I suspect it was incomplete, as this block is not synchronized as a whole, and so two concurrent branches could both pick the same changelog file.

            This is probably reported against the Subversion plugin since that is the most widely used SCM plugin to actually treat changelog.xml as an XML file. Despite the name, the format is up to the SCM plugin to interpret, and the Git plugin writes and reads a completely non-XML format. (The Mercurial plugin also uses XML but is probably too little used for people to hit this and report it.)

            Show
            jglick Jesse Glick added a comment - Studying the fix of JENKINS-47201 by Devin Nusbaum I suspect it was incomplete, as this block is not synchronized as a whole, and so two concurrent branches could both pick the same changelog file. This is probably reported against the Subversion plugin since that is the most widely used SCM plugin to actually treat changelog.xml as an XML file. Despite the name, the format is up to the SCM plugin to interpret, and the Git plugin writes and reads a completely non-XML format. (The Mercurial plugin also uses XML but is probably too little used for people to hit this and report it.)
            Hide
            dnusbaum Devin Nusbaum added a comment - - edited

            Studying the fix of JENKINS-47201 by Devin Nusbaum I suspect it was incomplete, as this block is not synchronized as a whole, and so two concurrent branches could both pick the same changelog file.

            Looks like it, I think we could reorder some statements in the method so that we could lock Run when looking for the file without needing to lock prev as well, although maybe that code could be refactored to use the temp file APIs or something to be more robust.

            See https://github.com/jenkinsci/workflow-scm-step-plugin/pull/32.

            Show
            dnusbaum Devin Nusbaum added a comment - - edited Studying the fix of JENKINS-47201 by Devin Nusbaum I suspect it was incomplete, as this block is not synchronized as a whole, and so two concurrent branches could both pick the same changelog file. Looks like it, I think we could reorder some statements in the method so that we could lock Run when looking for the file without needing to lock prev as well, although maybe that code could be refactored to use the temp file APIs or something to be more robust. See https://github.com/jenkinsci/workflow-scm-step-plugin/pull/32 .
            Hide
            davida2009 David Aldrich added a comment -

            The SAXParseException is now occuring repeatedly (every build) for us.

            Show
            davida2009 David Aldrich added a comment - The SAXParseException is now occuring repeatedly (every build) for us.
            Hide
            dnusbaum Devin Nusbaum added a comment -

            David Aldrich I've attached an incremental build of the Pipeline SCM Step Plugin that contains PR #32. Feel free to try it out (preferably in a test environment first) to let us know if it addresses the problem. I am not sure when I will have time to test the code further myself and merge/release that change, so if you are blocked I would recommend testing that build.

            workflow-scm-step-2.8-rc326.7286a182d923.hpi

            Show
            dnusbaum Devin Nusbaum added a comment - David Aldrich I've attached an incremental build of the Pipeline SCM Step Plugin that contains PR #32 . Feel free to try it out (preferably in a test environment first) to let us know if it addresses the problem. I am not sure when I will have time to test the code further myself and merge/release that change, so if you are blocked I would recommend testing that build. workflow-scm-step-2.8-rc326.7286a182d923.hpi
            Hide
            davida2009 David Aldrich added a comment -

            Thanks Devin, I will give it a try.

            Show
            davida2009 David Aldrich added a comment - Thanks Devin, I will give it a try.
            Hide
            davida2009 David Aldrich added a comment - - edited

            Devin Nusbaum I have been running workflow-scm-step-2.8-rc326.7286a182d923.hpi for 3 weeks and have not seen a single instance of the SAXParseException. It has also been running fine with no resource locks for 3 days. So it looks as though PR #32 fixes the problem.

            Please will you consider releasing a formal update of the plugin?

            Show
            davida2009 David Aldrich added a comment - - edited Devin Nusbaum I have been running workflow-scm-step-2.8-rc326.7286a182d923.hpi for 3 weeks and have not seen a single instance of the SAXParseException. It has also been running fine with no resource locks for 3 days. So it looks as though PR #32 fixes the problem. Please will you consider releasing a formal update of the plugin?
            Hide
            benjboyer Benjamin Boyer added a comment - - edited

            Hi Devin Nusbaum,

            I'm having the same issue for. I'll use workflow-scm-step-2.8-rc326.7286a182d923.hpi to confirm if it's fixing the "ConcurrentModificationException". I'll test it against 100+ parallelized repositories (in groups of 12 concurrent extractions)

            Thanks in advance

            Show
            benjboyer Benjamin Boyer added a comment - - edited Hi Devin Nusbaum , I'm having the same issue for. I'll use workflow-scm-step-2.8-rc326.7286a182d923.hpi to confirm if it's fixing the "ConcurrentModificationException". I'll test it against 100+ parallelized repositories (in groups of 12 concurrent extractions) Thanks in advance
            Hide
            skojenkins Andre Morales added a comment -

            I was having the same issue with a pipeline of 23 parallel  checkouts. It was failing randomly. 

            I have installed plugin workflow-scm-step-2.8-rc326.7286a182d923.hpi and everything goes well now.  Since installation, all 23 agents running in parallel are checking out correctly.

             

            Show
            skojenkins Andre Morales added a comment - I was having the same issue with a pipeline of 23 parallel  checkouts. It was failing randomly.  I have installed plugin  workflow-scm-step-2.8-rc326.7286a182d923.hpi and everything goes well now.  Since installation, all 23 agents running in parallel are checking out correctly.  
            Hide
            thefifo Wolfgang Issovits added a comment -

            Is there a chance to get a new release of the plugin with this fix included?
            That would be great as this issue is really problematic for larger parallel pipelines.

            Show
            thefifo Wolfgang Issovits added a comment - Is there a chance to get a new release of the plugin with this fix included? That would be great as this issue is really problematic for larger parallel pipelines.
            Hide
            twerner Torsten Werner added a comment -

            Please, release the fix. The issue is really a problem for us.

            Show
            twerner Torsten Werner added a comment - Please, release the fix. The issue is really a problem for us.
            Hide
            dnusbaum Devin Nusbaum added a comment -

            A fix for this issue was just released in Pipeline SCM Step Plugin version 2.8.

            Show
            dnusbaum Devin Nusbaum added a comment - A fix for this issue was just released in Pipeline SCM Step Plugin version 2.8.

              People

              Assignee:
              dnusbaum Devin Nusbaum
              Reporter:
              gitt Slawa Giterman
              Votes:
              18 Vote for this issue
              Watchers:
              21 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: