Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59328

job hangs although failed after ConcurrentModificationException of ExwsAllocateActionImpl

XMLWordPrintable

      I have a pipeline Job that does some simple checks for all nodes every 5 minutes.
      Sometimes (something like twice a day) this job fails with ConcurrentModificationException in ExwsAllocateActionImpl (on a Windows node while accessing a file on the network). This has been happening ever since but I didn't bother because the next build ran fine again.

      But since some updates (in July or August) the build hangs in a strange state after this exception.
      Somehow the build is still running although it is failed already.
      Therefore the next build doesn't run because it waits for the previous build to finish (I configured to not allow concurrent builds).
      I can't abort the failed build because it is already failed.
      So I have to delete the failed build in order to get the next build running.

      It doesn't happen anymore since I switched "Pipeline speed/durability override" from
      performance optimized to maximum durability

      Log:

      Started by timer
      Running in Durability level: PERFORMANCE_OPTIMIZED
      [Pipeline] Start of Pipeline
      [Pipeline] stage
      [Pipeline] { (master)
      [Pipeline] sleep
      Sleeping for 1 sec
      [Pipeline] node
      Running on Jenkins in /rsync/JenkinsJobs/workspace/work_ServerCheck
      [Pipeline] {
      [Pipeline] sleep
      Sleeping for 1 sec
      [Pipeline] isUnix
      [Pipeline] sh
      + df /rsync
      [Pipeline] sh
      + uptime
      [Pipeline] isUnix
      [Pipeline] isUnix
      [Pipeline] echo
      master: 37.99999952316284
      [Pipeline] echo
      we have 34710392 bytes left on master
      [Pipeline] fileOperations
      File Delete Operation:
      /rsync/JenkinsJobs/workspace/work_ServerCheck/build_152473_master.txt deleting....
      Success.
      [Pipeline] sleep
      Sleeping for 1 sec
      [Pipeline] writeFile
      [Pipeline] exwsAllocate
      Disk allocation strategy was not provided as step parameter. Fallback to the strategy defined in the Jenkins global config
      Using Disk allocation strategy: 'Select the Disk with the most usable space'
      Selected Disk ID 'S_Produkte__Builds' from the Disk Pool ID 'ArchivedBuilds'
      The path on Disk is: unmerged/work_ServerCheck
      [Pipeline] exws
      Searching for disk definitions in the External Workspace Templates from Jenkins global config
      Running in /media/Builds/unmerged/work_ServerCheck
      [Pipeline]

      { [Pipeline] pwd [Pipeline] fileOperations File Delete Operation: /media/Builds/unmerged/work_ServerCheck/build_152473_master.txt deleting.... Success. [Pipeline] sleep Sleeping for 1 sec [Pipeline] }

      [Pipeline] // exws
      [Pipeline] fileOperations
      File Copy Operation:
      /rsync/JenkinsJobs/workspace/work_ServerCheck/build_152474_master.txt
      [Pipeline] sleep
      Sleeping for 1 sec
      [Pipeline] exws
      Searching for disk definitions in the External Workspace Templates from Jenkins global config
      Running in /media/Builds/unmerged/work_ServerCheck
      [Pipeline]

      { [Pipeline] fileExists [Pipeline] }
      [Pipeline] // exws
      [Pipeline] sleep
      Sleeping for 1 sec
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] stage
      [Pipeline] { (domvsaf1)
      [Pipeline] sleep
      Sleeping for 1 sec
      [Pipeline] node
      Running on domvsaf1 in /rsync/JenkinsJobs/workspace/work_ServerCheck
      [Pipeline] { [Pipeline] sleep Sleeping for 1 sec [Pipeline] isUnix [Pipeline] sh + df /rsync [Pipeline] sh + uptime [Pipeline] isUnix [Pipeline] isUnix [Pipeline] echo domvsaf1: 0.9999999776482582 [Pipeline] echo we have 39575068 bytes left on domvsaf1 [Pipeline] fileOperations File Delete Operation: [Pipeline] sleep Sleeping for 1 sec /rsync/JenkinsJobs/workspace/work_ServerCheck/build_152473_domvsaf1.txt deleting.... Success. [Pipeline] writeFile [Pipeline] exwsAllocate Disk allocation strategy was not provided as step parameter. Fallback to the strategy defined in the Jenkins global config Using Disk allocation strategy: 'Select the Disk with the most usable space' Selected Disk ID 'S_Produkte__Builds' from the Disk Pool ID 'ArchivedBuilds' The path on Disk is: unmerged/work_ServerCheck [Pipeline] exws Searching for disk definitions in the External Workspace Templates from Jenkins global config Running in /media/Builds/unmerged/work_ServerCheck [Pipeline] \{ [Pipeline] pwd [Pipeline] fileOperations File Delete Operation: [Pipeline] sleep Sleeping for 1 sec /media/Builds/unmerged/work_ServerCheck/build_152473_domvsaf1.txt deleting.... Success. [Pipeline] }
      [Pipeline] // exws
      [Pipeline] fileOperations
      File Copy Operation:
      [Pipeline] sleep
      Sleeping for 1 sec
      /rsync/JenkinsJobs/workspace/work_ServerCheck/build_152474_domvsaf1.txt
      [Pipeline] exws
      Searching for disk definitions in the External Workspace Templates from Jenkins global config
      Running in /media/Builds/unmerged/work_ServerCheck
      [Pipeline] {[Pipeline] fileExists[Pipeline] }

      [Pipeline] // exws
      [Pipeline] sleep
      Sleeping for 1 sec
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] stage
      [Pipeline] { (SRV1625u)
      [Pipeline] sleep
      Sleeping for 1 sec
      [Pipeline] node
      Running on SRV1625u in /rsync/JenkinsJobs/workspace/work_ServerCheck
      [Pipeline] {
      [Pipeline] sleep
      Sleeping for 1 sec
      [Pipeline] isUnix
      [Pipeline] sh
      + df /rsync
      [Pipeline] sh
      + uptime
      [Pipeline] isUnix
      [Pipeline] isUnix
      [Pipeline] echo
      SRV1625u: 0.9999999776482582
      [Pipeline] echo
      we have 19422016 bytes left on SRV1625u
      [Pipeline] fileOperations
      File Delete Operation:
      [Pipeline] sleep
      Sleeping for 1 sec
      /rsync/JenkinsJobs/workspace/work_ServerCheck/build_152473_SRV1625u.txt deleting....
      Success.
      [Pipeline] writeFile
      [Pipeline] exwsAllocate
      Disk allocation strategy was not provided as step parameter. Fallback to the strategy defined in the Jenkins global config
      Using Disk allocation strategy: 'Select the Disk with the most usable space'
      Selected Disk ID 'S_Produkte__Builds' from the Disk Pool ID 'ArchivedBuilds'
      The path on Disk is: unmerged/work_ServerCheck
      [Pipeline] exws
      Searching for disk definitions in the External Workspace Templates from Jenkins global config
      Running in /media/Builds/unmerged/work_ServerCheck
      [Pipeline]
      { [Pipeline] pwd [Pipeline] fileOperations File Delete Operation: [Pipeline] sleep Sleeping for 1 sec /media/Builds/unmerged/work_ServerCheck/build_152473_SRV1625u.txt deleting.... Success. [Pipeline] }

      [Pipeline] // exws
      [Pipeline] fileOperations
      File Copy Operation:
      [Pipeline] sleep
      Sleeping for 1 sec
      /rsync/JenkinsJobs/workspace/work_ServerCheck/build_152474_SRV1625u.txt
      [Pipeline] exws
      Searching for disk definitions in the External Workspace Templates from Jenkins global config
      Running in /media/Builds/unmerged/work_ServerCheck
      [Pipeline]
      { [Pipeline] fileExists [Pipeline] }

      [Pipeline] // exws
      [Pipeline] sleep
      Sleeping for 1 sec
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] stage
      [Pipeline] { (SRV4708)
      [Pipeline] sleep
      Sleeping for 1 sec
      [Pipeline] node
      Running on SRV4708 in /rsync/JenkinsJobs/workspace/work_ServerCheck
      [Pipeline]

      { [Pipeline] sleep Sleeping for 1 sec [Pipeline] isUnix [Pipeline] sh + df /rsync [Pipeline] sh + uptime [Pipeline] isUnix [Pipeline] isUnix [Pipeline] echo SRV4708: 0.0 [Pipeline] echo we have 363949504 bytes left on SRV4708 [Pipeline] fileOperations File Delete Operation: [Pipeline] sleep Sleeping for 1 sec /rsync/JenkinsJobs/workspace/work_ServerCheck/build_152473_SRV4708.txt deleting.... Success. [Pipeline] writeFile [Pipeline] exwsAllocate Disk allocation strategy was not provided as step parameter. Fallback to the strategy defined in the Jenkins global config Using Disk allocation strategy: 'Select the Disk with the most usable space' Selected Disk ID 'S_Produkte__Builds' from the Disk Pool ID 'ArchivedBuilds' The path on Disk is: unmerged/work_ServerCheck [Pipeline] exws Searching for disk definitions in the External Workspace Templates from Jenkins global config Running in /media/Builds/unmerged/work_ServerCheck [Pipeline] \{ [Pipeline] pwd [Pipeline] fileOperations File Delete Operation: [Pipeline] sleep Sleeping for 1 sec /media/Builds/unmerged/work_ServerCheck/build_152473_SRV4708.txt deleting.... Success. [Pipeline] }

      [Pipeline] // exws
      [Pipeline] fileOperations
      File Copy Operation:
      [Pipeline] sleep
      Sleeping for 1 sec
      /rsync/JenkinsJobs/workspace/work_ServerCheck/build_152474_SRV4708.txt
      [Pipeline] exws
      Searching for disk definitions in the External Workspace Templates from Jenkins global config
      Running in /media/Builds/unmerged/work_ServerCheck
      [Pipeline] {[Pipeline] fileExists[Pipeline] }

      [Pipeline] // exws
      [Pipeline] sleep
      Sleeping for 1 sec
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] stage
      [Pipeline] { (domvsafb1)
      [Pipeline] sleep
      Sleeping for 1 sec
      [Pipeline] node
      Running on domvsafb1 in e:\jenkins\workspace\work_ServerCheck
      [Pipeline] {
      [Pipeline] sleep
      Sleeping for 1 sec
      [Pipeline] isUnix
      [Pipeline] sh
      + df /cygdrive/e/jenkins/workspace/
      [Pipeline] sh
      + wmic cpu get loadpercentage
      [Pipeline] isUnix
      [Pipeline] isUnix
      [Pipeline] echo
      domvsafb1: 12.0
      [Pipeline] echo
      we have 12600340 bytes left on domvsafb1
      [Pipeline] fileOperations
      File Delete Operation:
      [Pipeline] sleep
      Sleeping for 1 sec
      e:\jenkins\workspace\work_ServerCheck\build_152473_domvsafb1.txt deleting....
      Success.
      [Pipeline] writeFile
      [Pipeline] exwsAllocate
      Disk allocation strategy was not provided as step parameter. Fallback to the strategy defined in the Jenkins global config
      Using Disk allocation strategy: 'Select the Disk with the most usable space'
      Selected Disk ID 'S_Produkte__Builds' from the Disk Pool ID 'ArchivedBuilds'
      The path on Disk is: unmerged/work_ServerCheck
      [Pipeline] exws
      Searching for disk definitions in the External Workspace Templates from Jenkins global config
      Searching for disk definitions in the Node config
      Running in \\servername_Builds\unmerged\work_ServerCheck
      [Pipeline]

      { [Pipeline] pwd [Pipeline] fileOperations File Delete Operation: [Pipeline] sleep Sleeping for 1 sec \\servername\_Builds\unmerged\work_ServerCheck\build_152473_domvsafb1.txt deleting.... Success. [Pipeline] }

      [Pipeline] // exws
      [Pipeline] fileOperations
      File Copy Operation:
      [Pipeline] sleep
      Sleeping for 1 sec
      e:\jenkins\workspace\work_ServerCheck\build_152474_domvsafb1.txt
      [Pipeline] exws
      Searching for disk definitions in the External Workspace Templates from Jenkins global config
      Searching for disk definitions in the Node config
      Running in \\servername_Builds\unmerged\work_ServerCheck
      [Pipeline]

      { [Pipeline] fileExists [Pipeline] }

      [Pipeline] // exws
      [Pipeline] sleep
      Sleeping for 1 sec
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] stage
      [Pipeline] { (domvsafb2)
      [Pipeline] sleep
      Sleeping for 1 sec
      [Pipeline] node
      Running on domvsafb2 in e:\jenkins\workspace\work_ServerCheck
      [Pipeline]

      { [Pipeline] sleep Sleeping for 1 sec [Pipeline] isUnix [Pipeline] sh + df /cygdrive/e/jenkins/workspace/ [Pipeline] sh + wmic cpu get loadpercentage [Pipeline] isUnix [Pipeline] isUnix [Pipeline] echo domvsafb2: 20.0 [Pipeline] echo we have 6471024 bytes left on domvsafb2 [Pipeline] fileOperations File Delete Operation: [Pipeline] sleep Sleeping for 1 sec e:\jenkins\workspace\work_ServerCheck\build_152473_domvsafb2.txt deleting.... Success. [Pipeline] writeFile [Pipeline] exwsAllocate Disk allocation strategy was not provided as step parameter. Fallback to the strategy defined in the Jenkins global config Using Disk allocation strategy: 'Select the Disk with the most usable space' [Pipeline] End of Pipeline java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:73) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:263) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:250) Caused: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.ewm.actions.ExwsAllocateActionImpl#allocatedWorkspaces for class org.jenkinsci.plugins.ewm.actions.ExwsAllocateActionImpl at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:254) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:222) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:208) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:149) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88) at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64) at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:263) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:250) Caused: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:254) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:222) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:208) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:149) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015) at com.thoughtworks.xstream.XStream.toXML(XStream.java:988) at hudson.util.XStream2.toXMLUTF8(XStream2.java:313) at org.jenkinsci.plugins.workflow.support.PipelineIOUtils.writeByXStream(PipelineIOUtils.java:34) at org.jenkinsci.plugins.workflow.job.WorkflowRun.save(WorkflowRun.java:1137) at hudson.BulkChange.commit(BulkChange.java:98) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1475) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:37) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Finished: FAILURE [Pipeline] exws Searching for disk definitions in the External Workspace Templates from Jenkins global config Searching for disk definitions in the Node config Running in \\servername_Builds\unmerged\work_ServerCheck [Pipeline] \{ [Pipeline] pwd (show) [Pipeline] fileOperations File Delete Operation: [Pipeline] sleep Sleeping for 1 sec \\servername\_Builds\unmerged\work_ServerCheck\build_152473_domvsafb2.txt deleting.... Success. [Pipeline] }

      [Pipeline] // exws
      [Pipeline] fileOperations
      File Copy Operation:
      [Pipeline] sleep
      Sleeping for 1 sec
      e:\jenkins\workspace\work_ServerCheck\build_152474_domvsafb2.txt
      [Pipeline] exws
      Searching for disk definitions in the External Workspace Templates from Jenkins global config
      Searching for disk definitions in the Node config
      Running in \\servername_Builds\unmerged\work_ServerCheck
      [Pipeline] {[Pipeline] fileExists[Pipeline] }

      [Pipeline] // exws
      [Pipeline] sleep
      Sleeping for 1 sec
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] node
      Running on SRV4708 in /rsync/JenkinsJobs/workspace/work_ServerCheck
      [Pipeline]

      { [Pipeline] fileOperations File Create Operation: [Pipeline] plot [Pipeline] fileOperations File Create Operation: [Pipeline] plot [Pipeline] archiveArtifacts Archiving artifacts [Pipeline] logParser [Pipeline] }

      [Pipeline] // node
      data_load.csv file already exists, replacing the content with the provided content.
      Creating file: /rsync/JenkinsJobs/workspace/work_ServerCheck/data_load.csv
      data_disk.csv file already exists, replacing the content with the provided content.
      Creating file: /rsync/JenkinsJobs/workspace/work_ServerCheck/data_disk.csv

            alexsomai Alexandru Somai
            romanz Roman Zwi
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: