Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54480

ConcurrentModificationException in CPS VM thread (CpsFlowExecution): org.jenkinsci.plugins.workflow.cps.DSL

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Duplicate
    • Component/s: workflow-cps-plugin
    • Labels:
      None
    • Environment:
      Jenkins (Core) 2.149
      Pipeline Groovy (i.e. workflow-cps) Plugin 2.60
    • Similar Issues:

      Description

      This happened just once so far; about two days after updating a set of Jenkins plugins (but I am not sure at all, if the CME is related to the update), e.g.:

      • workflow-api:2.31 (was: 2.29)
      • workflow-basic-steps:2.12 (was: 2.11)
      • workflow-cps:2.60 (was: 2.57)
      • workflow-durable-task-step:2.25 (was: 2.22)
      • workflow-support:2.21 (was: 2.20)

      This is the stacktrace from Jenkins console log right before the end of the pipeline build:

      2018-11-02 02:22:19 WARNING [org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem]   Unexpected exception in CPS VM thread: CpsFlowExecution[Owner[ACME-DB/174:ACME-DB
      #174]]
      java.util.ConcurrentModificationException
              at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
              at java.util.ArrayList$Itr.next(ArrayList.java:859)
              at org.jenkinsci.plugins.workflow.cps.DSL$ThreadTaskImpl.invokeBody(DSL.java:653)
              at org.jenkinsci.plugins.workflow.cps.DSL$ThreadTaskImpl.eval(DSL.java:623)
              at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:192)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
              at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
              at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
              at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      

      So the bulk of the build is already finished, there is just one kind of build KPI stage (in a finally block) that does some build stats analysis. So normally (successful) the build finishes like this:

      ...
      [Pipeline] stage
      [Pipeline] { (kpi)
      [Pipeline] echo
      Listing all KPIs...
      [Pipeline] echo
      
      [Pipeline] echo
      Writing new KPI file...
      [Pipeline] node
      Running on Jenkins in /var/lib/jenkins/workspace/ACME-DB
      [Pipeline] {
      [Pipeline] ws
      Running in /var/lib/jenkins/workspace/ACME-DB-WS-kpi
      [Pipeline] {
      [Pipeline] echo
      Build Pipeline KPIs:
        - Build duration: 51 minutes, 37.684 seconds ... time duration since build start
        - Time duration since build scheduling: 51 minutes, 37.692 seconds
        - Pipeline net execution time duration: 51 minutes, 26.809 seconds ... in "acme.build"
      Found no SVN changeset => no cycle time KPI
      [Pipeline] pwd
      [Pipeline] plot
      [Pipeline] }
      [Pipeline] // ws
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] End of Pipeline
      Finished: SUCCESS

      And in this failure case it finished like this:

      ...
      [Pipeline] stage
      [Pipeline] { (kpi)
      [Pipeline] echo
      Listing all KPIs...
      [Pipeline] echo
      
      [Pipeline] echo
      Writing new KPI file...
      [Pipeline] node
      Running on Jenkins in /var/lib/jenkins/workspace/ACME-DB
      [Pipeline] End of Pipeline
      java.util.ConcurrentModificationException
      	at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
      	at java.util.ArrayList$Itr.next(ArrayList.java:859)
      	at org.jenkinsci.plugins.workflow.cps.DSL$ThreadTaskImpl.invokeBody(DSL.java:653)
      	at org.jenkinsci.plugins.workflow.cps.DSL$ThreadTaskImpl.eval(DSL.java:623)
      	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:192)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Finished: FAILURE

      And the corresponding code is mostly "hidden" in a shared pipeline library:

        ...
      } finally {
        script.stage('kpi') {
          buildContext.kpiManager.complete(currentBuild)
      
          script.node {
            acme.ws(deleteAfterUse: true, locationSuffix: '-kpi') {
              buildContext.buildKpi.setBuildEnd()
              buildContext.buildKpi.report()
            }
          }
        }
      }
      

        Attachments

          Issue Links

            Activity

            Hide
            dnusbaum Devin Nusbaum added a comment -

            JENKINS-58878 was also caused by concurrency-related issues with DSL$ThreadTaskImpl.invokeBody (e.g. ArrayIndexOutOfBoundsException from the same code), and was fixed by adding synchronization so I suspect that Pipeline: Groovy plugin 2.76 might have fixed this issue.

            I am going to go ahead and speculatively close this as a duplicate of that issue since it has not had any activity since it was reported, but if anyone sees it when running Pipeline: Groovy 2.76 or newer, feel free to reopen the issue.

            Show
            dnusbaum Devin Nusbaum added a comment - JENKINS-58878 was also caused by concurrency-related issues with DSL$ThreadTaskImpl.invokeBody (e.g. ArrayIndexOutOfBoundsException from the same code), and was fixed by adding synchronization so I suspect that Pipeline: Groovy plugin 2.76 might have fixed this issue. I am going to go ahead and speculatively close this as a duplicate of that issue since it has not had any activity since it was reported, but if anyone sees it when running Pipeline: Groovy 2.76 or newer, feel free to reopen the issue.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              reinholdfuereder Reinhold F├╝reder
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: