Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44597

ConcurrentModificationException on releasing lock

    XMLWordPrintable

Details

    • Bug
    • Status: Reopened (View Workflow)
    • Major
    • Resolution: Unresolved
    • None
    • Jenkins ver. 2.46.2 (LTS)
      Java 1.8.0_131-b11 (Oracle)
      Lockable Resources plugin 2.0
      Pipeline 2.5
    • lockable-resources 2.1

    Description

      We are using the lockable resources plugin extensively in our Jenkins Pipeline Scripts, today one of our Builds failed altough the actual build process was successful. The following error is shown in the log.

      [lockable-resources] released lock on [LOCK]
      java.util.ConcurrentModificationException
      	at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
      	at java.util.ArrayList$Itr.next(ArrayList.java:851)
      	at org.jenkins.plugins.lockableresources.LockableResourcesManager.getNextQueuedContext(LockableResourcesManager.java:445)
      	at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlockNames(LockableResourcesManager.java:351)
      	at org.jenkins.plugins.lockableresources.LockStepExecution$Callback.finished(LockStepExecution.java:84)
      	at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onSuccess(BodyExecutionCallback.java:114)
      	at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:362)
      	at com.cloudbees.groovy.cps.Outcome.resumeFrom(Outcome.java:73)
      	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:146)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
      	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:165)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:330)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:242)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:230)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:748)

      We are holding onto best practices for locks, to not use them from within node blocks, but outside so no executor is blocked as long as we wait for the lock:

      lock(label:'lock-group'){
          node {
              // here comes the build
          }
      }

      hope that helps.

      Our build infrastructure consists of two servers (a master with 1 executor) and a slave (with 3 executors).

      One thing we also noticed is that locking and unlocking (even when using the UI) is often super slow (taking up to a minute), but that may be another problem.

      Attachments

        Issue Links

          Activity

            dageissl Daniel Geißler added a comment - May this be a bug adressed by: https://github.com/jenkinsci/lockable-resources-plugin/pull/57  ?
            marksyms Mark Syms added a comment -

            It certainly looks like it.

            marksyms Mark Syms added a comment - It certainly looks like it.

            This seems to have been addressed in release 2.1, please reopen with new evidence if it is still an issue.

            tgr Tobias Gruetzmacher added a comment - This seems to have been addressed in release 2.1, please reopen with new evidence if it is still an issue.

            We encountered a similar issue with version 2.7. It was only one incident out of hundreds of runs, but might as well keep this ticket open if it can still happen.

            [Pipeline] Start of Pipeline
            [Pipeline] lock
            Trying to acquire lock on [rcprod]
            Resource [rcprod] did not exist. Created.
            [Pipeline] // lock
            [Pipeline] End of Pipeline
            java.util.ConcurrentModificationException
            	at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
            	at java.util.ArrayList$Itr.next(ArrayList.java:859)
            	at org.jenkins.plugins.lockableresources.LockableResourcesManager.fromName(LockableResourcesManager.java:182)
            	at org.jenkins.plugins.lockableresources.queue.LockableResourcesStruct.<init>(LockableResourcesStruct.java:83)
            	at org.jenkins.plugins.lockableresources.LockStepExecution.start(LockStepExecution.java:56)
            	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:286)
            	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:179)
            	... 

            We too are putting the lock outside the node:

            lock("rcprod") {
                node() {
                    ...
                }
            }

            Versions:

            Jenkins 2.225
            Lockable Resources plugin 2.7
            Pipeline 2.6
            Java openjdk version "1.8.0_212" 
            jsmilani John Smilanick added a comment - We encountered a similar issue with version 2.7. It was only one incident out of hundreds of runs, but might as well keep this ticket open if it can still happen. [Pipeline] Start of Pipeline [Pipeline] lock Trying to acquire lock on [rcprod] Resource [rcprod] did not exist. Created. [Pipeline] // lock [Pipeline] End of Pipeline java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909) at java.util.ArrayList$Itr.next(ArrayList.java:859) at org.jenkins.plugins.lockableresources.LockableResourcesManager.fromName(LockableResourcesManager.java:182) at org.jenkins.plugins.lockableresources.queue.LockableResourcesStruct.<init>(LockableResourcesStruct.java:83) at org.jenkins.plugins.lockableresources.LockStepExecution.start(LockStepExecution.java:56) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:286) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:179) ... We too are putting the lock outside the node: lock( "rcprod" ) { node() { ... } } Versions: Jenkins 2.225 Lockable Resources plugin 2.7 Pipeline 2.6 Java openjdk version "1.8.0_212"

            I found a related issue that may have resolved this in 2.8: JENKINS-57727

            I will update our plugins and see if I can verify it fixes my issue.

            jsmilani John Smilanick added a comment - I found a related issue that may have resolved this in 2.8:  JENKINS-57727 I will update our plugins and see if I can verify it fixes my issue.

            People

              Unassigned Unassigned
              dageissl Daniel Geißler
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: