Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49006

ConcurrentModificationException in getThrottledPipelineRunsForCategory

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Labels:
      None
    • Environment:
    • Similar Issues:
    • Released As:
      2.0.2

      Description

      I started two builds of a pipeline job which uses the throttle step against a category. The category is configured to permit a maximum of one build at a time. The second build hit the throttled node step, waited for somewhere around 30–40 minutes, and then started executing anyway, despite the first build not having exited the throttled node step yet. I found this in my Jenkins log:

      SEVERE: Timer task hudson.model.Queue$MaintainTask@63d577c failed
      java.util.ConcurrentModificationException
      	at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1211)
      	at java.util.TreeMap$EntryIterator.next(TreeMap.java:1247)
      	at java.util.TreeMap$EntryIterator.next(TreeMap.java:1242)
      	at hudson.plugins.throttleconcurrents.ThrottleJobProperty.getThrottledPipelineRunsForCategory(ThrottleJobProperty.java:316)
      	at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.throttleCheckForCategoriesOnNode(ThrottleQueueTaskDispatcher.java:132)
      	at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canTakeImpl(ThrottleQueueTaskDispatcher.java:101)
      	at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canTake(ThrottleQueueTaskDispatcher.java:55)
      	at hudson.model.queue.QueueTaskDispatcher.canTake(QueueTaskDispatcher.java:102)
      	at hudson.model.Queue$JobOffer.getCauseOfBlockage(Queue.java:273)
      	at hudson.model.Queue.maintain(Queue.java:1597)
      	at hudson.model.Queue$MaintainTask.doRun(Queue.java:2833)
      	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
      	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      

        Attachments

          Activity

          haw777 Christopher Head created issue -
          Hide
          sk_nano Sascha Kiedrowski added a comment -

          We're having the same problem here.

          OS: Windows 2012 R2, Windows 2016, Jenkins 2.89.3, TCB 2.0.1

          Show
          sk_nano Sascha Kiedrowski added a comment - We're having the same problem here. OS: Windows 2012 R2, Windows 2016, Jenkins 2.89.3, TCB 2.0.1
          Hide
          sk_nano Sascha Kiedrowski added a comment -

          I've created a pull request to fix this issue:

          https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/54

          Show
          sk_nano Sascha Kiedrowski added a comment - I've created a pull request to fix this issue: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/54
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          Unfortunately I was unable to find time to work on plugin over last year. I decided to step down as maintainer and to mark the plugin for adoption.

          If anybody is interested, let me know

          Show
          oleg_nenashev Oleg Nenashev added a comment - Unfortunately I was unable to find time to work on plugin over last year. I decided to step down as maintainer and to mark the plugin for adoption. If anybody is interested, let me know
          oleg_nenashev Oleg Nenashev made changes -
          Field Original Value New Value
          Assignee Oleg Nenashev [ oleg_nenashev ]
          Hide
          medianick Nick Jones added a comment -

          We were bitten by this today; a PR build (on a multibranch pipeline job) started freezing – appearing to still be in progress in the UI, but not assigned to any executor, and never progressing past the node step, with this in the Jenkins system log:

          Apr 23, 2019 7:36:18 AM WARNING jenkins.util.AtmostOneTaskExecutor$1 call
          null
          java.util.ConcurrentModificationException
          	at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1211)
          	at java.util.TreeMap$EntryIterator.next(TreeMap.java:1247)
          	at java.util.TreeMap$EntryIterator.next(TreeMap.java:1242)
          	at hudson.plugins.throttleconcurrents.ThrottleJobProperty.getThrottledPipelineRunsForCategory(ThrottleJobProperty.java:316)
          	at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.throttleCheckForCategoriesAllNodes(ThrottleQueueTaskDispatcher.java:283)
          	at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRunImpl(ThrottleQueueTaskDispatcher.java:256)
          	at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRun(ThrottleQueueTaskDispatcher.java:218)
          	at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRun(ThrottleQueueTaskDispatcher.java:176)
          	at hudson.model.Queue.getCauseOfBlockageForItem(Queue.java:1197)
          	at hudson.model.Queue.maintain(Queue.java:1554)
          	at hudson.model.Queue$1.call(Queue.java:325)
          	at hudson.model.Queue$1.call(Queue.java:322)
          	at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:108)
          	at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:98)
          	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:112)
          	at java.lang.Thread.run(Thread.java:748)
          

          I tried aborting and restarting it multiple times but it consistently behaved the same way; the only way we got it to build again was by commenting out the throttle block entirely. It would be marginally better if it had simply failed the build, but far better if this ConcurrentModificationException were not happening at all. Incidentially, this PR was a switch to use Scripted Pipeline due to JENKINS-45140, since Declarative Syntax didn't seem to throttle at all.

          Show
          medianick Nick Jones added a comment - We were bitten by this today; a PR build (on a multibranch pipeline job) started freezing – appearing to still be in progress in the UI, but not assigned to any executor, and never progressing past the node step, with this in the Jenkins system log: Apr 23, 2019 7:36:18 AM WARNING jenkins.util.AtmostOneTaskExecutor$1 call null java.util.ConcurrentModificationException at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1211) at java.util.TreeMap$EntryIterator.next(TreeMap.java:1247) at java.util.TreeMap$EntryIterator.next(TreeMap.java:1242) at hudson.plugins.throttleconcurrents.ThrottleJobProperty.getThrottledPipelineRunsForCategory(ThrottleJobProperty.java:316) at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.throttleCheckForCategoriesAllNodes(ThrottleQueueTaskDispatcher.java:283) at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRunImpl(ThrottleQueueTaskDispatcher.java:256) at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRun(ThrottleQueueTaskDispatcher.java:218) at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRun(ThrottleQueueTaskDispatcher.java:176) at hudson.model.Queue.getCauseOfBlockageForItem(Queue.java:1197) at hudson.model.Queue.maintain(Queue.java:1554) at hudson.model.Queue$1.call(Queue.java:325) at hudson.model.Queue$1.call(Queue.java:322) at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:108) at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:98) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:112) at java.lang.Thread.run(Thread.java:748) I tried aborting and restarting it multiple times but it consistently behaved the same way; the only way we got it to build again was by commenting out the throttle block entirely. It would be marginally better if it had simply failed the build, but far better if this ConcurrentModificationException were not happening at all. Incidentially, this PR was a switch to use Scripted Pipeline due to JENKINS-45140 , since Declarative Syntax didn't seem to throttle at all.
          Hide
          basil Basil Crow added a comment -

          Christopher Head, Sascha Kiedrowski, and Nick Jones, this issue should be resolved on master and I am preparing a release of Throttle Concurrent Builds with the fix for this issue. There is an incremental build available here. Are any of you interested in testing this incremental build before I do an official release?

          Show
          basil Basil Crow added a comment - Christopher Head , Sascha Kiedrowski , and Nick Jones , this issue should be resolved on master and I am preparing a release of Throttle Concurrent Builds with the fix for this issue. There is an incremental build available here . Are any of you interested in testing this incremental build before I do an official release?
          Hide
          haw777 Christopher Head added a comment -

          Sorry, I can’t test it. I switched to Lockable Resources a long time ago, initially to fix this bug but now we use some other features too.

          Show
          haw777 Christopher Head added a comment - Sorry, I can’t test it. I switched to Lockable Resources a long time ago, initially to fix this bug but now we use some other features too.
          Hide
          medianick Nick Jones added a comment - - edited

          Basil Crow Yes, I'd be happy to get it tested on our side. We too had switched to Lockable Resources (like Christopher Head), though in our case was a bit of a kludge (to achieve throttling using a predefined quantity of resources that share the same category), so I'd be happy to retry with this new incremental build on our test Jenkins system. I'll grab it and post back here with my findings.

          Show
          medianick Nick Jones added a comment - - edited Basil Crow Yes, I'd be happy to get it tested on our side. We too had switched to Lockable Resources (like Christopher Head ), though in our case was a bit of a kludge (to achieve throttling using a predefined quantity of resources that share the same category), so I'd be happy to retry with this new incremental build on our test Jenkins system. I'll grab it and post back here with my findings.
          basil Basil Crow made changes -
          Assignee Basil Crow [ basil ]
          Hide
          basil Basil Crow added a comment -
          Show
          basil Basil Crow added a comment - Fixed in jenkinsci/throttle-concurrent-builds-plugin#58 .
          basil Basil Crow made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Fixed but Unreleased [ 10203 ]
          basil Basil Crow made changes -
          Remote Link This issue links to "jenkinsci/throttle-concurrent-builds-plugin#58 (Web Link)" [ 24428 ]
          Hide
          basil Basil Crow added a comment -
          Show
          basil Basil Crow added a comment - Released in Throttle Concurrent Builds 2.0.2 .
          basil Basil Crow made changes -
          Released As 2.0.2
          Status Fixed but Unreleased [ 10203 ] Resolved [ 5 ]

            People

            Assignee:
            basil Basil Crow
            Reporter:
            haw777 Christopher Head
            Votes:
            5 Vote for this issue
            Watchers:
            9 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: