Throttle Concurrent Builds blocking Jenkins queue

This issue is archived. You can view it, but you can't modify it. Learn more

XMLWordPrintable

      Jenkins had stopped responding to browser requests for Jenkins pages and I think it may be caused by the recent upgrade to Throttle Concurrent Builds 1.8.1

      Requests were getting blocked waiting on 0x00000004181e4520

      "Handling GET /jenkins/ : RequestHandlerThread[#171]" daemon prio=10 tid=0x00000000168ee800 nid=0x193b waiting for monitor entry [0x000000004335b000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at hudson.model.Queue.getItems(Queue.java:687)
	- waiting to lock <0x00000004181e4520> (a hudson.model.Queue)
	at hudson.model.Queue$CachedItemList.get(Queue.java:216)
	at hudson.model.Queue.getApproximateItemsQuickly(Queue.java:717)
	at hudson.model.View.getApproximateQueueItemsQuickly(View.java:483)
	at sun.reflect.GeneratedMethodAccessor355.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

      This seems to be waiting on Throttle Concurrent Builds code.
      Further dumps after 10 minutes, 20 minutes and 30 minutes showed this same stack trace.

      "Thread-126" daemon prio=10 tid=0x00002aaae0529800 nid=0x1785 runnable [0x0000000046590000]
   java.lang.Thread.State: RUNNABLE
	at java.util.WeakHashMap$HashIterator.hasNext(WeakHashMap.java:875)
	at java.util.AbstractCollection.toArray(AbstractCollection.java:139)
	at java.util.ArrayList.<init>(ArrayList.java:164)
	at hudson.plugins.throttleconcurrents.ThrottleJobProperty.getCategoryProjects(ThrottleJobProperty.java:141)
	- locked <0x000000041a79b778> (a java.util.HashMap)
	at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRun(ThrottleQueueTaskDispatcher.java:118)
	at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRun(ThrottleQueueTaskDispatcher.java:90)
	at hudson.model.Queue.isBuildBlocked(Queue.java:937)
	at hudson.model.Queue.maintain(Queue.java:1006)
	- locked <0x00000004181e4520> (a hudson.model.Queue)
	at hudson.model.Queue$1.call(Queue.java:303)
	at hudson.model.Queue$1.call(Queue.java:300)
	at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:69)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:104)
	at java.lang.Thread.run(Thread.java:724)

   Locked ownable synchronizers:
	- None

      CPU usage was at ~100% for this thread for the 30 minutes that I was watching it before I restarted Jenkins.
      (6021 = 0x1785)

        PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                   
 6021 rcbuild_  35  10 18.0g 6.2g  32m R 99.7 19.7  30:45.97 java

      I have rolled back to Throttle Concurrent Builds 1.8 for now.

      Still learning how to investigate thread dumps but please let me know if there is anything I can do to help.

        1. throttle-concurrents.hpi
          26 kB

            Assignee:
            Oleg Nenashev
            Reporter:
            Geoff Cummings
            Archiver:
            Jenkins Service Account

              Created:
              Updated:
              Resolved:
              Archived: