Jenkins had stopped responding to browser requests for Jenkins pages and I think it may be caused by the recent upgrade to Throttle Concurrent Builds 1.8.1
Requests were getting blocked waiting on 0x00000004181e4520
"Handling GET /jenkins/ : RequestHandlerThread[#171]" daemon prio=10 tid=0x00000000168ee800 nid=0x193b waiting for monitor entry [0x000000004335b000]
java.lang.Thread.State: BLOCKED (on object monitor)
at hudson.model.Queue.getItems(Queue.java:687)
- waiting to lock <0x00000004181e4520> (a hudson.model.Queue)
at hudson.model.Queue$CachedItemList.get(Queue.java:216)
at hudson.model.Queue.getApproximateItemsQuickly(Queue.java:717)
at hudson.model.View.getApproximateQueueItemsQuickly(View.java:483)
at sun.reflect.GeneratedMethodAccessor355.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
This seems to be waiting on Throttle Concurrent Builds code.
Further dumps after 10 minutes, 20 minutes and 30 minutes showed this same stack trace.
"Thread-126" daemon prio=10 tid=0x00002aaae0529800 nid=0x1785 runnable [0x0000000046590000]
java.lang.Thread.State: RUNNABLE
at java.util.WeakHashMap$HashIterator.hasNext(WeakHashMap.java:875)
at java.util.AbstractCollection.toArray(AbstractCollection.java:139)
at java.util.ArrayList.<init>(ArrayList.java:164)
at hudson.plugins.throttleconcurrents.ThrottleJobProperty.getCategoryProjects(ThrottleJobProperty.java:141)
- locked <0x000000041a79b778> (a java.util.HashMap)
at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRun(ThrottleQueueTaskDispatcher.java:118)
at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRun(ThrottleQueueTaskDispatcher.java:90)
at hudson.model.Queue.isBuildBlocked(Queue.java:937)
at hudson.model.Queue.maintain(Queue.java:1006)
- locked <0x00000004181e4520> (a hudson.model.Queue)
at hudson.model.Queue$1.call(Queue.java:303)
at hudson.model.Queue$1.call(Queue.java:300)
at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:69)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:104)
at java.lang.Thread.run(Thread.java:724)
Locked ownable synchronizers:
- None
CPU usage was at ~100% for this thread for the 30 minutes that I was watching it before I restarted Jenkins.
(6021 = 0x1785)
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6021 rcbuild_ 35 10 18.0g 6.2g 32m R 99.7 19.7 30:45.97 java
I have rolled back to Throttle Concurrent Builds 1.8 for now.
Still learning how to investigate thread dumps but please let me know if there is anything I can do to help.
Bug
Blocker
