-
Bug
-
Resolution: Fixed
-
Blocker
-
Jenkins 1.544
Jenkins had stopped responding to browser requests for Jenkins pages and I think it may be caused by the recent upgrade to Throttle Concurrent Builds 1.8.1
Requests were getting blocked waiting on 0x00000004181e4520
"Handling GET /jenkins/ : RequestHandlerThread[#171]" daemon prio=10 tid=0x00000000168ee800 nid=0x193b waiting for monitor entry [0x000000004335b000] java.lang.Thread.State: BLOCKED (on object monitor) at hudson.model.Queue.getItems(Queue.java:687) - waiting to lock <0x00000004181e4520> (a hudson.model.Queue) at hudson.model.Queue$CachedItemList.get(Queue.java:216) at hudson.model.Queue.getApproximateItemsQuickly(Queue.java:717) at hudson.model.View.getApproximateQueueItemsQuickly(View.java:483) at sun.reflect.GeneratedMethodAccessor355.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
This seems to be waiting on Throttle Concurrent Builds code.
Further dumps after 10 minutes, 20 minutes and 30 minutes showed this same stack trace.
"Thread-126" daemon prio=10 tid=0x00002aaae0529800 nid=0x1785 runnable [0x0000000046590000] java.lang.Thread.State: RUNNABLE at java.util.WeakHashMap$HashIterator.hasNext(WeakHashMap.java:875) at java.util.AbstractCollection.toArray(AbstractCollection.java:139) at java.util.ArrayList.<init>(ArrayList.java:164) at hudson.plugins.throttleconcurrents.ThrottleJobProperty.getCategoryProjects(ThrottleJobProperty.java:141) - locked <0x000000041a79b778> (a java.util.HashMap) at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRun(ThrottleQueueTaskDispatcher.java:118) at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRun(ThrottleQueueTaskDispatcher.java:90) at hudson.model.Queue.isBuildBlocked(Queue.java:937) at hudson.model.Queue.maintain(Queue.java:1006) - locked <0x00000004181e4520> (a hudson.model.Queue) at hudson.model.Queue$1.call(Queue.java:303) at hudson.model.Queue$1.call(Queue.java:300) at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:69) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:104) at java.lang.Thread.run(Thread.java:724) Locked ownable synchronizers: - None
CPU usage was at ~100% for this thread for the 30 minutes that I was watching it before I restarted Jenkins.
(6021 = 0x1785)
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6021 rcbuild_ 35 10 18.0g 6.2g 32m R 99.7 19.7 30:45.97 java
I have rolled back to Throttle Concurrent Builds 1.8 for now.
Still learning how to investigate thread dumps but please let me know if there is anything I can do to help.
- is blocking
-
-
- Resolved
-
- is duplicated by
-
-
- Resolved
-
- is related to
-
-
- Resolved
-
[JENKINS-21044] Throttle Concurrent Builds blocking Jenkins queue
| Assignee | Original: Andrew Bayer [ abayer ] | New: Oleg Nenashev [ oleg_nenashev ] |
BTW, I confirm that the current implementation may cause huge calculation efforts in the case of big number of jobs with a same category:
- keySet() causes a useless copy of the data
- copy to a new array leads to an additional copy of items
It makes sense to extend a the cache's lock time, but to iterate the HashMap directly.
I'll reassign issue to Jesse Glick, who is an author of https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/10
Jesse, do you have some time to fix the issue? If no, I can do it on my own
| Assignee | Original: Oleg Nenashev [ oleg_nenashev ] | New: Jesse Glick [ jglick ] |
| Link |
New:
This issue is blocking |
| Labels | New: performance |
I do not actually run this plugin anywhere so I have no real way of confirming whether a given change improves or degrades performance on a large installation.
Extending the scope of the lock on propertiesByCategory may avoid overhead, but runs the risk of deadlocks, if foreign calls such as getItem are included.
| Assignee | Original: Jesse Glick [ jglick ] | New: Oleg Nenashev [ oleg_nenashev ] |
BTW the actual thread dump shows an issue in WeakHashMap.HashIterator.hasNext, such as an endless loop. The keySet method does not copy any data, and there is no indication that the array copy overhead is relevant. If I am right, iterating the map directly would not make any difference.
What JRE is being used to run Jenkins? If not the most recent Java 7, try updating and see if the issue remains.
Could you briefly describe your case (number of projects, categories structure, frequency of job submissions, etc.)?