-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major
-
Component/s: leastload-plugin
-
None
-
Environment:* leastload:71.vb_585f7a_dc4f5 'Least Load plugin'
The ludicrous mode introduced `availableNodesThisRoundByLabel` which required updated to `refreshAvailableNodes`.
The new code is expensive on large controllers with many nodes and labels because of the nested loop. It loops the total of `nodes * labels` times.
I have not reproduced it myself. But a CloudBees customer's controller on at least two occasions entered a "death spiral" so to say. The new queue items keep being added. But the ephemeral nodes do not seem to be removed, threads waiting for a Queue lock. But the queue is held by the "Perisodic Jenkins queue maintenance" which seems to spend minutes in `refreshAvailableNodes`. e.g.,:
"AtmostOneTaskExecutor[Periodic Jenkins queue maintenance] [#1426051]" #11377240 [3647] daemon prio=5 os_prio=0 cpu=14329647.83ms elapsed=18110.00s tid=0x00007ffeac0d1b50 nid=3647 runnable [0x00007ffa7d80d000] java.lang.Thread.State: RUNNABLE at java.util.TreeMap.put(java.base@21.0.11/TreeMap.java:861) at java.util.TreeMap.put(java.base@21.0.11/TreeMap.java:569) at java.util.TreeSet.add(java.base@21.0.11/TreeSet.java:259) at jenkins.model.Jenkins.getLabels(Jenkins.java:2068) at org.bstick12.jenkinsci.plugins.leastload.LeastLoadBalancer.refreshAvailableNodes(LeastLoadBalancer.java:253) at org.bstick12.jenkinsci.plugins.leastload.LeastLoadBalancer.map(LeastLoadBalancer.java:141) at hudson.model.LoadBalancer$2.map(LoadBalancer.java:167) at hudson.model.Queue.maintain(Queue.java:1789) at hudson.model.Queue$1.call(Queue.java:340) at hudson.model.Queue$1.call(Queue.java:337) at jenkins.util.AtmostOneTaskExecutor.lambda$maybeRun$0(AtmostOneTaskExecutor.java:107) at jenkins.util.AtmostOneTaskExecutor$$Lambda/0x00000008010f2660.call(Unknown Source) at jenkins.security.ImpersonatingExecutorService.lambda$wrap$1(ImpersonatingExecutorService.java:75) at jenkins.security.ImpersonatingExecutorService$$Lambda/0x00000008010f2888.call(Unknown Source) at java.util.concurrent.FutureTask.run(java.base@21.0.11/FutureTask.java:317) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:121) at java.lang.Thread.runWith(java.base@21.0.11/Thread.java:1596) at java.lang.Thread.run(java.base@21.0.11/Thread.java:1583) Locked ownable synchronizers: - <0x0000000249da5468> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)