-
Bug
-
Resolution: Fixed
-
Major
-
metrics plugin 4.0.2.7
During performance testing, we see that the number of thread that always increases, when it should not as the number of concurrent user is stable (= 1) and the job queue is also a fixed number.
Looking at thread dump, hundred of thread are:
"QueueSubTaskMetrics [#6]" Id=350 Group=main WAITING on hudson.model.queue.FutureImpl@6f7dbdac"QueueSubTaskMetrics [#6]" Id=350 Group=main WAITING on hudson.model.queue.FutureImpl@6f7dbdac at java.lang.Object.wait(Native Method) - waiting on hudson.model.queue.FutureImpl@6f7dbdac at java.lang.Object.wait(Object.java:502) at hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:79) at jenkins.metrics.impl.JenkinsMetricProviderImpl.lambda$asSupplier$3(JenkinsMetricProviderImpl.java:1142) at jenkins.metrics.impl.JenkinsMetricProviderImpl$$Lambda$670/282561367.get(Unknown Source) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@22c49544
an unbounded thread pool is suspected here: https://github.com/jenkinsci/metrics-plugin/blob/21e83be64f85d343c3c9b0e0b0956021d74ade95/src/main/java/jenkins/metrics/impl/JenkinsMetricProviderImpl.java#L836-L839
jstack and top -H output see attachments.
- causes
-
JENKINS-69817 Sub Task Time in Queue is not getting captured in metrics plugin
-
- Resolved
-
- depends on
-
JENKINS-66947 Make ExecutorListener an extension point
-
- Closed
-
- is duplicated by
-
JENKINS-66941 o.e.j.server LowOnThreads recurring error, Service restart is needed
-
- Reopened
-
- links to
AFAICT a thread (or two) will be consumed for every executable. Without https://javadoc.jenkins.io/hudson/model/ExecutorListener.html being a global extension point (https://github.com/jenkinsci/jenkins/blob/6adc9c0519073b6ab72c0c3d5780d0e2ec21b294/core/src/main/java/hudson/slaves/SlaveComputer.java#L342-L349 does not suffice) I am not sure how else to fix this without losing the functionality of the queue metric. If you lengthen https://github.com/jenkinsci/metrics-plugin/blob/06a8665d291df51a3d0c10df96ccca0ecbe1f560/src/test/java/jenkins/metrics/impl/JenkinsMetricProviderImplTest.java#L96 and grep the jstack output for QueueSubTaskMetrics you will see them.