Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-66506

Ever increasing number of threads in the metrics plugin

      During performance testing, we see that the number of thread that always increases, when it should not as the number of concurrent user is stable (= 1) and the job queue is also a fixed number.

      Looking at thread dump, hundred of thread are:

      "QueueSubTaskMetrics [#6]" Id=350 Group=main WAITING on hudson.model.queue.FutureImpl@6f7dbdac"QueueSubTaskMetrics [#6]" Id=350 Group=main WAITING on hudson.model.queue.FutureImpl@6f7dbdac at java.lang.Object.wait(Native Method) -  waiting on hudson.model.queue.FutureImpl@6f7dbdac at java.lang.Object.wait(Object.java:502) at hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:79) at jenkins.metrics.impl.JenkinsMetricProviderImpl.lambda$asSupplier$3(JenkinsMetricProviderImpl.java:1142) at jenkins.metrics.impl.JenkinsMetricProviderImpl$$Lambda$670/282561367.get(Unknown Source) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@22c49544 

      an unbounded thread pool is suspected here: https://github.com/jenkinsci/metrics-plugin/blob/21e83be64f85d343c3c9b0e0b0956021d74ade95/src/main/java/jenkins/metrics/impl/JenkinsMetricProviderImpl.java#L836-L839

      jstack and top -H output see attachments.

       

        1. jstack2.txt
          3.41 MB
        2. top-dash-h.txt
          277 kB

          [JENKINS-66506] Ever increasing number of threads in the metrics plugin

          Jesse Glick added a comment -

          AFAICT a thread (or two) will be consumed for every executable. Without https://javadoc.jenkins.io/hudson/model/ExecutorListener.html being a global extension point (https://github.com/jenkinsci/jenkins/blob/6adc9c0519073b6ab72c0c3d5780d0e2ec21b294/core/src/main/java/hudson/slaves/SlaveComputer.java#L342-L349 does not suffice) I am not sure how else to fix this without losing the functionality of the queue metric. If you lengthen https://github.com/jenkinsci/metrics-plugin/blob/06a8665d291df51a3d0c10df96ccca0ecbe1f560/src/test/java/jenkins/metrics/impl/JenkinsMetricProviderImplTest.java#L96 and grep the jstack output for QueueSubTaskMetrics you will see them.

          Jesse Glick added a comment - AFAICT a thread (or two) will be consumed for every executable. Without https://javadoc.jenkins.io/hudson/model/ExecutorListener.html being a global extension point ( https://github.com/jenkinsci/jenkins/blob/6adc9c0519073b6ab72c0c3d5780d0e2ec21b294/core/src/main/java/hudson/slaves/SlaveComputer.java#L342-L349 does not suffice) I am not sure how else to fix this without losing the functionality of the queue metric. If you lengthen https://github.com/jenkinsci/metrics-plugin/blob/06a8665d291df51a3d0c10df96ccca0ecbe1f560/src/test/java/jenkins/metrics/impl/JenkinsMetricProviderImplTest.java#L96 and grep the jstack output for QueueSubTaskMetrics you will see them.

            jglick Jesse Glick
            megathaum megathaum
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: