Details
-
Type:
Bug
-
Status: Resolved (View Workflow)
-
Priority:
Critical
-
Resolution: Fixed
-
Component/s: core
-
Labels:
-
Environment:Metrics 4.0.2.5
-
Similar Issues:
-
Released As:jenkins-2.205
Description
In one large instance we can see 3000 threads with this shape:
"QueueSubTaskMetrics [#11342]" #5508106 daemon prio=5 os_prio=0 tid=0x00007efcf085a800 nid=0x52c7 in Object.wait() [0x00007efbccb32000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:75) - locked <0x0000000512c01d10> (a hudson.model.queue.FutureImpl) at jenkins.metrics.impl.JenkinsMetricProviderImpl.lambda$asSupplier$3(JenkinsMetricProviderImpl.java:1142) at jenkins.metrics.impl.JenkinsMetricProviderImpl$$Lambda$388/1851215464.get(Unknown Source) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Locked ownable synchronizers: - <0x00000004dc401fb0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
There is way less number of jobs running or waiting (130) and the number of these threads never decreases.
After investigating the code:
- Metrics listens to the queue with a QueueListener creating threads with the name QueueSubTaskMetrics
- The 3000 threads shown are all waiting for a future to be completed.
- This future is waiting for the task to start and complete: https://github.com/jenkinsci/metrics-plugin/commit/e803bc3b82b54bfe27e66a6393dedf53bdf1896e#diff-b02885f3ba6b4982b5322b73e664c0b6R907, which never happens. It’s hung on hudson.remoting.AsyncFutureImpl.get:
while(!completed)
wait();
I think it could be avoided calling the get method with a timeout instead of getting blocked there: https://github.com/jenkinsci/metrics-plugin/commit/e803bc3b82b54bfe27e66a6393dedf53bdf1896e#diff-b02885f3ba6b4982b5322b73e664c0b6R1049
Attachments
Issue Links
- links to
Activity
Field | Original Value | New Value |
---|---|---|
Priority | Major [ 3 ] | Critical [ 2 ] |
Assignee | Ramon Leon [ mramonleon ] |
Status | Open [ 1 ] | In Progress [ 3 ] |
Comment | [ I'm working right now on trying to reproduce these leaks playing with the cancellation of the build. Whatever clue or suggestion is welcome. :sweat: Thanks! ] |
Remote Link | This issue links to "PR #4346 (Web Link)" [ 23945 ] |
Status | In Progress [ 3 ] | In Review [ 10005 ] |
Labels | plugin | lts-candidate plugin |
Labels | lts-candidate plugin | core lts-candidate plugin |
Component/s | core [ 15593 ] | |
Component/s | metrics-plugin [ 18751 ] |
Labels | core lts-candidate plugin | core lts-candidate metrics |
Summary | Possible thread leak 'QueueSubTaskMetrics' in metrics | Possible thread leak 'QueueSubTaskMetrics' in metrics - Allow finishing builds when SubTask badly fail |
Resolution | Fixed [ 1 ] | |
Status | In Review [ 10005 ] | Fixed but Unreleased [ 10203 ] |
Released As | jenkins-2.205 | |
Status | Fixed but Unreleased [ 10203 ] | Resolved [ 5 ] |
Labels | core lts-candidate metrics | 2.204.1-fixed core metrics |