-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Jenkins 2.303.3, Jira plugin 3.6
A controller running Jira plugin 3.6 leaks hundreds of httpclient-io and I/O dispatcher threads in the span of a couple of days. I'm pretty sure these two are related, though I don't see a direct connection between them. The httpclient threads are all labeled 'httpclient-io:thread-1' even though their IDs are different. Their stack trace:
java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) - locked <0x00000002bb45c4b0> (a java.lang.Thread) at java.lang.Thread.join(Thread.java:1331) at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.close(CloseableHttpAsyncClientBase.java:104) at com.atlassian.httpclient.apache.httpcomponents.CompletableFuturePromiseHttpPromiseAsyncClient.closeClient(CompletableFuturePromiseHttpPromiseAsyncClient.java:75) at com.atlassian.httpclient.apache.httpcomponents.CompletableFuturePromiseHttpPromiseAsyncClient.access$300(CompletableFuturePromiseHttpPromiseAsyncClient.java:22) at com.atlassian.httpclient.apache.httpcomponents.CompletableFuturePromiseHttpPromiseAsyncClient$1.doFailed(CompletableFuturePromiseHttpPromiseAsyncClient.java:58) at com.atlassian.httpclient.apache.httpcomponents.CompletableFuturePromiseHttpPromiseAsyncClient$ThreadLocalContextAwareFutureCallback.lambda$failed$1(CompletableFuturePromiseHttpPromiseAsyncClient.java:126) at com.atlassian.httpclient.apache.httpcomponents.CompletableFuturePromiseHttpPromiseAsyncClient$ThreadLocalContextAwareFutureCallback$$Lambda$1437/1808038226.run(Unknown Source) at com.atlassian.httpclient.apache.httpcomponents.CompletableFuturePromiseHttpPromiseAsyncClient.runInContext(CompletableFuturePromiseHttpPromiseAsyncClient.java:89) at com.atlassian.httpclient.apache.httpcomponents.CompletableFuturePromiseHttpPromiseAsyncClient$ThreadLocalContextAwareFutureCallback.failed(CompletableFuturePromiseHttpPromiseAsyncClient.java:126) at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:137) at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.executionFailed(DefaultClientExchangeHandlerImpl.java:101) at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:426) at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.connectionRequestFailed(AbstractClientExchangeHandler.java:348) at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.access$100(AbstractClientExchangeHandler.java:62) at org.apache.http.impl.nio.client.AbstractClientExchangeHandler$1.failed(AbstractClientExchangeHandler.java:392) at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:137) at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$1.failed(PoolingNHttpClientConnectionManager.java:316) at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:137) at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:169) at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:632) at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:898) at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:198) - locked <0x00000002bb45c7c0> (a org.apache.http.impl.nio.reactor.SessionRequestImpl) at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:213) at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:158) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351) at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) at java.lang.Thread.run(Thread.java:750) Locked ownable synchronizers: - <0x00000002da1debb0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
The I/O dispatcher threads have this stack trace:
java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000002bb481268> (a sun.nio.ch.Util$3) - locked <0x00000002bb481278> (a java.util.Collections$UnmodifiableSet) - locked <0x00000002bb481220> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) at java.lang.Thread.run(Thread.java:750) Locked ownable synchronizers: - None
In the controller logs, we see many exceptions like this:
2022-10-21 13:34:03.147+0000 [id=1834904] WARNING h.plugins.jira.JiraRestService#getUser: Jira REST client get user error. cause: null
java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1769)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
at io.atlassian.util.concurrent.Promises$OfStage.get(Promises.java:357)
at com.atlassian.jira.rest.client.internal.async.DelegatingPromise.get(DelegatingPromise.java:106)
at hudson.plugins.jira.JiraRestService.getUser(JiraRestService.java:318)
at hudson.plugins.jira.JiraMailAddressResolver.findMailAddressFor(JiraMailAddressResolver.java:43)
at hudson.tasks.MailAddressResolver.resolve(MailAddressResolver.java:122)
...
My theory is that the connection timeout is where the thread is getting leaked. The exceptions are intermittent, so there does seem to be a connectivity problem that comes and goes - but the plugin obviously shouldn't be leaking threads when that happens.