-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
Jenkins 2.107.3
cloudfoundry-plugin 2.2.1
I came across a small Jenkins instance with the Cloudfoundry plugin installed, which was starting to use more memory than usual. The JVM threads dump (kill -3) shows 448 threads like this one:
"cloudfoundry-client-nio-1" #83131 daemon prio=5 os_prio=0 tid=0x00007f1a790b0800 nid=0x2e83 runnable [0x00007f1a50077000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000e761c9f0> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x00000000e761daa8> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000e761c8f8> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:752) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:408) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at java.lang.Thread.run(Thread.java:745)
They were all numbered from ...-nio-1 to ...-nio-4, so I assume that's actually 112 instances of threads pools with 4 threads each.
I've not tried patching anything (I've opted for a "let's" restart Jenkins" workaround), but from a quick look at the plugin sources, I think the issue comes from fresh ConnectionContext objects being created for each usage of the CF API, which are never properly closed/disposed. In the CF Java client lib, I see there is a dispose() method for that purpose, which is annotated @PreDestroy, so things are probably automagically cleaned up when the lib is used in a Spring or JEE context, but not in Jenkins.
https://github.com/cloudfoundry/cf-java-client/blob/v3.13.0.RELEASE/cloudfoundry-client-reactor/src/main/java/org/cloudfoundry/reactor/_DefaultConnectionContext.java#L71
So, I think in your plugin you should explicitly call this method, something like this (in CloudFoundryPushTask.perform(), CloudFoundryUtils.doTestConnection(), or wherever else you create a fresh ConnectionContext):
ConnectionContext connectionContext = createConnectionContext(run, workspace, listener);
try {
// ... do stuff with CF client API
} finally {
if (connectionContext instanceof _DefaultConnectionContext) {
// shutdown the connections pool
((_DefaultConnectionContext) connectionContext).dispose();
}
}
An alternative would be to keep a global, limited, cache of reusable ConnectionContext instances (in a map indexed by connection parameters, or something like that), but it sounds like more work to implement.
Oh, and of course, since I've not actually tested anything, I might be completely wrong in the above explanations...
