Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53136

Threads leak from cloudfoundry-plugin

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Minor
    • Resolution: Unresolved
    • cloudfoundry-plugin
    • None
    • Jenkins 2.107.3
      cloudfoundry-plugin 2.2.1

    Description

      I came across a small Jenkins instance with the Cloudfoundry plugin installed, which was starting to use more memory than usual. The JVM threads dump (kill -3) shows 448 threads like this one:

      "cloudfoundry-client-nio-1" #83131 daemon prio=5 os_prio=0 tid=0x00007f1a790b0800 nid=0x2e83 runnable [0x00007f1a50077000]
         java.lang.Thread.State: RUNNABLE
      	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
      	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
      	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
      	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
      	- locked <0x00000000e761c9f0> (a io.netty.channel.nio.SelectedSelectionKeySet)
      	- locked <0x00000000e761daa8> (a java.util.Collections$UnmodifiableSet)
      	- locked <0x00000000e761c8f8> (a sun.nio.ch.EPollSelectorImpl)
      	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
      	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
      	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:752)
      	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:408)
      	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
      	at java.lang.Thread.run(Thread.java:745)
      

      They were all numbered from ...-nio-1 to ...-nio-4, so I assume that's actually 112 instances of threads pools with 4 threads each.

      I've not tried patching anything (I've opted for a "let's" restart Jenkins" workaround), but from a quick look at the plugin sources, I think the issue comes from fresh ConnectionContext objects being created for each usage of the CF API, which are never properly closed/disposed. In the CF Java client lib, I see there is a dispose() method for that purpose, which is annotated @PreDestroy, so things are probably automagically cleaned up when the lib is used in a Spring or JEE context, but not in Jenkins.
      https://github.com/cloudfoundry/cf-java-client/blob/v3.13.0.RELEASE/cloudfoundry-client-reactor/src/main/java/org/cloudfoundry/reactor/_DefaultConnectionContext.java#L71

      So, I think in your plugin you should explicitly call this method, something like this (in CloudFoundryPushTask.perform(), CloudFoundryUtils.doTestConnection(), or wherever else you create a fresh ConnectionContext):

                  ConnectionContext connectionContext = createConnectionContext(run, workspace, listener);
                  try {
                      // ... do stuff with CF client API
                  } finally {
                      if (connectionContext instanceof _DefaultConnectionContext) {
                          // shutdown the connections pool
                          ((_DefaultConnectionContext) connectionContext).dispose();
                      }
                  }
      

      An alternative would be to keep a global, limited, cache of reusable ConnectionContext instances (in a map indexed by connection parameters, or something like that), but it sounds like more work to implement.

      Oh, and of course, since I've not actually tested anything, I might be completely wrong in the above explanations...

      Attachments

        Activity

          There are no comments yet on this issue.

          People

            olamy Olivier Lamy
            tom_gl Thomas de Grenier de Latour
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: