Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47967

Memory leak (BouncyCastleProvider instances retained by JceSecurity)

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • jclouds-plugin
    • None
    • Jenkins 2.60.3
      jclouds-jenkins 2.14
      bouncycastle-api 2.16.2
      OpenJDK 8u111-b14
      Linux (Ubuntu 14.04)

    Description

      Here is (in attachment) an Eclipse Memory Analyzer report for a heap dump I made on a Jenkins master process which uses the JClouds plugin (version 2.14).  It shows many (~590) instances of BouncyCastleProvider (a JCE Provider) are retained by JceSecurity. This JCE class keeps a reference to any provider which has ever been used, in an IdentityMap, meaning it can easily leak memory when used in a wrong way.  This is described here:
      https://bugs.openjdk.java.net/browse/JDK-8168469

      I've use a BTrace probe to track instantiations of the BouncyCastleProvider objects (I will attach it too for reference, if someone wants to play with it, but it's really simple). 

      One first instance of the provider comes from the bouncycastle-api plugin initialization, as expected (the plugin registers BouncyCastle as a JCE provider) :

      org.bouncycastle.jce.provider.BouncyCastleProvider.<init>(Unknown Source)
jenkins.bouncycastle.api.SecurityProviderInitializer.addSecurityProvider(SecurityProviderInitializer.java:65)
jenkins.bouncycastle.api.SecurityProviderInitializer.<clinit>(SecurityProviderInitializer.java:52)
<snip>

      The other instances are created by the JCloud API, each time a JCloud context is created by the Jenkins JClouds plugin. For instance, when clicking the "Test connection" button in the Blob Storage global configuration, we get that:

      org.bouncycastle.jce.provider.BouncyCastleProvider.<init>(Unknown Source)
org.jclouds.encryption.bouncycastle.BouncyCastleCrypto.<init>(BouncyCastleCrypto.java:33)
org.jclouds.encryption.bouncycastle.BouncyCastleCrypto$$FastClassByGuice$$29a9bb14.newInstance(<generated>)
com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
com.google.inject.Scopes$1$1.get(Scopes.java:65)
com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)
com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:204)
com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:198)
com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:198)
com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:179)
com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
com.google.inject.Guice.createInjector(Guice.java:95)
org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:405)
org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:328)
org.jclouds.ContextBuilder.buildView(ContextBuilder.java:615)
org.jclouds.ContextBuilder.buildView(ContextBuilder.java:595)
jenkins.plugins.jclouds.blobstore.BlobStoreProfile.ctx(BlobStoreProfile.java:174)
jenkins.plugins.jclouds.blobstore.BlobStoreProfile.ctx(BlobStoreProfile.java:190)
jenkins.plugins.jclouds.blobstore.BlobStoreProfile$DescriptorImpl.doTestConnection(BlobStoreProfile.java:328)
<snip>

      The org.jclouds.encryption.bouncycastle.BouncyCastleCrypto class is an implementation of org.jclouds.crypto.Crypto (a utility layer for using JCE methods), and it automatically registers a new instance of BouncyCastle's provider when instantiated :

      It is instantiated because BouncyCastleCryptoModule is included in EnterpriseConfigurationModule:

      ... and EnterpriseConfigurationModule is used by the Jenkins JClouds plugin:

      Because of all that, more BouncyCastleProvider instances can be leaked quite quickly; on the Jenkins I've been diagnosing, it was something like +50 instances for each run of single job which was publishing a bunch of files to some S3 blob store.

      One way to avoid that would be to reuse JCloud contexts in the plugin, instead of creating new ones so often, but when I look at the plugin sources I understand that it would be difficult.

      An other way would be to stop using BouncyCastleCrypto, and instead use an equivalent implementation which would not register more than one BouncyCastleProvider instance to JCE. I have implemented something like that (replacing both BouncyCastleCrypto and EnterpriseConfigurationModule by some Jenkins specific variants), and so far it works just fine. I will submit a PR.

      Finally, that's also something which could be fixed in Jcloud itself. I will try to open a bug report there too.

      Attachments

        Activity

          Ascending order - Click to sort in descending order
          felfert Fritz Elfert added a comment -

          Thanks for the comprehensive report.
          -Fritz

          felfert Fritz Elfert added a comment - Thanks for the comprehensive report. -Fritz
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Fritz Elfert
          Path:
          jclouds-plugin/src/main/java/jenkins/plugins/jclouds/blobstore/BlobStoreProfile.java
          jclouds-plugin/src/main/java/jenkins/plugins/jclouds/compute/JCloudsCloud.java
          jclouds-plugin/src/main/java/jenkins/plugins/jclouds/modules/JenkinsBouncyCastleCrypto.java
          jclouds-plugin/src/main/java/jenkins/plugins/jclouds/modules/JenkinsConfigurationModule.java
          http://jenkins-ci.org/commit/jclouds-plugin/8d959300746548830846a81095811a0efe2a14db
          Log:
          Merge pull request #124 from thomasgl-orange/JENKINS-47967

          JENKINS-47967 workaround JCE Providers memory leak

          Compare: https://github.com/jenkinsci/jclouds-plugin/compare/c16b15c8b973...8d9593007465

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Fritz Elfert Path: jclouds-plugin/src/main/java/jenkins/plugins/jclouds/blobstore/BlobStoreProfile.java jclouds-plugin/src/main/java/jenkins/plugins/jclouds/compute/JCloudsCloud.java jclouds-plugin/src/main/java/jenkins/plugins/jclouds/modules/JenkinsBouncyCastleCrypto.java jclouds-plugin/src/main/java/jenkins/plugins/jclouds/modules/JenkinsConfigurationModule.java http://jenkins-ci.org/commit/jclouds-plugin/8d959300746548830846a81095811a0efe2a14db Log: Merge pull request #124 from thomasgl-orange/ JENKINS-47967 JENKINS-47967 workaround JCE Providers memory leak Compare: https://github.com/jenkinsci/jclouds-plugin/compare/c16b15c8b973...8d9593007465
          tom_gl Thomas de Grenier de Latour added a comment -

          Thanks for merging the PR so quickly.

          I have opened an issue in the JClouds tracker: https://issues.apache.org/jira/browse/JCLOUDS-1354

          tom_gl Thomas de Grenier de Latour added a comment - Thanks for merging the PR so quickly. I have opened an issue in the JClouds tracker: https://issues.apache.org/jira/browse/JCLOUDS-1354

          People

            felfert Fritz Elfert
            tom_gl Thomas de Grenier de Latour
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: