-
Bug
-
Resolution: Fixed
-
Major
-
None
-
Jenkins 2.60.3
jclouds-jenkins 2.14
bouncycastle-api 2.16.2
OpenJDK 8u111-b14
Linux (Ubuntu 14.04)
Here is (in attachment) an Eclipse Memory Analyzer report for a heap dump I made on a Jenkins master process which uses the JClouds plugin (version 2.14). It shows many (~590) instances of BouncyCastleProvider (a JCE Provider) are retained by JceSecurity. This JCE class keeps a reference to any provider which has ever been used, in an IdentityMap, meaning it can easily leak memory when used in a wrong way. This is described here:
https://bugs.openjdk.java.net/browse/JDK-8168469
I've use a BTrace probe to track instantiations of the BouncyCastleProvider objects (I will attach it too for reference, if someone wants to play with it, but it's really simple).
One first instance of the provider comes from the bouncycastle-api plugin initialization, as expected (the plugin registers BouncyCastle as a JCE provider) :
org.bouncycastle.jce.provider.BouncyCastleProvider.<init>(Unknown Source) jenkins.bouncycastle.api.SecurityProviderInitializer.addSecurityProvider(SecurityProviderInitializer.java:65) jenkins.bouncycastle.api.SecurityProviderInitializer.<clinit>(SecurityProviderInitializer.java:52) <snip>
The other instances are created by the JCloud API, each time a JCloud context is created by the Jenkins JClouds plugin. For instance, when clicking the "Test connection" button in the Blob Storage global configuration, we get that:
org.bouncycastle.jce.provider.BouncyCastleProvider.<init>(Unknown Source) org.jclouds.encryption.bouncycastle.BouncyCastleCrypto.<init>(BouncyCastleCrypto.java:33) org.jclouds.encryption.bouncycastle.BouncyCastleCrypto$$FastClassByGuice$$29a9bb14.newInstance(<generated>) com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60) com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) com.google.inject.Scopes$1$1.get(Scopes.java:65) com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54) com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:204) com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:198) com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:198) com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:179) com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109) com.google.inject.Guice.createInjector(Guice.java:95) org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:405) org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:328) org.jclouds.ContextBuilder.buildView(ContextBuilder.java:615) org.jclouds.ContextBuilder.buildView(ContextBuilder.java:595) jenkins.plugins.jclouds.blobstore.BlobStoreProfile.ctx(BlobStoreProfile.java:174) jenkins.plugins.jclouds.blobstore.BlobStoreProfile.ctx(BlobStoreProfile.java:190) jenkins.plugins.jclouds.blobstore.BlobStoreProfile$DescriptorImpl.doTestConnection(BlobStoreProfile.java:328) <snip>
The org.jclouds.encryption.bouncycastle.BouncyCastleCrypto class is an implementation of org.jclouds.crypto.Crypto (a utility layer for using JCE methods), and it automatically registers a new instance of BouncyCastle's provider when instantiated :
It is instantiated because BouncyCastleCryptoModule is included in EnterpriseConfigurationModule:
- https://github.com/jclouds/jclouds/blob/rel/jclouds-2.0.0/drivers/bouncycastle/src/main/java/org/jclouds/encryption/bouncycastle/config/BouncyCastleCryptoModule.java#L31
- https://github.com/jclouds/jclouds/blob/rel/jclouds-2.0.0/drivers/enterprise/src/main/java/org/jclouds/enterprise/config/EnterpriseConfigurationModule.java#L53
... and EnterpriseConfigurationModule is used by the Jenkins JClouds plugin:
- https://github.com/jenkinsci/jclouds-plugin/blob/jclouds-jenkins-2.14/jclouds-plugin/src/main/java/jenkins/plugins/jclouds/blobstore/BlobStoreProfile.java#L167
- https://github.com/jenkinsci/jclouds-plugin/blob/jclouds-jenkins-2.14/jclouds-plugin/src/main/java/jenkins/plugins/jclouds/compute/JCloudsCloud.java#L272
Because of all that, more BouncyCastleProvider instances can be leaked quite quickly; on the Jenkins I've been diagnosing, it was something like +50 instances for each run of single job which was publishing a bunch of files to some S3 blob store.
One way to avoid that would be to reuse JCloud contexts in the plugin, instead of creating new ones so often, but when I look at the plugin sources I understand that it would be difficult.
An other way would be to stop using BouncyCastleCrypto, and instead use an equivalent implementation which would not register more than one BouncyCastleProvider instance to JCE. I have implemented something like that (replacing both BouncyCastleCrypto and EnterpriseConfigurationModule by some Jenkins specific variants), and so far it works just fine. I will submit a PR.
Finally, that's also something which could be fixed in Jcloud itself. I will try to open a bug report there too.
