-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major
-
Component/s: ssh-credentials-plugin, ssh-plugin
-
None
-
Environment:Oracle Linux 7.9
JDK 11.0.18
Jenkins 2.375.2
Credentials plugin 1214.v1de940103927
SSH Credentials plugin 305.v8f4381501156
SSH plugin 2.6.1
bouncycastle-api plugin 2.27
I upgraded one of my build enviromments from Jenkins 2.346.2 to Jenkins 2.375.2 today, and upgraded all plugins installed on it to the latest versions (including, but not limited to bouncycastle-api and subversion). After doing so, all jobs running on agent nodes begun failing with the following error:
Started by user <REDACTED> Running as <REDACTED> [EnvInject] - Loading node environment variables. Building remotely on <REDACTED> in workspace <REDACTED> Cleaning up <REDACTED> Updating svn+ssh://<REDACTED> at revision '<REDACTED>' Using sole credentials <REDACTED> in realm ‘<REDACTED>’ FATAL: java.lang.ExceptionInInitializerError java.lang.NullPointerException   at java.base/javax.crypto.ProviderVerifier.verify(ProviderVerifier.java:122)   at java.base/javax.crypto.JceSecurity.verifyProvider(JceSecurity.java:191)   at java.base/javax.crypto.JceSecurity.getVerificationResult(JceSecurity.java:217)   at java.base/javax.crypto.Cipher.getInstance(Cipher.java:688) Caused: java.lang.SecurityException: JCE cannot authenticate the provider BC   at java.base/javax.crypto.Cipher.getInstance(Cipher.java:692)   at java.base/javax.crypto.Cipher.getInstance(Cipher.java:623)   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)   at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)   at java.base/java.lang.reflect.Method.invoke(Method.java:566)   at org.apache.sshd.common.util.security.SecurityEntityFactory$2.getInstance(SecurityEntityFactory.java:130)   at org.apache.sshd.common.util.security.SecurityUtils.getCipher(SecurityUtils.java:748)   at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator$LazyKeyLengthsHolder.detectSupportedKeySizes(AESPrivateKeyObfuscator.java:134)   at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator$LazyKeyLengthsHolder.<clinit>(AESPrivateKeyObfuscator.java:121) Caused: java.lang.ExceptionInInitializerError   at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator.getAvailableKeyLengths(AESPrivateKeyObfuscator.java:110)   at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator.getSupportedKeySizes(AESPrivateKeyObfuscator.java:51)   at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator.resolveKeyLength(AESPrivateKeyObfuscator.java:87)   at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator.applyPrivateKeyCipher(AESPrivateKeyObfuscator.java:58)   at org.apache.sshd.common.config.keys.loader.pem.AbstractPEMResourceKeyPairParser.applyPrivateKeyCipher(AbstractPEMResourceKeyPairParser.java:227)   at org.apache.sshd.common.config.keys.loader.pem.AbstractPEMResourceKeyPairParser.extractKeyPairs(AbstractPEMResourceKeyPairParser.java:170)   at org.apache.sshd.common.config.keys.loader.AbstractKeyPairResourceParser.loadKeyPairs(AbstractKeyPairResourceParser.java:117)   at org.apache.sshd.common.config.keys.loader.KeyPairResourceParser$2.loadKeyPairs(KeyPairResourceParser.java:166)   at org.apache.sshd.common.config.keys.loader.pem.PEMResourceParserUtils$1.loadKeyPairs(PEMResourceParserUtils.java:53)   at org.apache.sshd.common.config.keys.loader.KeyPairResourceParser$2.loadKeyPairs(KeyPairResourceParser.java:166)   at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:157)   at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:148)   at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:139)   at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:130)   at org.apache.sshd.common.util.security.SecurityUtils.loadKeyPairIdentities(SecurityUtils.java:522)   at org.tmatesoft.svn.core.internal.io.svn.SVNSSHPrivateKeyUtil.isValidPrivateKey(SVNSSHPrivateKeyUtil.java:99)   at org.tmatesoft.svn.core.internal.io.svn.SVNSSHConnector.open(SVNSSHConnector.java:102)   at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:80)   at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:1282)   at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.testConnection(SVNRepositoryImpl.java:100)   at org.tmatesoft.svn.core.io.SVNRepository.getRepositoryUUID(SVNRepository.java:268)   at org.tmatesoft.svn.core.internal.wc2.SvnRepositoryAccess.createRepository(SvnRepositoryAccess.java:103)   at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgRepositoryAccess.createRepository(SvnNgRepositoryAccess.java:211)   at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:210)   at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115) Also:  hudson.remoting.Channel$CallSiteStackTrace: Remote call to <REDACTED>     at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)     at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)     at hudson.remoting.Channel.call(Channel.java:1000)     at hudson.FilePath.act(FilePath.java:1186)     at hudson.FilePath.act(FilePath.java:1175)     at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:970)     at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:892)     at hudson.scm.SCM.checkout(SCM.java:540)     at hudson.model.AbstractProject.checkout(AbstractProject.java:1241)     at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)     at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)     at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)     at hudson.model.Run.execute(Run.java:1900)     at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)     at hudson.model.ResourceController.execute(ResourceController.java:107)     at hudson.model.Executor.run(Executor.java:449) Caused: java.lang.RuntimeException   at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.throwThrowable(SvnNgAbstractUpdate.java:918)   at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:125)   at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:40)   at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:18)   at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)   at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)   at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)   at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)   at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)   at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)   at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)   at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:165)   at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)   at hudson.scm.SubversionSCM$CheckOutUpdateTask.perform(SubversionSCM.java:1086)   at hudson.scm.SubversionSCM$CheckOutUpdateTask.run(SubversionSCM.java:1067)   at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1037)   at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1020)   at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3492)   at hudson.remoting.UserRequest.perform(UserRequest.java:211)   at hudson.remoting.UserRequest.perform(UserRequest.java:54)   at hudson.remoting.Request$2.run(Request.java:377)   at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)   at java.base/java.lang.Thread.run(Thread.java:834) Archiving artifacts Started calculate disk usage of build Finished Calculation of disk usage of build in 0 seconds Started calculate disk usage of workspace Finished Calculation of disk usage of workspace in 0 seconds Extended Email Publisher is currently disabled in project settings Finished: FAILURE
This impacts agents on Windows, Linux, and macOS whether launched via JNLP or SSH.
Â
The remoting logs don't indicate any issues:
Feb 01, 2023 3:04:31 PM org.apache.sshd.common.util.security.AbstractSecurityProviderRegistrar getOrCreateProvider INFO: getOrCreateProvider(BC) created instance of org.bouncycastle.jce.provider.BouncyCastleProvider Feb 01, 2023 3:04:31 PM org.apache.sshd.common.util.security.AbstractSecurityProviderRegistrar getOrCreateProvider INFO: getOrCreateProvider(EdDSA) created instance of net.i2p.crypto.eddsa.EdDSASecurityProvider
Enabling debugging of jar sigignature validation (-Djavax.net.debug=jar) indicates no issues. Jobs run successfully on the built-in executors on the master instance. They also run successfully if the SSH key is decrypted and loaded into Jenkins with no password.
Â
I theorize that this is somehow an artifact of the migration from the prior SSH plugin to the newer Mina-based solution. Not having any familiarity with the plumbing responsible for shipping BC to the nodes and loading it, nor the wiring of BC and Mina, this is only a theory.
