-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Oracle Linux 7.9
JDK 11.0.18
Jenkins 2.375.2
Credentials plugin 1214.v1de940103927
SSH Credentials plugin 305.v8f4381501156
SSH plugin 2.6.1
bouncycastle-api plugin 2.27
I upgraded one of my build enviromments from Jenkins 2.346.2 to Jenkins 2.375.2 today, and upgraded all plugins installed on it to the latest versions (including, but not limited to bouncycastle-api and subversion). After doing so, all jobs running on agent nodes begun failing with the following error:
Started by user <REDACTED> Running as <REDACTED> [EnvInject] - Loading node environment variables. Building remotely on <REDACTED> in workspace <REDACTED> Cleaning up <REDACTED> Updating svn+ssh://<REDACTED> at revision '<REDACTED>' Using sole credentials <REDACTED> in realm ‘<REDACTED>’ FATAL: java.lang.ExceptionInInitializerError java.lang.NullPointerException at java.base/javax.crypto.ProviderVerifier.verify(ProviderVerifier.java:122) at java.base/javax.crypto.JceSecurity.verifyProvider(JceSecurity.java:191) at java.base/javax.crypto.JceSecurity.getVerificationResult(JceSecurity.java:217) at java.base/javax.crypto.Cipher.getInstance(Cipher.java:688) Caused: java.lang.SecurityException: JCE cannot authenticate the provider BC at java.base/javax.crypto.Cipher.getInstance(Cipher.java:692) at java.base/javax.crypto.Cipher.getInstance(Cipher.java:623) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.sshd.common.util.security.SecurityEntityFactory$2.getInstance(SecurityEntityFactory.java:130) at org.apache.sshd.common.util.security.SecurityUtils.getCipher(SecurityUtils.java:748) at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator$LazyKeyLengthsHolder.detectSupportedKeySizes(AESPrivateKeyObfuscator.java:134) at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator$LazyKeyLengthsHolder.<clinit>(AESPrivateKeyObfuscator.java:121) Caused: java.lang.ExceptionInInitializerError at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator.getAvailableKeyLengths(AESPrivateKeyObfuscator.java:110) at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator.getSupportedKeySizes(AESPrivateKeyObfuscator.java:51) at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator.resolveKeyLength(AESPrivateKeyObfuscator.java:87) at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator.applyPrivateKeyCipher(AESPrivateKeyObfuscator.java:58) at org.apache.sshd.common.config.keys.loader.pem.AbstractPEMResourceKeyPairParser.applyPrivateKeyCipher(AbstractPEMResourceKeyPairParser.java:227) at org.apache.sshd.common.config.keys.loader.pem.AbstractPEMResourceKeyPairParser.extractKeyPairs(AbstractPEMResourceKeyPairParser.java:170) at org.apache.sshd.common.config.keys.loader.AbstractKeyPairResourceParser.loadKeyPairs(AbstractKeyPairResourceParser.java:117) at org.apache.sshd.common.config.keys.loader.KeyPairResourceParser$2.loadKeyPairs(KeyPairResourceParser.java:166) at org.apache.sshd.common.config.keys.loader.pem.PEMResourceParserUtils$1.loadKeyPairs(PEMResourceParserUtils.java:53) at org.apache.sshd.common.config.keys.loader.KeyPairResourceParser$2.loadKeyPairs(KeyPairResourceParser.java:166) at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:157) at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:148) at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:139) at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:130) at org.apache.sshd.common.util.security.SecurityUtils.loadKeyPairIdentities(SecurityUtils.java:522) at org.tmatesoft.svn.core.internal.io.svn.SVNSSHPrivateKeyUtil.isValidPrivateKey(SVNSSHPrivateKeyUtil.java:99) at org.tmatesoft.svn.core.internal.io.svn.SVNSSHConnector.open(SVNSSHConnector.java:102) at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:80) at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:1282) at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.testConnection(SVNRepositoryImpl.java:100) at org.tmatesoft.svn.core.io.SVNRepository.getRepositoryUUID(SVNRepository.java:268) at org.tmatesoft.svn.core.internal.wc2.SvnRepositoryAccess.createRepository(SvnRepositoryAccess.java:103) at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgRepositoryAccess.createRepository(SvnNgRepositoryAccess.java:211) at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:210) at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115) Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to <REDACTED> at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356) at hudson.remoting.Channel.call(Channel.java:1000) at hudson.FilePath.act(FilePath.java:1186) at hudson.FilePath.act(FilePath.java:1175) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:970) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:892) at hudson.scm.SCM.checkout(SCM.java:540) at hudson.model.AbstractProject.checkout(AbstractProject.java:1241) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521) at hudson.model.Run.execute(Run.java:1900) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44) at hudson.model.ResourceController.execute(ResourceController.java:107) at hudson.model.Executor.run(Executor.java:449) Caused: java.lang.RuntimeException at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.throwThrowable(SvnNgAbstractUpdate.java:918) at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:125) at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:40) at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:18) at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20) at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21) at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239) at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294) at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311) at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291) at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387) at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:165) at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168) at hudson.scm.SubversionSCM$CheckOutUpdateTask.perform(SubversionSCM.java:1086) at hudson.scm.SubversionSCM$CheckOutUpdateTask.run(SubversionSCM.java:1067) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1037) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1020) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3492) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:377) at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Archiving artifacts Started calculate disk usage of build Finished Calculation of disk usage of build in 0 seconds Started calculate disk usage of workspace Finished Calculation of disk usage of workspace in 0 seconds Extended Email Publisher is currently disabled in project settings Finished: FAILURE
This impacts agents on Windows, Linux, and macOS whether launched via JNLP or SSH.
The remoting logs don't indicate any issues:
Feb 01, 2023 3:04:31 PM org.apache.sshd.common.util.security.AbstractSecurityProviderRegistrar getOrCreateProvider INFO: getOrCreateProvider(BC) created instance of org.bouncycastle.jce.provider.BouncyCastleProvider Feb 01, 2023 3:04:31 PM org.apache.sshd.common.util.security.AbstractSecurityProviderRegistrar getOrCreateProvider INFO: getOrCreateProvider(EdDSA) created instance of net.i2p.crypto.eddsa.EdDSASecurityProvider
Enabling debugging of jar sigignature validation (-Djavax.net.debug=jar) indicates no issues. Jobs run successfully on the built-in executors on the master instance. They also run successfully if the SSH key is decrypted and loaded into Jenkins with no password.
I theorize that this is somehow an artifact of the migration from the prior SSH plugin to the newer Mina-based solution. Not having any familiarity with the plumbing responsible for shipping BC to the nodes and loading it, nor the wiring of BC and Mina, this is only a theory.
