[BROKEN] Agents fail to launch on v2029, reverting to v2018 fixes

XMLWordPrintable

    • Type: Bug
    • Resolution: Fixed
    • Priority: Blocker
    • Component/s: ec2-plugin
    • None
    • Environment:
      Jenkins 2.516.2 (current LTS version)
    • ec2:2032.v92a_4b_e703974

      When we updated the EC2 plugin from 2018.vc13c882e22db_ to 2029.v50458f410ef9, Agents were not longer able to start. Reverting to 2018.vc13c882e22db_ fixed the problem, so it's pretty clearly a bad release, at least in our environment.

      Here's what we saw in the logs:

      Sep 09, 2025 8:38:52 AM WARNING hudson.plugins.ec2.EC2Cloud logUnable to check the server key
java.lang.IllegalArgumentException: Illegal base64 character 5d
    at java.base/java.util.Base64$Decoder.decode0(Unknown Source)
    at java.base/java.util.Base64$Decoder.decode(Unknown Source)
    at java.base/java.util.Base64$Decoder.decode(Unknown Source)
    at PluginClassLoader for ec2//hudson.plugins.ec2.ssh.verifiers.SshHostKeyVerificationStrategy.getKeyFromLine(SshHostKeyVerificationStrategy.java:178)
    at PluginClassLoader for ec2//hudson.plugins.ec2.ssh.verifiers.SshHostKeyVerificationStrategy.getHostKeyFromConsole(SshHostKeyVerificationStrategy.java:115)
    at PluginClassLoader for ec2//hudson.plugins.ec2.ssh.verifiers.CheckNewHardStrategy.verify(CheckNewHardStrategy.java:50)
    at PluginClassLoader for ec2//hudson.plugins.ec2.ssh.verifiers.SshHostKeyVerificationStrategy.verify(SshHostKeyVerificationStrategy.java:79)
    at PluginClassLoader for ec2//hudson.plugins.ec2.ssh.EC2SSHLauncher$ServerKeyVerifierImpl.verifyServerKey(EC2SSHLauncher.java:404)
    at PluginClassLoader for mina-sshd-api-core//org.apache.sshd.client.session.AbstractClientSession.checkKeys(AbstractClientSession.java:620)
    at PluginClassLoader for mina-sshd-api-core//org.apache.sshd.common.session.helpers.AbstractSession.handleKexMessage(AbstractSession.java:776)
    at PluginClassLoader for mina-sshd-api-core//org.apache.sshd.common.session.helpers.AbstractSession.doHandleMessage(AbstractSession.java:631)
    at PluginClassLoader for mina-sshd-api-core//org.apache.sshd.common.session.helpers.AbstractSession.lambda$handleMessage$0(AbstractSession.java:555)
    at PluginClassLoader for mina-sshd-api-common//org.apache.sshd.common.util.threads.ThreadUtils.runAsInternal(ThreadUtils.java:68)
    at PluginClassLoader for mina-sshd-api-core//org.apache.sshd.common.session.helpers.AbstractSession.handleMessage(AbstractSession.java:554)
    at PluginClassLoader for mina-sshd-api-core//org.apache.sshd.common.session.helpers.AbstractSession.decode(AbstractSession.java:1738)
    at PluginClassLoader for mina-sshd-api-core//org.apache.sshd.common.session.helpers.AbstractSession.messageReceived(AbstractSession.java:515)
    at PluginClassLoader for mina-sshd-api-core//org.apache.sshd.common.session.helpers.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:76)
    at PluginClassLoader for mina-sshd-api-core//org.apache.sshd.mina.MinaService.messageReceived(MinaService.java:163)
    at PluginClassLoader for mina-sshd-api-core//org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:1015)
    at PluginClassLoader for mina-sshd-api-core//org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
    at PluginClassLoader for mina-sshd-api-core//org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
    at PluginClassLoader for mina-sshd-api-core//org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
    at PluginClassLoader for mina-sshd-api-core//org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:122)
    at PluginClassLoader for mina-sshd-api-core//org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
    at PluginClassLoader for mina-sshd-api-core//org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:643)
    at PluginClassLoader for mina-sshd-api-core//org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:539)
    at PluginClassLoader for mina-sshd-api-core//org.apache.mina.core.polling.AbstractPollingIoProcessor.access$1200(AbstractPollingIoProcessor.java:68)
    at PluginClassLoader for mina-sshd-api-core//org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.process(AbstractPollingIoProcessor.java:1224)
    at PluginClassLoader for mina-sshd-api-core//org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.process(AbstractPollingIoProcessor.java:1213)
    at PluginClassLoader for mina-sshd-api-core//org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:683)
    at PluginClassLoader for mina-sshd-api-core//org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

            Assignee:
            Allan BURDAJEWICZ
            Reporter:
            Matthew Webber
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: