• EC2 Plugin 1.50.3

      Version 1.50.2 introduces security mitigations by proposing new options for SSH.

      2 of the 3 options have been introduced by SSH version 7.6:

      • ssh(1): expand the StrictHostKeyChecking option with two new
        settings. The first "accept-new" will automatically accept
        hitherto-unseen keys but will refuse connections for changed or
        invalid hostkeys. This is a safer subset of the current behaviour
        of StrictHostKeyChecking=no. The second setting "off", is a synonym
        for the current behaviour of StrictHostKeyChecking=no: accept new
        host keys, and continue connection for hosts with incorrect
        hostkeys. A future release will change the meaning of
        StrictHostKeyChecking=no to the behaviour of "accept-new". bz#2400

      Although it was released almost 3 years ago, this seriously breaks compatibility with non-recent Jenkins installations.

      For instance, the current default Docker image for Jenkins is currently based off Debian Stretch which provides SSH 7.4 and doesn't support these new options:

      $ docker run --rm -ti jenkins/jenkins:2.235 ssh -o StrictHostKeyChecking=off
      command-line line 0: unsupported option "off".
      $ docker run --rm -ti jenkins/jenkins:2.235 ssh -o StrictHostKeyChecking=accept-new
      command-line line 0: unsupported option "accept-new".
      $ docker run --rm -ti jenkins/jenkins:lts ssh -o StrictHostKeyChecking=accept-new
      command-line line 0: unsupported option "accept-new".
      

          [JENKINS-62195] ec2-1.50.2 doesn't work with SSH <7.5

          Jonathan Ballet added a comment - Merged in https://github.com/jenkinsci/ec2-plugin/pull/455

          This caused havoc on a build fleet with a docker master node as described in the report and had to downgrade. Please pull the release.

          Jason Potkanski added a comment - This caused havoc on a build fleet with a docker master node as described in the report and had to downgrade. Please pull the release.

          Oleg Nenashev added a comment -

          It would be great to get a release indeed.

          raihaan would you be interested?

           

          Oleg Nenashev added a comment - It would be great to get a release indeed. raihaan would you be interested?  

          Hey guys, I just made a new release with some bugfixes. Which includes the PR linked in this ticket

          Raihaan Shouhell added a comment - Hey guys, I just made a new release with some bugfixes. Which includes the PR linked in this ticket

          Jonathan Ballet added a comment - - edited

          raihaan Thanks for the release, I'll test it during the weekend!

          FYI, the changelog on GitHub doesn't contain that fix, I don't know if it can be added after the release has been created but I think it would help people to see what has been fixed or not!

          Jonathan Ballet added a comment - - edited raihaan Thanks for the release, I'll test it during the weekend! FYI, the changelog on GitHub doesn't contain that fix , I don't know if it can be added after the release has been created but I think it would help people to see what has been fixed or not!

          Oleg Nenashev added a comment -

          Oleg Nenashev added a comment - https://github.com/jenkinsci/ec2-plugin/commit/91f48a7eb7aa1270970b92ece38606a97543deae  includes it. Several pull requests were missing in the release draft I fixed the changelog:  https://github.com/jenkinsci/ec2-plugin/releases/tag/ec2-1.50.3

          Thanks oleg_nenashev for the updated changelog!

          Jonathan Ballet added a comment - Thanks oleg_nenashev for the updated changelog!

          Antoine Hamon added a comment -

           
          I was able to reproduce this issue with Jenkins 2.237 (official docker image) & EC2 plugin 1.50.3
          After upgrading the server (from version 2.233), slaves appeared offlines and Jenkins were not able to re-connect to them:

          May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
          INFO: Launching instance: i-xxxxxxx
          May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
          INFO: bootstrap()
          May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
          INFO: Getting keypair...
          May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
          INFO: Using private key jenkins_slaves_new (SHA-1 fingerprint xx:xx:xx:xx:xx:xx:xx:xx)
          May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
          INFO: Authenticating as centos
          May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
          INFO: Connecting to 10.100.11.222 on port 22, with timeout 10000.
          May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
          INFO: Connection allowed after the host key has been verified
          May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
          INFO: Connected via SSH.
          May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
          INFO: connect fresh as root
          May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
          INFO: Connecting to 10.100.11.222 on port 22, with timeout 10000.
          May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
          INFO: Connection allowed after the host key has been verified
          May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
          INFO: Connected via SSH.
          May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
          INFO: Creating tmp directory (/tmp) if it does not existMay 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
          INFO: Verifying: java -fullversion
          openjdk full version "1.8.0_252-b09"
          May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
          INFO: Verifying: which scp
          /usr/bin/scp
          May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud
          INFO: Copying remoting.jar to: /tmp
          May 22, 2020 5:43:34 PM hudson.plugins.ec2.EC2Cloud
          INFO: Launching remoting agent (via SSH client process): ssh -o StrictHostKeyChecking=accept-new -i /tmp/ec2_1385185844708247542.pem centos@10.100.11.222 -p 22  java  -jar /tmp/remoting.jar -workDir /jenkins
          [05/22/20 17:43:34] Launching agent
          $ ssh -o StrictHostKeyChecking=accept-new -i /tmp/ec2_1385185844708728885.pem centos@10.100.11.222 -p 22  java  -jar /tmp/remoting.jar -workDir /jenkins
          command-line line 0: unsupported option "accept-new".
          ERROR: Unable to launch the agent for EC2 (Slaves) - Slave (i-xxxxxxx)
          java.io.EOFException: unexpected stream termination
          	at hudson.remoting.ChannelBuilder.negotiate(ChannelBuilder.java:415)
          	at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:360)
          	at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:423)
          	at hudson.slaves.CommandLauncher.launch(CommandLauncher.java:165)
          	at hudson.plugins.ec2.ssh.EC2UnixLauncher.launchScript(EC2UnixLauncher.java:257)
          	at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:48)
          	at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:296)
          	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
          	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          	at java.lang.Thread.run(Thread.java:748)
          

          We still see the ssh -o StrictHostKeyChecking=accept-new

          Antoine Hamon added a comment -   I was able to reproduce this issue with Jenkins 2.237 (official docker image) & EC2 plugin 1.50.3 After upgrading the server (from version 2.233), slaves appeared offlines and Jenkins were not able to re-connect to them: May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Launching instance: i-xxxxxxx May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: bootstrap() May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Getting keypair... May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Using private key jenkins_slaves_new (SHA-1 fingerprint xx:xx:xx:xx:xx:xx:xx:xx) May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Authenticating as centos May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.100.11.222 on port 22, with timeout 10000. May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Connection allowed after the host key has been verified May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Connected via SSH. May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: connect fresh as root May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Connecting to 10.100.11.222 on port 22, with timeout 10000. May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Connection allowed after the host key has been verified May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Connected via SSH. May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Creating tmp directory (/tmp) if it does not existMay 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Verifying: java -fullversion openjdk full version "1.8.0_252-b09" May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Verifying: which scp /usr/bin/scp May 22, 2020 5:43:33 PM hudson.plugins.ec2.EC2Cloud INFO: Copying remoting.jar to: /tmp May 22, 2020 5:43:34 PM hudson.plugins.ec2.EC2Cloud INFO: Launching remoting agent (via SSH client process): ssh -o StrictHostKeyChecking=accept-new -i /tmp/ec2_1385185844708247542.pem centos@10.100.11.222 -p 22 java -jar /tmp/remoting.jar -workDir /jenkins [05/22/20 17:43:34] Launching agent $ ssh -o StrictHostKeyChecking=accept-new -i /tmp/ec2_1385185844708728885.pem centos@10.100.11.222 -p 22 java -jar /tmp/remoting.jar -workDir /jenkins command-line line 0: unsupported option "accept-new". ERROR: Unable to launch the agent for EC2 (Slaves) - Slave (i-xxxxxxx) java.io.EOFException: unexpected stream termination at hudson.remoting.ChannelBuilder.negotiate(ChannelBuilder.java:415) at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:360) at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:423) at hudson.slaves.CommandLauncher.launch(CommandLauncher.java:165) at hudson.plugins.ec2.ssh.EC2UnixLauncher.launchScript(EC2UnixLauncher.java:257) at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:48) at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:296) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) We still see the  ssh -o StrictHostKeyChecking=accept-new

          kanshi If you are using that old version of SSH (which doesn't support these options), you sadly need to keep the Host Key Verification Strategy setting of your AMI templates to off (the least secure, but the only compatible version).
          AFAIK, this was the value used before the new release of this plugin anyway.

          Jonathan Ballet added a comment - kanshi If you are using that old version of SSH (which doesn't support these options), you sadly need to keep the Host Key Verification Strategy setting of your AMI templates to off (the least secure, but the only compatible version). AFAIK, this was the value used before the new release of this plugin anyway.

          Antoine Hamon added a comment -

          Thanks multani I indeed missed that

          Antoine Hamon added a comment - Thanks multani I indeed missed that

            mramonleon Ramon Leon
            multani Jonathan Ballet
            Votes:
            4 Vote for this issue
            Watchers:
            16 Start watching this issue

              Created:
              Updated:
              Resolved: