• EC2 Plugin 1.50.3

      Version 1.50.2 introduces security mitigations by proposing new options for SSH.

      2 of the 3 options have been introduced by SSH version 7.6:

      • ssh(1): expand the StrictHostKeyChecking option with two new
        settings. The first "accept-new" will automatically accept
        hitherto-unseen keys but will refuse connections for changed or
        invalid hostkeys. This is a safer subset of the current behaviour
        of StrictHostKeyChecking=no. The second setting "off", is a synonym
        for the current behaviour of StrictHostKeyChecking=no: accept new
        host keys, and continue connection for hosts with incorrect
        hostkeys. A future release will change the meaning of
        StrictHostKeyChecking=no to the behaviour of "accept-new". bz#2400

      Although it was released almost 3 years ago, this seriously breaks compatibility with non-recent Jenkins installations.

      For instance, the current default Docker image for Jenkins is currently based off Debian Stretch which provides SSH 7.4 and doesn't support these new options:

      $ docker run --rm -ti jenkins/jenkins:2.235 ssh -o StrictHostKeyChecking=off
      command-line line 0: unsupported option "off".
      $ docker run --rm -ti jenkins/jenkins:2.235 ssh -o StrictHostKeyChecking=accept-new
      command-line line 0: unsupported option "accept-new".
      $ docker run --rm -ti jenkins/jenkins:lts ssh -o StrictHostKeyChecking=accept-new
      command-line line 0: unsupported option "accept-new".
      

          [JENKINS-62195] ec2-1.50.2 doesn't work with SSH <7.5

          Jonathan Ballet created issue -
          David Troup made changes -
          Comment [ You can change the strategy in the config

           

          Host Key Verification Strategy in cloud config ]

          Daniel Beck added a comment -

          oleg_nenashev FYA (platform SIG)

          Daniel Beck added a comment - oleg_nenashev FYA (platform SIG)

          John Jeffers added a comment -

          Confirmed, happening here as well. We are using the latest LTS image, jenkins/jenkins:2.222.3

          root@jenkins-master-fb7584fbb-s6nnl:/# ssh -V
          OpenSSH_7.4p1 Debian-10+deb9u7, OpenSSL 1.0.2u 20 Dec 2019

          Also worth noting that when I attempted to downgrade the plugin, it did not downgrade properly and instead seemed to uninstall the plugin, taking all of its config with it. I had to manually downgrade and restore config.xml from a backup. I believe this has something to do with the ec2.xml file it drops into $JENKINS_HOME, because I could not get 1.50.1 working again until I removed that file.

          John Jeffers added a comment - Confirmed, happening here as well. We are using the latest LTS image, jenkins/jenkins:2.222.3 root@jenkins-master-fb7584fbb-s6nnl:/# ssh -V OpenSSH_7.4p1 Debian-10+deb9u7, OpenSSL 1.0.2u 20 Dec 2019 Also worth noting that when I attempted to downgrade the plugin, it did not downgrade properly and instead seemed to uninstall the plugin, taking all of its config with it. I had to manually downgrade and restore config.xml from a backup. I believe this has something to do with the ec2.xml file it drops into $JENKINS_HOME, because I could not get 1.50.1 working again until I removed that file.
          Oleg Nenashev made changes -
          Assignee Original: FABRIZIO MANFREDI [ thoulen ]
          Oleg Nenashev made changes -
          Labels New: regression

          Oleg Nenashev added a comment -

          We also hit the issues after upgrading the plugin on ci.jenkins.io which currently uses the plugin to provision agents in AWS. https://groups.google.com/forum/#!topic/jenkinsci-dev/2_WmJWSjtuc for a general discussion about agents stability, CC markewaite.

          danielbeck FYI this plugin is not really within the scope of the platform SIG. I am working to get the issue reviewed by the maintainers, but it is unlikely to happen immediately due to bank holidays, etc.

          Oleg Nenashev added a comment - We also hit the issues after upgrading the plugin on ci.jenkins.io which currently uses the plugin to provision agents in AWS.  https://groups.google.com/forum/#!topic/jenkinsci-dev/2_WmJWSjtuc  for a general discussion about agents stability, CC markewaite . danielbeck FYI this plugin is not really within the scope of the platform SIG. I am working to get the issue reviewed by the maintainers, but it is unlikely to happen immediately due to bank holidays, etc.
          Ramon Leon made changes -
          Assignee New: Ramon Leon [ mramonleon ]

          Ryan Campbell added a comment -

          Noting in case it isn't clear, that a valid workaround is to update the ssh client to a more recent version which supports these more secure options.

          Ryan Campbell added a comment - Noting in case it isn't clear, that a valid workaround is to update the ssh client to a more recent version which supports these more secure options.

          Daniel Beck added a comment -

          oleg_nenashev As this problem seems to occur because of very outdated base images, it's reasonable to inform the SIG about the consequences of that.

          Daniel Beck added a comment - oleg_nenashev As this problem seems to occur because of very outdated base images, it's reasonable to inform the SIG about the consequences of that.

            mramonleon Ramon Leon
            multani Jonathan Ballet
            Votes:
            4 Vote for this issue
            Watchers:
            16 Start watching this issue

              Created:
              Updated:
              Resolved: