After further looking into this, Debian 9 (Stretch) is EOL as of June 2022. Debian Buster includes a newer version of OpenSSH that should support the "accept-new" option.

Thanks for checking koan00. I confirm that the Jenkins project does not support Linux operating systems that are no longer supported by the operating system provider. I intentionally dropped Debian 9 from my test environments at the end of June 2022 when Debian 9 reached end of life.

This should probably be reopened. It breaks Amazon Linux 2.0 as well - which is well and truly supported by AWS...

From the Amazon Linux FAQ:

Q. When will support for Amazon Linux 2 end?

Amazon Linux 2 end of support date (End of Life, or EOL) has been extended by one year from 2023-06-30 to 2024-06-30 to provide customers with ample time to migrate to Amazon Linux 2022. Amazon Linux 2022 Generally Available release is scheduled for later this year, and will be supported for five years.

We are also having the same issue. We are running Jenkins on a server with a CentOS 7 distribution. That provides OpenSSH_7.4p1 as well. CentOS 7 is supported until June 2024.

I've updated the plugin documentation to note this problem. I won't have time to do more than that for at least a week, possibly much more than that.

Users of Linux variants that provide old OpenSSH versions (OpenSSH 7.4 was released over 5 years ago) have the following options that are available now:

• Use the "Manually provided keys" host key verification strategy. I've attached host keys for several popular providers at the end of this comment
• Use the "Known hosts file" host key verification strategy and provide a known_hosts file on the agents with values for the required hosts
• Enable JGit and use JGit instead of command line git on those older OpenSSH versions
• Switch the repository URL's in job definitions from ssh protocol to https protocol and provide a username / password credential for the clone instead of a private key credential
• Use the "No verification" host key verification strategy (not recommended)

Known hosts values from servers tested as part of git client plugin development

Here are some of the host keys used by hosts tested in git client plugin development:

bitbucket.org ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAubiN81eDcafrgMeLzaFPsw2kNvEcqTKl/VqLat/MaB33pZy0y3rJZtnqwR2qOOvbwKZYKiEO1O6VqNEBxKvJJelCq0dTXWT5pbO2gDXC6h6QDXCaHo6pOHGPUy+YBaGQRGuSusMEASYiWunYN0vCAI8QaXnWMXNMdFP3jHAJH0eDsoiGnLPBlBp4TNm6rYI74nMzgz3B9IikW4WVK+dc8KZJZWYjAuORU3jc1c/NPskD2ASinf8v3xnfXeukU0sJ5N6m5E8VLjObPEO+mN2t/FZTMZLiFqPWc/ALSqnMnnhwrNi2rbfg/rd/IpL8Le3pSBne8+seeFVBoGqzHM9yXw==
git.assembla.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCgF2nbN5k2W4gUAkyE7+iI5T5Ng2f5xiL4+gGkARwRhftogPzqYi3+IM4PZnDW6nKPzWGo+b+mx2LR4FO0+2IE=
git.assembla.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN+whKLd9tzS4IIbZD7rCgly2LNxlvxef4JvwSaL/YZ7
git.assembla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAr06hg4ep5IKc85rjYZkEmer+BMkpLfsYi7iOMWytiqyow0jqMiVg7t32a4WW0349QV4rmfcMg5ULuIP//mBjSb2G1jSmqPvWx4JCzZLKu3nmsx1YdBWmVvn/qyVLQ5E+NCSkJJORKqIIgiL1WuAEaxUODYmBExwv1eMZWBpyeKZrrsBEmogKt2Q+igQ31+rb1fQI9/hDE+SktOpT9sWJ9brSDgsQNXAtV52DFDA8mzq7FL+4TbLjDoDdK7v3LFYn2cnN2kD57gW8qvP6V6mrkjc6I2aCNBHwdcIynUQ8HKdU+Sd4u7pZsCCK7odgPCcM5Jj+/PladSR8+DCtvs6JWw==
github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=
github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl
github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
gitlab.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY=
gitlab.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAfuCHKVTjquxvt6CM6tdG4SLp1Btn/nOeHHE5UOzRdf
jenkins-git-plugin.git.beanstalkapp.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAs8kTMKf0NYeC1msU/qSYI8pdBzU+CWr1vPNAEaLKY1647UpZ5XtaGvU4JYZVpHXE8GJr0mmOUZAkpLkoj9I70qJFCpAQkxB+VDotxEgZ1SVwci/RLdoDbyRfHnt0gnwyiP5mzLXw0aBNvyIHcU9dSYxe5ZVMwSbxsOtwWeaPap1IKjGIiR/6UJ93+md+7nhbhf9A3QNHzF9w91DjJLwX4OvutodGi7Y8ox+17+PHujcz5WshEz8rIpE2GqatQydF3dKmtbGhR1baLsNqjL2PzE9A6E+to3+0KCUp/CH7whMwIdMM8pEHp6EX9o0bmPIq4uwFggvlCU6dAWuN7jCgXQ==
vs-ssh.visualstudio.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H

My apologies for the disruption. I have CentOS 7 in my test environment and should have detected this failure during my testing. I am sure that I've missed some automated tests that would have detected this problem before release.

The accept-new option seems to be added in OpenSSH 7.6. The pre-7.6 usage is covered at: https://repology.org/project/openssh/versions

I was also hit by this issue, just spotted out our pipelines filed. I'm also on CentOS 7.

What is wrong with this whole project recently? Week after week something breaks completely. 

And now this... not only it breaks compatibility, but also lowers security.

Well done.

kredens I'm sincerely sorry that I missed the 5 year old versions of OpenSSH in my testing of the security fix. I'll implement better automation and better interactive testing for future releases.

I'm not seeing how this lowers security. Can you explain further how you believe it lowers security?

As far as I can tell, the change allows users to choose the old behavior (don't check host keys at all) or choose one of several ways to enhance the security of their installation by checking the host keys from git repository providers. It is most unfortunate that the default method does not work as expected on OpenSSH versions prior to 7.6, but I don't see how that lowers security.

How is the "Accept-new" more secure than the previous default behaviour?

Anyway - please excuse my tone, I had to roll back manually some plugins twice within a little more than a week, on a software that needs to be restarted completely to either update or do whatever to its plugins - it gets a little annoying.

Apparently OpenSSH 7.4 is still quite widely used around the world. Also sad reality is some of us don't have much of a word on what we're running our software on.

 How is the "Accept-new" more secure than the previous default behaviour?

The ssh man page describes it best. It says:

If this flag is set to accept-new then ssh will automatically add new host keys to the user's known_hosts file, but will not permit connections to hosts with changed host keys.

The crucial improvement there is "not permit connections to hosts with changed host keys". If a host key has been used once, it becomes known and will then be reused. If the host key changes, then the connection is rejected.

markewaite - May I suggest that a CentOS 7 test be added for future tests?

I believe that Amazon base their Amazon Linux 2 on a CentOS 7 base - but with a ton of customisation.

Amazon Linux 2022 when it becomes available properly is based on Fedora - so it should already have newer versions of a lot of things - however it may well be worthwhile having a Fedora test (if it isn't already there) for the sake of completion.

Thanks crcinau. Thanks for the suggestion. I already run the tests frequently on CentOS 7, Red Hat Enterprise Linux 8, Ubuntu 18, Ubuntu 20, Ubuntu 22, FreeBSD 12, FreeBSD 13, Debian 10, Debian 11, and Debian Testing. Unfortunate that no automated test caught this case in the many times that it was run on CentOS 7. Likely cause is probably a little too much mocking and not enough end to end testing in the automated tests. I'm always happy to receive pull requests that propose more automated tests for the git client plugin.

We are using centOS in our environment and faced the similar issue. As a workaround I manually added (https://updates.jenkins.io/download/plugins/git-client/3.11.0/git-client.hpi) previous version of git-client and it worked for me.

Many thanks to ashupanwar10! Downgrading Git Client Plugin from 3.11.1 to 3.11.0 completely solved the "accept-new" host key problem for my Jenkins 3.6.1 running on Amazon Linux 2. These are the versions of my Linux 2 software:

• Amazon Linux 2: aws-cli/1.18.147 Python/2.7.18 Linux/5.10.130-118.517.amzn2.x86_64 botocore/1.18.6
• SSH: OpenSSH_7.4p1, OpenSSL 1.0.2k-fips (26 Jan 2017)
• Git: 2.37.1
• Jenkins: 2.361
• Git Client Plugin: 3.11.0

To downgrade the Git Client Plugin, we can download a .hpi file of Git Client Plugin 3.11.0 from:
https://updates.jenkins.io/download/plugins/git-client/3.11.0/git-client.hpi

After downloading the .hpi file, we can follow the manually "deploy" plugin instruction of the Advanced Installation section provided by this page:
https://www.jenkins.io/doc/book/managing/plugins/#from-the-web-ui-2 

While downgrading is an option, it takes many more steps than a change to the host key verification strategy in the Jenkins global security configuration. If you switch the host key verification strategy to "No verification", then it is the same as a downgrade, without all the steps required for the downgrade.

Why a simple info about additional new option in security settings wasn't added to changelog? No one reads entire documentation after every update, that's what changelogs (or in this case - release logs) are for. So much time wasted for investigations and rollbacks...

For those facing the same issue and can't update their OpenSSH version.
Go to: https://jenkins_instance/configureSecurity/

Scroll all the way down to: "Git Host Key Verification Configuration"

It doesn't have to be "No verification" - "known_hosts" option works just fine

"known_hosts" option worked for us as a workaround - definitely give that a try first before resorting to no verification.

A prototype build of an updated git client plugin version is now available from https://ci.jenkins.io/job/Plugins/job/git-client-plugin/job/PR-882/ .

kredens, tony_teknique, syol, mrj1971, and koan00 this is your chance to help with the testing of the plugin before it is released. Please report your test results either as a comment in this issue or as a comment in https://github.com/jenkinsci/git-client-plugin/pull/882 . I've described my testing. You're encouraged to look for other ways where users might be broken by the changes.

Hey,

thanks for investigating and providing a couple of pull requests to mitigate the issue. Leaving this here for informational reasons, the current implementation is a blocker for ATH too, see https://github.com/jenkinsci/acceptance-test-harness/issues/877, because it's running on CentOS 7 as well.

Thanks koan00. Testing on a Debian 10 system would be a help for the project, since it would be checking that a system with a newer OpenSSH version was not harmed by the changes.

markewaite I have just tested a new build (git-client-3.11.2-rc3139.978264ccc070.hpi) on Ubuntu 18.04 LTS which is based on Debian 10. It works well.

Thanks theirix!

I believe I'm experiencing this as well on CentOS 7, Jenkins 2.346.2 after upgrading one of the git/pipeline related plugins (git, git-client, git, github, pipeline-groovy-lib, scm-api). I can't easily downgrade git-client because it's part of a string of dependencies related to pipelines.

I originally posted on https://issues.jenkins.io/browse/JENKINS-69207, but I think this is the correct issue I'm having.

Thank you.

P.S. markewaite Thank you for this plugin and your responsiveness.

P.P.S. The comment from syol above was the simplest resolution for me.

markroland if the log file contains a reference to accept-new, then this is the issue you're seeing. Use "Manage Jenkins" -> "Configure Global Security" -> "Git Host Key Verification Configuration" to change the configuration from "Accept first connection" to "Known hosts file". No need to downgrade the plugin. No need to restart the Jenkins controller.

If that setting does not resolve it, then you can switch to "Manually provided keys" and provide the ssh host keys for the ssh servers you use.

If that setting does not resolve it, then you can switch to "No verification" and accept that you've reintroduced the man in the middle attack vector.

Another shameless plug from me that you'll have better results if you move to a different operating system as your time and your schedules allow. CentOS 7 is supported but the utilities included in CentOS 7 are very old versions, including command line git 1.8.3.

Command line git has recently released 2.37.1 and has been releasing new minor versions roughly every 3 months. You're sacrificing performance by staying with that operating system.

Thank you and I agree. We've updated our production environments, but as you can imagine CI is lagging behind. 

markewaite  - Thanks for your detailed explanation. I have tested the latest git-client-3.11.2-rc3139.978264ccc070.hpi with my Amazon Linux 2, Jenkins 2.361, and the "Known hosts file" set in the Git Host Key Verification Configuration. This combination works great and I didn't need to downgrade any plugin.  These are my software versions installed on my Amazon Linux 2 EC2:

• Amazon Linux 2: aws-cli/1.18.147 Python/2.7.18 Linux/5.10.130-118.517.amzn2.x86_64 botocore/1.18.6
• SSH: OpenSSH_7.4p1, OpenSSL 1.0.2k-fips (26 Jan 2017)
• Git: 2.37.1
• Jenkins: 2.361
• Git Client Plugin: 3.11.2-rc3139.978264ccc070
• Use "Manage Jenkins" -> "Configure Global Security" -> "Git Host Key Verification Configuration" -> Change the configuration from "Accept first connection" to "Known hosts file"

Thanks a lot!

Thanks nokyai

Released as git client plugin 3.11.2