I have Jenkins installed on an AWS instance (not docker).

I have two types of projects set up with two different ways to access Git repositories:

• Multibranch pipelines use the access token
• PR-builders use the ssh key.

After updating this plugin (among others) my PR-builders started to fail but the multibranch pipelines were running normally.

I accessed the Jenkins instance and was able to do a git clone from within the instance, with the Jenkins user but the PRs couldn't use the saved credentials to access the repositories.

I managed to solve it by deleting the updated version of the plugin and restoring the old one.

I have other tasks in front of me but as soon as I can, if no one finds out what happened I intend to create a new instance, install Jenkins and try to replicate the problem to see if I understand what is happening.

ladis if your AWS instance is running AWS Linux 2, CentOS 7, Oracle Linux 7, Scientific Linux 7, or some other Red Hat Enterprise Linux 7 variant, then you were affected by JENKINS-69149. You can read more about it in the git client plugin documentation. AWS Linux 2 and the other RHEL 7 variants provide OpenSSH 7.4 instead of the more modern OpenSSH versions that are provided with other operating systems. Those operating systems also provide command line git 1.8.3 instead of the much, much newer versions of command line git that are included in other operating systems.

This issue is investigating a problem that only ethorsa has been able to duplicate. If others can duplicate the problem based on the instructions from ethorsa, I'm happy to learn more. I was unable to duplicate the failure. ethorsa is continuing the investigation to identify what may be at the root of the issue.

I was able to test my steps on a pre-update instance where Git is still working without any issues.
Unlike the broken instances, that one sets the mentioned SSH envs correctly:

GIT_ASKPASS = echo
GIT_SSH = <script path>
GIT_SSH_VARIANT = ssh

Same test case (Multibranch Pipeline, ed25519 key and Pipeline code). For the sake of completeness, that's the code used:

pipeline {
    agent {
        label /* insert a suitable label here */
    }

    stages {
        stage("Test") {
            steps {
                println("x")
            }
        }
    }
}

These versions are installed on the instance:

• Jenkins 2.332.1 LTS
• Git 4.11.3
• Git Client 3.11.0
• SSH Credentials 277.v95c2fec1c047
• Pipeline Multibranch 716.vc692a_e52371b_ (same version)
• Pipeline SCM Step 400.v6b_89a_1317c9a_ (same version)

GIT_SSH and others aren't passed to Agent-side Git calls, causing missing SSH arguments, thus failing authentication:

(envs visible to the agent SSH executable)

Resulting SSH calls:

# Working instance
ssh -i <path to keyfile>.key -l git -o StrictHostKeyChecking=yes -o UserKnownHostsFile="<path to known hosts file>" -o SendEnv=GIT_PROTOCOL <Git server> git-upload-pack '/<Git repo path>'


# Broken instance
ssh -o SendEnv=GIT_PROTOCOL <Git server> git-upload-pack '/<Git repo path>'

markewaite Debugging turned out that the credentials parameter of CliGitAPIImpl.launchCommandWithCredentials() is null on the agent (and defaultCredentials is too).
Hence, the SSH key branch isn't taken and no authentication is performed.

Calls of this method on Controller or a working instance do receive a com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey object.

Credentials plugin had some recent changes related to credentials / agent transmission (changelog, PR), but I don't know whether they could relate to this issue.

I think I have a related problem. I'm seeing that the checkout, using LFS, fails ONLY when I try to build a tag, but it works when I build a branch.

I'm running Jenkins in docker and I have a workaround in adding a local .ssh folder with relevant files to the container. That way the ssh agent can use the local file resources but it also means that I cannot pass different credentials to git from Jenkins. At least not when it should check out the Jenkinsfile.

I first thought it was only related to the way known_hosts is handled since a recent update of the plugin but when I tried just adding known_hosts to the container the error changed from
Host key verification failed
to
Permission denied (publickey)
So when I added a complete .ssh folder to the container it started working.

cb2 I doubt that your issue is the same as this issue. I recommend that you submit a separate issue report with enough details that others can duplicate the issue using a fresh installation. ethorsa does not mention tags anywhere in the description and sees the failure on one system while not seeing it on another system.

Finally, the root cause is found: Authorize Projects seems to cause all the trouble.

Disabling that plugin passes all SSH parameters etc. through remoting down to on-agent Git. Fetches work flawlessly.

While multiple instances fail with "permission deneid", not all are hit by the problem though. So some instances with same versions, settings and Authorize Projects enabled work, while others fail. Howerver, once hit by the problem it wont recover.

Workaround: Disable or remove Authorize Projects plugin.

I have updated the component if this issue. Thanks so far!

Thanks for sharing the results of the investigation ethorsa!

markewaite you are correct. The investigation by ethorsa confirms it as I don't even have the "Authorize Projects" plugin installed.