Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-68177

After running upgrades to latest version Git can not longer connect to our git repository

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: Blocker Blocker
    • git-plugin
    • Jenkins v2.332.1
      Git plugin v4.11.0
      Credentials v1087.v16065d268466
      CentOS 7.4

      Before upgrading we were able to connect to our code repository with no problem via Source Code Management configuration in a Freestyle jenkins job. After running upgrades we are getting the following error on the job configuration page: 

      Failed to connect to repository : Command "git ls-remote -h ssh://git@code-repo/path/to/repo.git HEAD" returned status code 128:
      stdout:
      stderr: ssh_exchange_identification: Connection closed by remote host
      fatal: Could not read from remote repository.
      
      Please make sure you have the correct access rights
      and the repository exists.

      If I ssh onto the host and change user to the jenkins user.. I have no problem accessing our repository via cloning or ssh -T git@repourl. I can confirm that the execute shell is retrieving the correct key by running the ssh -T command in the execute shell of the job. Even when doing that from the job I get this error: `stderr: ssh_exchange_identification: Connection closed by remote host`.

      I have created new credentials in jenkins UI and attached them to the Credentials option under Source Code Management in the job and are still receiving this error.

      I have tried downgrading but some of the upgrades from where I updated from are pretty major and downgrading did not go smoothly. 

      This is stopping us from deploying code as we cannot connect to our repository using the Source Code Management tool in the Jenkins UI.

      I have no problem cloning the repo from command line. I have tried deleting the workspace, downgrading a few of the plugins, adding the Wipe out repository & force clone in Additional Behaviors under Source Code Management, and a few others.

      Here is a bigger trace when I run `ssh -T git@repourl` from the execute shell:

      Running as SYSTEM
      [EnvInject] - Loading node environment variables.
      Building in workspace /var/lib/jenkins/workspace/git-test
      The recommended git tool is: NONE
      using credential private-id-of-key
       > git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/git-test/.git # timeout=10
      Fetching changes from the remote Git repository
       > git config remote.origin.url ssh://git@repo.com/path/to/repo.git # timeout=10
      Fetching upstream changes from ssh://git@repo.com/path/to/repo.git
       > git --version # timeout=10
       > git --version # 'git version 1.8.3.1'
      using GIT_SSH to set credentials jenkins key for connecting to repo
       > git fetch --tags --progress ssh://git@repo.com/path/to/repo.git +refs/heads/*:refs/remotes/origin/* # timeout=10
      ERROR: Error fetching remote repo 'origin'
      hudson.plugins.git.GitException: Failed to fetch from ssh://git@repo.com/path/to/repo.git
      	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:993)
      	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1234)
      	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1294)
      	at hudson.scm.SCM.checkout(SCM.java:540)
      	at hudson.model.AbstractProject.checkout(AbstractProject.java:1215)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:645)
      	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:517)
      	at hudson.model.Run.execute(Run.java:1896)
      	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
      	at hudson.model.ResourceController.execute(ResourceController.java:101)
      	at hudson.model.Executor.run(Executor.java:442)
      Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress ssh://git@repo.com/path/to/repo.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
      stdout: 
      stderr: trace: built-in: git 'fetch' '--tags' '--progress' 'ssh://git@repo.com/path/to/repo.git' '+refs/heads/*:refs/remotes/origin/*'
      trace: run_command: '/var/lib/jenkins/workspace/git-test@tmp/jenkins-gitclient-ssh2530634127535164370.sh-copy' 'git@repo.com' 'git-upload-pack '\''/path/to/repo.git'\'''
      write: Broken pipe
      fatal: Could not read from remote repository.
      
      Please make sure you have the correct access rights
      and the repository exists.
      
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2671)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2096)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:84)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:618)
      	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:991)
      	... 11 more
      ERROR: Error fetching remote repo 'origin'
      [Slack Notifications] found #23 as previous completed, non-aborted build
      Finished: FAILURE
      

       

       

          [JENKINS-68177] After running upgrades to latest version Git can not longer connect to our git repository

          Zach added a comment -

          Well, I can't say that was the response I was expecting to get. I was planning to dive into why this functionality broke after upgrading jenkins once I had the time. I guess if its not your issue you can close out this bug report. Im still happy to report back whatever ends up fixing it for me (hopefully ssh_config changes like noted above) so if others run into this issue they have a fix! 

          Zach added a comment - Well, I can't say that was the response I was expecting to get. I was planning to dive into why this functionality  broke after   upgrading jenkins once I had the time. I guess if its not your issue you can close out this bug report. Im still happy to report back whatever ends up fixing it for me (hopefully ssh_config changes like noted above) so if others run into this issue they have a fix! 

          Mark Waite added a comment -

          We can't duplicate the issue you're seeing dapple. It seems specific to your environment and is best investigated in your environment. We'd love to hear what you learn from your investigation.

          Mark Waite added a comment - We can't duplicate the issue you're seeing dapple . It seems specific to your environment and is best investigated in your environment. We'd love to hear what you learn from your investigation.

          Dyllan added a comment -

          I'm experiencing the exact same issue.

          Dyllan added a comment - I'm experiencing the exact same issue.

          Mark Waite added a comment -

          dyllan hopefully the suggestions elsewhere in this issue report can help you identify what's different about your environment and the environments of others. We can't duplicate the problem.

          Mark Waite added a comment - dyllan hopefully the suggestions elsewhere in this issue report can help you identify what's different about your environment and the environments of others. We can't duplicate the problem.

          Dyllan added a comment -

          markewaite yup fair enough, although I believe there are a lot more people who are experiencing the same issue but haven't reported it here. I've done a deep dive online and there are quite a few people describing the same problem but just in different ways. I unfortunately do not have the luxury of reverting to a backup as my oldest backup includes the update and downgrading wont work due to the plugin dependency issues. For interest I've tried using an app password on Bitbucket with the same result.

          Dyllan added a comment - markewaite yup fair enough, although I believe there are a lot more people who are experiencing the same issue but haven't reported it here. I've done a deep dive online and there are quite a few people describing the same problem but just in different ways. I unfortunately do not have the luxury of reverting to a backup as my oldest backup includes the update and downgrading wont work due to the plugin dependency issues. For interest I've tried using an app password on Bitbucket with the same result.

          Mark Waite added a comment -

          dyllan a Bitbucket app password uses https as its transport. Since it uses https, then the ideas in this bug report related to ssh configuration changes won't help in your case. This issue is specifically focused on the ssh transport for git, not the https transport. If you're seeing an issue with Bitbucket app passwords, then that's a different issue than this issue and should be reported as a different issue.

          Mark Waite added a comment - dyllan a Bitbucket app password uses https as its transport. Since it uses https, then the ideas in this bug report related to ssh configuration changes won't help in your case. This issue is specifically focused on the ssh transport for git, not the https transport. If you're seeing an issue with Bitbucket app passwords, then that's a different issue than this issue and should be reported as a different issue.

          Dyllan added a comment -

          Thanks markewaite you're right.

          Dyllan added a comment - Thanks markewaite you're right.

          Basil Crow added a comment -

          Try the debug build from https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/git-client/3.11.1-rc3062.0ff23e738b_16/ and compare the results to running the same Git command in an interactive shell as the Jenkins user with GIT_SSH_COMMAND=ssh -vvv.

          Basil Crow added a comment - Try the debug build from https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/git-client/3.11.1-rc3062.0ff23e738b_16/ and compare the results to running the same Git command in an interactive shell as the Jenkins user with GIT_SSH_COMMAND=ssh -vvv .

          We had the same issue, and we're using IPA too. The problem for us was caused by ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h that IPA adds to /etc/ssh/ssh_config.

          Issue was resolved by adding .ssh/config in jenkins home directory to disable it:

           

          Host *

             ProxyCommand none

          The issue obviously related to IPA, because on Jenkins slaves (which are not registered in IPA) this did not happen.

          Paulina Budzon added a comment - We had the same issue, and we're using IPA too. The problem for us was caused by ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h  that IPA adds to /etc/ssh/ssh_config. Issue was resolved by adding .ssh/config in jenkins home directory to disable it:   Host *    ProxyCommand none The issue obviously related to IPA, because on Jenkins slaves (which are not registered in IPA) this did not happen.

          Mark Waite added a comment -

          Closing as "Cannot reproduce" based on confirmation from others that they see the same problem when IPA is configured in their SSH global configuration. The git plugin won't attempt to override that level of global configuration.

          Mark Waite added a comment - Closing as "Cannot reproduce" based on confirmation from others that they see the same problem when IPA is configured in their SSH global configuration. The git plugin won't attempt to override that level of global configuration.

            Unassigned Unassigned
            dapple Zach
            Votes:
            2 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: