Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64387

SSH private key doesn't work with git plugin on Fedora 20 and Scientific Linux 6

XMLWordPrintable

      Git Credential Manager SSH Username with private key doesn't always work with git plugin

      I have a master/slave configuration.

      Git-Master (Windows 10)
  +--Git-Slave (Ubuntu 18.04)
  +--Git-Slave (OpenSuse 15)
  +--Git-Slave (CentOS 7)
  +--Git-Slave (Fedora 20)
  +--Git-Slave (Scientific Linux 6)

      I have a matrix job configuration with a simple action of cloning a git repository to the local machine using the `git` SCM plugin.

      This job works fine for Ubuntu, OpenSuse and CentOS.

      This job fails for Fedora 20 and Scientific Linux.

      The failure message appears to be

      2020-12-01 16:17:06  > /usr/bin/git fetch --tags --progress ssh://git@git:2222/3rdParty/Dicom.git +refs/heads/*:refs/remotes/origin/* # timeout=10
2020-12-01 16:17:06 ERROR: Error cloning remote repo 'origin'
2020-12-01 16:17:06 hudson.plugins.git.GitException: Command "/usr/bin/git fetch --tags --progress ssh://git@git:2222/3rdParty/Dicom.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
2020-12-01 16:17:06 stdout: 
2020-12-01 16:17:06 stderr: Permission denied (publickey).
2020-12-01 16:17:06 fatal: Could not read from remote repository.
2020-12-01 16:17:06 
2020-12-01 16:17:06 Please make sure you have the correct access rights
2020-12-01 16:17:06 and the repository exists.

      The SSH public/private files are not located in the individual node's ~/.ssh directory. These credentials are supposed to come from the Jenkins SSH Username with private Key Credential Manager input. But it doesn't work on Fedora 20 or Scientific Linux.

      Google searching has always mentioned making sure the public/private key files are in the ~/.ssh directory, but it appears they aren't using Jenkins for credentials at that point.

      I have tried the following to debug the situation:

      1. Creds ONLY in Jenkins SSH Username with private Key Credential Manager – Fails
      2. Creds in Jenkins SSH Username and in ~/.ssh – Fails
      3. Do not use Jenkins SSH Username with private Key, only use ~/.ssh – Pass

      Obviously, I want to use the Jenkins Credential manager because it scales better. You configure it once on the master and all the slaves use it. However it's not working for fedora20 or SciLin 6.

      Other information that may be important:
      Git versions:

      Ubuntu 2.29.2
      OpenSuse 2.16.4
      CentOS 1.8.3.1
      Fedora 20 1.9.3
      Scientific Linux 2.29.2*

      * I manually compiled a new git for scilin thinking the problem may be due to the version of git that was installed. It didn't make a differrence.

      Jenkins version: 2.249.3

      Error Details

      Fedora 20 Error Details
      2020-12-01 16:17:06 Building remotely on LIN-FEDORA20 (Linux Fedora) in workspace /var/lib/jenkins/remote/workspace/TOOLS-DEBUG/JRR-GIT-Test/label/LIN-FEDORA20
2020-12-01 16:17:06 The recommended git tool is: NONE
2020-12-01 16:17:06 using credential e7237568-c0d2-4d61-973f-0f9cbbb9f16d
2020-12-01 16:17:06 Wiping out workspace first.
2020-12-01 16:17:06 Cloning the remote Git repository
2020-12-01 16:17:06 Cloning repository ssh://git@git:2222/3rdParty/Dicom.git
2020-12-01 16:17:06  > /usr/bin/git init /var/lib/jenkins/remote/workspace/TOOLS-DEBUG/JRR-GIT-Test/label/LIN-FEDORA20 # timeout=10
2020-12-01 16:17:06 Fetching upstream changes from ssh://git@git:2222/3rdParty/Dicom.git
2020-12-01 16:17:06  > /usr/bin/git --version # timeout=10
2020-12-01 16:17:06  > git --version # 'git version 1.9.3'
2020-12-01 16:17:06 using GIT_SSH to set credentials SSH
2020-12-01 16:17:06  > /usr/bin/git fetch --tags --progress ssh://git@git:2222/3rdParty/Dicom.git +refs/heads/*:refs/remotes/origin/* # timeout=10
2020-12-01 16:17:06 ERROR: Error cloning remote repo 'origin'
2020-12-01 16:17:06 hudson.plugins.git.GitException: Command "/usr/bin/git fetch --tags --progress ssh://git@git:2222/3rdParty/Dicom.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
2020-12-01 16:17:06 stdout: 
2020-12-01 16:17:06 stderr: Permission denied (publickey).
2020-12-01 16:17:06 fatal: Could not read from remote repository.
2020-12-01 16:17:06 
2020-12-01 16:17:06 Please make sure you have the correct access rights
2020-12-01 16:17:06 and the repository exists.
2020-12-01 16:17:06 
2020-12-01 16:17:06 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2450)
2020-12-01 16:17:06 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2051)
2020-12-01 16:17:06 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:84)
2020-12-01 16:17:06 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:573)
2020-12-01 16:17:06 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:802)
2020-12-01 16:17:06 	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
2020-12-01 16:17:06 	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)
2020-12-01 16:17:06 	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
2020-12-01 16:17:06 	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
2020-12-01 16:17:06 	at hudson.remoting.Request$2.run(Request.java:375)
2020-12-01 16:17:06 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
2020-12-01 16:17:06 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2020-12-01 16:17:06 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2020-12-01 16:17:06 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2020-12-01 16:17:06 	at java.lang.Thread.run(Thread.java:748)
2020-12-01 16:17:06 	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to LIN-FEDORA20
2020-12-01 16:17:06 		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
2020-12-01 16:17:06 		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
2020-12-01 16:17:06 		at hudson.remoting.Channel.call(Channel.java:1001)
2020-12-01 16:17:06 		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
2020-12-01 16:17:06 		at sun.reflect.GeneratedMethodAccessor1035.invoke(Unknown Source)
2020-12-01 16:17:06 		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-12-01 16:17:06 		at java.lang.reflect.Method.invoke(Method.java:498)
2020-12-01 16:17:06 		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
2020-12-01 16:17:06 		at com.sun.proxy.$Proxy73.execute(Unknown Source)
2020-12-01 16:17:06 		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1219)
2020-12-01 16:17:06 		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1297)
2020-12-01 16:17:06 		at hudson.scm.SCM.checkout(SCM.java:505)
2020-12-01 16:17:06 		at hudson.model.AbstractProject.checkout(AbstractProject.java:1206)
2020-12-01 16:17:06 		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
2020-12-01 16:17:06 		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
2020-12-01 16:17:06 		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
2020-12-01 16:17:06 		at hudson.model.Run.execute(Run.java:1894)
2020-12-01 16:17:06 		at hudson.matrix.MatrixRun.run(MatrixRun.java:153)
2020-12-01 16:17:06 		at hudson.model.ResourceController.execute(ResourceController.java:97)
2020-12-01 16:17:06 		at hudson.model.Executor.run(Executor.java:428)
2020-12-01 16:17:06 ERROR: Error cloning remote repo 'origin'
2020-12-01 16:17:06 Finished: FAILURE


      Scientific Linux Error Details
      2020-12-01 16:17:06 Building remotely on LIN-SCILIN-6 (Linux SciLin) in workspace /var/lib/jenkins/remote/workspace/TOOLS-DEBUG/JRR-GIT-Test/label/LIN-SCILIN-6
2020-12-01 16:17:06 The recommended git tool is: NONE
2020-12-01 16:17:06 using credential e7237568-c0d2-4d61-973f-0f9cbbb9f16d
2020-12-01 16:17:06 Wiping out workspace first.
2020-12-01 16:17:06 Cloning the remote Git repository
2020-12-01 16:17:06 Cloning repository ssh://git@git:2222/3rdParty/Dicom.git
2020-12-01 16:17:06  > /usr/bin/git init /var/lib/jenkins/remote/workspace/TOOLS-DEBUG/JRR-GIT-Test/label/LIN-SCILIN-6 # timeout=10
2020-12-01 16:17:06 Fetching upstream changes from ssh://git@git:2222/3rdParty/Dicom.git
2020-12-01 16:17:06  > /usr/bin/git --version # timeout=10
2020-12-01 16:17:06  > git --version # 'git version 1.7.1'
2020-12-01 16:17:06 using GIT_SSH to set credentials SSH
2020-12-01 16:17:06  > /usr/bin/git fetch --tags --progress ssh://git@git:2222/3rdParty/Dicom.git +refs/heads/*:refs/remotes/origin/* # timeout=10
2020-12-01 16:17:11 ERROR: Error cloning remote repo 'origin'
2020-12-01 16:17:11 hudson.plugins.git.GitException: Command "/usr/bin/git fetch --tags --progress ssh://git@git:2222/3rdParty/Dicom.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
2020-12-01 16:17:11 stdout: 
2020-12-01 16:17:11 stderr: Permission denied (publickey).
2020-12-01 16:17:11 fatal: The remote end hung up unexpectedly
2020-12-01 16:17:11 
2020-12-01 16:17:11 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2450)
2020-12-01 16:17:11 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2051)
2020-12-01 16:17:11 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:84)
2020-12-01 16:17:11 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:573)
2020-12-01 16:17:11 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:802)
2020-12-01 16:17:11 	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
2020-12-01 16:17:11 	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)
2020-12-01 16:17:11 	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
2020-12-01 16:17:11 	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
2020-12-01 16:17:11 	at hudson.remoting.Request$2.run(Request.java:375)
2020-12-01 16:17:11 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
2020-12-01 16:17:11 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2020-12-01 16:17:11 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2020-12-01 16:17:11 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2020-12-01 16:17:11 	at java.lang.Thread.run(Thread.java:748)
2020-12-01 16:17:11 	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to LIN-SCILIN-6
2020-12-01 16:17:11 		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
2020-12-01 16:17:11 		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
2020-12-01 16:17:11 		at hudson.remoting.Channel.call(Channel.java:1001)
2020-12-01 16:17:11 		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
2020-12-01 16:17:11 		at sun.reflect.GeneratedMethodAccessor1035.invoke(Unknown Source)
2020-12-01 16:17:11 		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-12-01 16:17:11 		at java.lang.reflect.Method.invoke(Method.java:498)
2020-12-01 16:17:11 		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
2020-12-01 16:17:11 		at com.sun.proxy.$Proxy73.execute(Unknown Source)
2020-12-01 16:17:11 		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1219)
2020-12-01 16:17:11 		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1297)
2020-12-01 16:17:11 		at hudson.scm.SCM.checkout(SCM.java:505)
2020-12-01 16:17:11 		at hudson.model.AbstractProject.checkout(AbstractProject.java:1206)
2020-12-01 16:17:11 		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
2020-12-01 16:17:11 		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
2020-12-01 16:17:11 		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
2020-12-01 16:17:11 		at hudson.model.Run.execute(Run.java:1894)
2020-12-01 16:17:11 		at hudson.matrix.MatrixRun.run(MatrixRun.java:153)
2020-12-01 16:17:11 		at hudson.model.ResourceController.execute(ResourceController.java:97)
2020-12-01 16:17:11 		at hudson.model.Executor.run(Executor.java:428)
2020-12-01 16:17:11 ERROR: Error cloning remote repo 'origin'
2020-12-01 16:17:11 Finished: FAILURE

          [JENKINS-64387] SSH private key doesn't work with git plugin on Fedora 20 and Scientific Linux 6

          Mark Waite added a comment -

          Fedora 20 is no longer maintained by the operating system provider. Fedora 33 is the current Fedora release.

          Scientific Linux has been announced that it is end of life as an operating system. The last update of Scientific Linux 6 was issued Nov 30, 2020 (per Wikipedia).

          I suspect in both cases that the issue is related to the version of ssh included in those operating system versions.

          You might be able to workaround the issue and continue using Jenkins credentials by generating an RSA private key / public key pair on the oldest of those configurations (probably Scientific Linux 6, but possibly Fedora 20), then install and use that RSA private key as a Jenkins credential and use it on your Git server as the credential for the repository. My guess is that the private key you're using in the Jenkins credential is not understood by those old ssh versions.

          Mark Waite added a comment - Fedora 20 is no longer maintained by the operating system provider. Fedora 33 is the current Fedora release. Scientific Linux has been announced that it is end of life as an operating system. The last update of Scientific Linux 6 was issued Nov 30, 2020 (per Wikipedia ). I suspect in both cases that the issue is related to the version of ssh included in those operating system versions. You might be able to workaround the issue and continue using Jenkins credentials by generating an RSA private key / public key pair on the oldest of those configurations (probably Scientific Linux 6, but possibly Fedora 20), then install and use that RSA private key as a Jenkins credential and use it on your Git server as the credential for the repository. My guess is that the private key you're using in the Jenkins credential is not understood by those old ssh versions.

          John Rocha added a comment -

          Hello markewaite. Thank you for the response.

          Why would the SSH key work independently on Fedora 20 and SciLin 6? Just not with the Jenkins Credential provider?

          What I mean by that is I can manually copy the id_rsa private key file to the ~/.ssh folder of all of the Linux machines and manually do the git clone with ssh and it works.

          Moreover, the jenkins git plugin will work when the id_rsa private key is in the machines ~/.ssh folder. It just fails when the credential plugin is used.

          That seems to imply that SSH is working correctly with the key on that version of Linux. It's only failing when being delivered via the Jenkins Credential Manager.

          What am I missing here?

          John Rocha added a comment - Hello markewaite . Thank you for the response. Why would the SSH key work independently on Fedora 20 and SciLin 6? Just not with the Jenkins Credential provider? What I mean by that is I can manually copy the id_rsa private key file to the ~/.ssh folder of all of the Linux machines and manually do the git clone with ssh and it works. Moreover, the jenkins git plugin will work when the id_rsa private key is in the machines ~/.ssh folder. It just fails when the credential plugin is used. That seems to imply that SSH is working correctly with the key on that version of Linux. It's only failing when being delivered via the Jenkins Credential Manager. What am I missing here?

          Mark Waite added a comment -

          I don't know rocha_stratovan.

          I've seen cases where I had to use an RSA private key instead of an ED25519 private key even though the ssh on the machine claimed to support ED25519. I've seen cases where I had to generate the RSA private key on the oldest operating system that I was trying to target even though there was no indication other than the key not being accepted. I don't know if those ideas will help you. I offer them as ideas that you might try.

          I'm not willing to spend time investigating operating systems that are past their end of life. It is enough work to maintain the operating systems that are not end of life and their versions of command line git. There have been many times that CentOS 7 and its command line git 1.8.3 (8 years old) have required special cases in the git plugin code or have required notations that a scenario does not work.

          Mark Waite added a comment - I don't know rocha_stratovan . I've seen cases where I had to use an RSA private key instead of an ED25519 private key even though the ssh on the machine claimed to support ED25519. I've seen cases where I had to generate the RSA private key on the oldest operating system that I was trying to target even though there was no indication other than the key not being accepted. I don't know if those ideas will help you. I offer them as ideas that you might try. I'm not willing to spend time investigating operating systems that are past their end of life. It is enough work to maintain the operating systems that are not end of life and their versions of command line git. There have been many times that CentOS 7 and its command line git 1.8.3 (8 years old) have required special cases in the git plugin code or have required notations that a scenario does not work.

          John Rocha added a comment -

          markewaite:

          Thank you for your continued patience and explanation.

          For future googler's that find this thread. I can confirm that it worked when I tried your suggestion of generating the key from the oldest Linux distribution I was using (fedora 20).

          The steps I did were

          1. Identify the oldest Linux machine (fedora 20)
          2. Access fedora terminal
          3. Use ssh-keygen to create an id_rsa & id_rsa.pub file in ~/.ssh
          4. Use the id_rsa file to create a new Jenkins SSH Username with private key credential
          5. Configure gitlab to use that id_rsa.pub file generated from fedora.
          6. Configure my jenkins job to use the newly added Jenkins SSH key.

          With this all of my windows and Linux systems, including Fedora20 and Scientific Linux 6 work using the one Jenkins SSH Username with private key credential entry.

          I appreciate your insights and patience.

          Cheers!

          John Rocha added a comment - markewaite : Thank you for your continued patience and explanation. For future googler's that find this thread. I can confirm that it worked when I tried your suggestion of generating the key from the oldest Linux distribution I was using (fedora 20). The steps I did were Identify the oldest Linux machine (fedora 20) Access fedora terminal Use ssh-keygen to create an id_rsa & id_rsa.pub file in ~/.ssh Use the id_rsa file to create a new Jenkins SSH Username with private key credential Configure gitlab to use that id_rsa.pub file generated from fedora. Configure my jenkins job to use the newly added Jenkins SSH key. With this all of my windows and Linux systems, including Fedora20 and Scientific Linux 6 work using the one Jenkins SSH Username with private key credential entry. I appreciate your insights and patience. Cheers!

            Unassigned Unassigned
            rocha_stratovan John Rocha
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: