Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53134

Git checkout fails when using an SSH key with a passphrase

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Major Major
    • git-plugin
    • None
    • Jenkins core 2.121.2 and 2.107.x
      Git Plugin latest
      Git api latest

      Issue:
      A checkout from the master can not be performed when using a ssh key with a passphrase. It will hang on this step:

      > git fetch --tags --progress git@github.com:alexanderrtaylor/support-kb-articles.git +refs/heads/*:refs/remotes/origin/*
      

      And eventually fail.

      Workaround:
      You can remove the passphrase and it will work normally

      Steps to reproduce:
      1. Create a SSH key with a specific passphrase like `testing` and add it to Jenkins
      2. Add ssh key to github
      3. Create a job(freestyle or pipeline the behavior is the same)
      4. Use the SSH key connect to the ssh location of the git url
      5. Watch as the job fails.

      This works normally if I use a shell step to clone from the agent but the git plugin seems to be causing the issue

          [JENKINS-53134] Git checkout fails when using an SSH key with a passphrase

          Devin Nusbaum added a comment -

          ataylor What operating system(s) are you testing on? Are you sure this isn't just a quirk of your local ssh setup? For example, on macOS, by default it looks like the git plugin is trying to read my ssh passphrase from stdin, rather than using the credentials in Jenkins, but I think that is an macOS-specific quirk (and I have not yet found a way to make it work).

          Devin Nusbaum added a comment - ataylor What operating system(s) are you testing on? Are you sure this isn't just a quirk of your local ssh setup? For example, on macOS, by default it looks like the git plugin is trying to read my ssh passphrase from stdin, rather than using the credentials in Jenkins, but I think that is an macOS-specific quirk (and I have not yet found a way to make it work).

          Mark Waite added a comment -

          Can you provide more details of the configuration where you've seen the problem?

          I run passphrase protected ssh private keys frequently in my regression testing environments with Windows, CentOS, Debian, Ubuntu, and some FreeBSD. I've not detected specific problems, though I've also not performed an exhaustive test of all configurations and all alternatives.

          There was a report that clone with command line git 2.17 and later on Windows may hang with passphrase protected private keys, while clone with command line git 2.9.3 on Windows was successful.

          Are you using a passphrase which includes a character that is special to the shell (like '*' or '$')? If so, that is a known problem. Command line git uses ssh for its private key authentication. Ssh expects to receive the passphrase from stdin. The plugin provides it from stdin with a shell script that uses the `echo` command. The `echo` command performs shell expansion on its arguments. Thus, shell special characters are not allowed in a passphrase with the git plugin. A pull request is pending which resolves that, but it is not ready for production.

          Mark Waite added a comment - Can you provide more details of the configuration where you've seen the problem? I run passphrase protected ssh private keys frequently in my regression testing environments with Windows, CentOS, Debian, Ubuntu, and some FreeBSD. I've not detected specific problems, though I've also not performed an exhaustive test of all configurations and all alternatives. There was a report that clone with command line git 2.17 and later on Windows may hang with passphrase protected private keys, while clone with command line git 2.9.3 on Windows was successful. Are you using a passphrase which includes a character that is special to the shell (like '*' or '$')? If so, that is a known problem. Command line git uses ssh for its private key authentication. Ssh expects to receive the passphrase from stdin. The plugin provides it from stdin with a shell script that uses the `echo` command. The `echo` command performs shell expansion on its arguments. Thus, shell special characters are not allowed in a passphrase with the git plugin. A pull request is pending which resolves that, but it is not ready for production.

          Alex Taylor added a comment -

          markewaitednusbaum
          This is MacOS and I see the same behavior as the customer who was running on windows

          >Are you using a passphrase which includes a character that is special to the shell

          Password was `testing` in my case

          I have not tried in Linux yet but I can certainly do it if you like. Could I attach a bundle of my instance where I reproduced so you can spin up a copy?

          Alex Taylor added a comment - markewaite dnusbaum This is MacOS and I see the same behavior as the customer who was running on windows >Are you using a passphrase which includes a character that is special to the shell Password was `testing` in my case I have not tried in Linux yet but I can certainly do it if you like. Could I attach a bundle of my instance where I reproduced so you can spin up a copy?

          Devin Nusbaum added a comment -

          ataylor What is the error message you are seeing, and what credentials type are you using in Jenkins (username/password, ssh username w/ private key, etc.)? I was able to get things working on a Linux machine, and can try again on Windows shortly.

          Devin Nusbaum added a comment - ataylor What is the error message you are seeing, and what credentials type are you using in Jenkins (username/password, ssh username w/ private key, etc.)? I was able to get things working on a Linux machine, and can try again on Windows shortly.

          Mark Waite added a comment -

          A bundle would be fine. After seeing this report, I ran passphrase based tests on Windows and Linux with different versions of command line git. All of them passed. I don't have a macOS machine available.

          The closest I have to macOS is my FreeBSD machine and it uses a passphrase protected private key as well as my Linux and Windows machines.

          Mark Waite added a comment - A bundle would be fine. After seeing this report, I ran passphrase based tests on Windows and Linux with different versions of command line git. All of them passed. I don't have a macOS machine available. The closest I have to macOS is my FreeBSD machine and it uses a passphrase protected private key as well as my Linux and Windows machines.

          Mark Waite added a comment -

          Is your Jenkins server process or agent process running in the foreground? Meaning, was it started from a shell prompt and is still running from a shell prompt? JNLP agents might be started that way. Docker-based servers are commonly started that way.

          OpenSSH will prompt for interactive input of the passphrase if it detects a terminal attached to its standard input.

          The workaround for the agent is to prefix the JNLP start command with `setsid`. That detaches the terminal. OpenSSH won't see a terminal and thus won't prompt.

          Mark Waite added a comment - Is your Jenkins server process or agent process running in the foreground? Meaning, was it started from a shell prompt and is still running from a shell prompt? JNLP agents might be started that way. Docker-based servers are commonly started that way. OpenSSH will prompt for interactive input of the passphrase if it detects a terminal attached to its standard input. The workaround for the agent is to prefix the JNLP start command with `setsid`. That detaches the terminal. OpenSSH won't see a terminal and thus won't prompt.

          Devin Nusbaum added a comment -

          markewaite Thanks for the pointer, I used https://github.com/jerrykuch/ersatz-setsid to run a JNLP agent on macOS using setsid and am no longer seeing errors on macOS using a credential of type 'SSH Username w/ Private Key' in Jenkins.

          Devin Nusbaum added a comment - markewaite Thanks for the pointer, I used https://github.com/jerrykuch/ersatz-setsid to run a JNLP agent on macOS using setsid and am no longer seeing errors on macOS using a credential of type 'SSH Username w/ Private Key' in Jenkins.

          Mark Waite added a comment -

          dnusbaum there is a setting in the git plugin which will prefix every command line git call with `setsid` (on platforms where a setsid program is available). However, it is a system property and not readily available to typical users.

          Mark Waite added a comment - dnusbaum there is a setting in the git plugin which will prefix every command line git call with `setsid` (on platforms where a setsid program is available). However, it is a system property and not readily available to typical users.

          Alex Taylor added a comment -

          dnusbaumsorry missed this. I used ssh with private key when I did my test. That is interesting about the setsid but I was not actually doing this checkout on an agent and was instead doing it on the master.

          Did you guys run it on the master as well?

          Alex Taylor added a comment - dnusbaum sorry missed this. I used ssh with private key when I did my test. That is interesting about the setsid but I was not actually doing this checkout on an agent and was instead doing it on the master. Did you guys run it on the master as well?

          Mark Waite added a comment -

          ataylor the crucial attribute is whether the process that invokes command line git (agent or master) has a controlling terminal attached. If you ran "jenkins.war" from the command line, then you were running in the foreground unless you took special steps like:

          • prefix command with `setsid`
          • redirect stdin from /dev/null (maybe, never tried this one, but think it should work)
          • run in the background and redirect stdin from /dev/null (maybe, never tried this one, but think it should work)

          Mark Waite added a comment - ataylor the crucial attribute is whether the process that invokes command line git (agent or master) has a controlling terminal attached. If you ran "jenkins.war" from the command line, then you were running in the foreground unless you took special steps like: prefix command with `setsid` redirect stdin from /dev/null (maybe, never tried this one, but think it should work) run in the background and redirect stdin from /dev/null (maybe, never tried this one, but think it should work)

          Mark Waite added a comment -

          I have not checked the behavior of a master running on Windows, just agents on Windows.

          Mark Waite added a comment - I have not checked the behavior of a master running on Windows, just agents on Windows.

          Devin Nusbaum added a comment -

          ataylor On my macOS master, it doesn't work for the reasons that Mark explained (the terminal is interactive, so it prompts for the key at the command line, which doesn't work). Once I used setsid and a local JNLP agent it worked fine.

          Are you getting a timeout, or a public key error?

          Devin Nusbaum added a comment - ataylor On my macOS master, it doesn't work for the reasons that Mark explained (the terminal is interactive, so it prompts for the key at the command line, which doesn't work). Once I used setsid and a local JNLP agent it worked fine. Are you getting a timeout, or a public key error?

          Alex Taylor added a comment -

          markewaite Yeah I think the setsid is probably affecting me because I am running Jenkins from the terminal.

          But my question would be how do I go about adding setsid on a master? I could normally add it onto the prefix to the agent but if I am doing a checkout on master then how would I add it?

          Alex Taylor added a comment - markewaite Yeah I think the setsid is probably affecting me because I am running Jenkins from the terminal. But my question would be how do I go about adding setsid on a master? I could normally add it onto the prefix to the agent but if I am doing a checkout on master then how would I add it?

          Mark Waite added a comment -

          If the master is running as a service (for example, as installed by the rpm file on CentOS, the deb file on Debian, the pkg file on FreeBSD), then setsid is not needed. Services are already running without a controlling terminal.

          If the master is running in the foreground and your computer has a command `setsid` in the PATH, then you could pass the java property:

          org.jenkinsci.plugins.gitclient.CliGitAPIImpl.useSETSID=true
          

          Mark Waite added a comment - If the master is running as a service (for example, as installed by the rpm file on CentOS, the deb file on Debian, the pkg file on FreeBSD), then setsid is not needed. Services are already running without a controlling terminal. If the master is running in the foreground and your computer has a command `setsid` in the PATH, then you could pass the java property: org.jenkinsci.plugins.gitclient.CliGitAPIImpl.useSETSID=true

          Devin Nusbaum added a comment -

          ataylor There is no setsid shell command on macOS, so you have to compile https://github.com/jerrykuch/ersatz-setsid locally (make setsid), but once you do that I would expect that you could just run `/path/to/setsid java -jar jenkins.war` and then run a build on on the master.

          Devin Nusbaum added a comment - ataylor There is no setsid shell command on macOS, so you have to compile https://github.com/jerrykuch/ersatz-setsid locally ( make setsid ), but once you do that I would expect that you could just run `/path/to/setsid java -jar jenkins.war` and then run a build on on the master.

          Mark Waite added a comment -

          ataylor the conversation seems to indicate that the problem you're detecting is related to executing Jenkins in the foreground on a machine that does not have the setsid command (not available on macOS or FreeBSD). I've marked this as resolved as a duplicate of JENKINS-20879.

          If I've misunderstood, please reopen the bug with more description of the failure mode.

          Mark Waite added a comment - ataylor the conversation seems to indicate that the problem you're detecting is related to executing Jenkins in the foreground on a machine that does not have the setsid command (not available on macOS or FreeBSD). I've marked this as resolved as a duplicate of JENKINS-20879 . If I've misunderstood, please reopen the bug with more description of the failure mode.

            Unassigned Unassigned
            ataylor Alex Taylor
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: