Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53134

Git checkout fails when using an SSH key with a passphrase

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Duplicate
    • git-plugin
    • None
    • Jenkins core 2.121.2 and 2.107.x
      Git Plugin latest
      Git api latest

    Description

      Issue:
      A checkout from the master can not be performed when using a ssh key with a passphrase. It will hang on this step:

      > git fetch --tags --progress git@github.com:alexanderrtaylor/support-kb-articles.git +refs/heads/*:refs/remotes/origin/*
      

      And eventually fail.

      Workaround:
      You can remove the passphrase and it will work normally

      Steps to reproduce:
      1. Create a SSH key with a specific passphrase like `testing` and add it to Jenkins
      2. Add ssh key to github
      3. Create a job(freestyle or pipeline the behavior is the same)
      4. Use the SSH key connect to the ssh location of the git url
      5. Watch as the job fails.

      This works normally if I use a shell step to clone from the agent but the git plugin seems to be causing the issue

      Attachments

        Issue Links

          Activity

            dnusbaum Devin Nusbaum added a comment -

            ataylor What operating system(s) are you testing on? Are you sure this isn't just a quirk of your local ssh setup? For example, on macOS, by default it looks like the git plugin is trying to read my ssh passphrase from stdin, rather than using the credentials in Jenkins, but I think that is an macOS-specific quirk (and I have not yet found a way to make it work).

            dnusbaum Devin Nusbaum added a comment - ataylor What operating system(s) are you testing on? Are you sure this isn't just a quirk of your local ssh setup? For example, on macOS, by default it looks like the git plugin is trying to read my ssh passphrase from stdin, rather than using the credentials in Jenkins, but I think that is an macOS-specific quirk (and I have not yet found a way to make it work).
            markewaite Mark Waite added a comment -

            Can you provide more details of the configuration where you've seen the problem?

            I run passphrase protected ssh private keys frequently in my regression testing environments with Windows, CentOS, Debian, Ubuntu, and some FreeBSD. I've not detected specific problems, though I've also not performed an exhaustive test of all configurations and all alternatives.

            There was a report that clone with command line git 2.17 and later on Windows may hang with passphrase protected private keys, while clone with command line git 2.9.3 on Windows was successful.

            Are you using a passphrase which includes a character that is special to the shell (like '*' or '$')? If so, that is a known problem. Command line git uses ssh for its private key authentication. Ssh expects to receive the passphrase from stdin. The plugin provides it from stdin with a shell script that uses the `echo` command. The `echo` command performs shell expansion on its arguments. Thus, shell special characters are not allowed in a passphrase with the git plugin. A pull request is pending which resolves that, but it is not ready for production.

            markewaite Mark Waite added a comment - Can you provide more details of the configuration where you've seen the problem? I run passphrase protected ssh private keys frequently in my regression testing environments with Windows, CentOS, Debian, Ubuntu, and some FreeBSD. I've not detected specific problems, though I've also not performed an exhaustive test of all configurations and all alternatives. There was a report that clone with command line git 2.17 and later on Windows may hang with passphrase protected private keys, while clone with command line git 2.9.3 on Windows was successful. Are you using a passphrase which includes a character that is special to the shell (like '*' or '$')? If so, that is a known problem. Command line git uses ssh for its private key authentication. Ssh expects to receive the passphrase from stdin. The plugin provides it from stdin with a shell script that uses the `echo` command. The `echo` command performs shell expansion on its arguments. Thus, shell special characters are not allowed in a passphrase with the git plugin. A pull request is pending which resolves that, but it is not ready for production.
            ataylor Alex Taylor added a comment -

            markewaitednusbaum
            This is MacOS and I see the same behavior as the customer who was running on windows

            >Are you using a passphrase which includes a character that is special to the shell

            Password was `testing` in my case

            I have not tried in Linux yet but I can certainly do it if you like. Could I attach a bundle of my instance where I reproduced so you can spin up a copy?

            ataylor Alex Taylor added a comment - markewaite dnusbaum This is MacOS and I see the same behavior as the customer who was running on windows >Are you using a passphrase which includes a character that is special to the shell Password was `testing` in my case I have not tried in Linux yet but I can certainly do it if you like. Could I attach a bundle of my instance where I reproduced so you can spin up a copy?
            dnusbaum Devin Nusbaum added a comment -

            ataylor What is the error message you are seeing, and what credentials type are you using in Jenkins (username/password, ssh username w/ private key, etc.)? I was able to get things working on a Linux machine, and can try again on Windows shortly.

            dnusbaum Devin Nusbaum added a comment - ataylor What is the error message you are seeing, and what credentials type are you using in Jenkins (username/password, ssh username w/ private key, etc.)? I was able to get things working on a Linux machine, and can try again on Windows shortly.
            markewaite Mark Waite added a comment -

            A bundle would be fine. After seeing this report, I ran passphrase based tests on Windows and Linux with different versions of command line git. All of them passed. I don't have a macOS machine available.

            The closest I have to macOS is my FreeBSD machine and it uses a passphrase protected private key as well as my Linux and Windows machines.

            markewaite Mark Waite added a comment - A bundle would be fine. After seeing this report, I ran passphrase based tests on Windows and Linux with different versions of command line git. All of them passed. I don't have a macOS machine available. The closest I have to macOS is my FreeBSD machine and it uses a passphrase protected private key as well as my Linux and Windows machines.
            markewaite Mark Waite added a comment -

            Is your Jenkins server process or agent process running in the foreground? Meaning, was it started from a shell prompt and is still running from a shell prompt? JNLP agents might be started that way. Docker-based servers are commonly started that way.

            OpenSSH will prompt for interactive input of the passphrase if it detects a terminal attached to its standard input.

            The workaround for the agent is to prefix the JNLP start command with `setsid`. That detaches the terminal. OpenSSH won't see a terminal and thus won't prompt.

            markewaite Mark Waite added a comment - Is your Jenkins server process or agent process running in the foreground? Meaning, was it started from a shell prompt and is still running from a shell prompt? JNLP agents might be started that way. Docker-based servers are commonly started that way. OpenSSH will prompt for interactive input of the passphrase if it detects a terminal attached to its standard input. The workaround for the agent is to prefix the JNLP start command with `setsid`. That detaches the terminal. OpenSSH won't see a terminal and thus won't prompt.
            dnusbaum Devin Nusbaum added a comment -

            markewaite Thanks for the pointer, I used https://github.com/jerrykuch/ersatz-setsid to run a JNLP agent on macOS using setsid and am no longer seeing errors on macOS using a credential of type 'SSH Username w/ Private Key' in Jenkins.

            dnusbaum Devin Nusbaum added a comment - markewaite Thanks for the pointer, I used https://github.com/jerrykuch/ersatz-setsid to run a JNLP agent on macOS using setsid and am no longer seeing errors on macOS using a credential of type 'SSH Username w/ Private Key' in Jenkins.
            markewaite Mark Waite added a comment -

            dnusbaum there is a setting in the git plugin which will prefix every command line git call with `setsid` (on platforms where a setsid program is available). However, it is a system property and not readily available to typical users.

            markewaite Mark Waite added a comment - dnusbaum there is a setting in the git plugin which will prefix every command line git call with `setsid` (on platforms where a setsid program is available). However, it is a system property and not readily available to typical users.
            ataylor Alex Taylor added a comment -

            dnusbaumsorry missed this. I used ssh with private key when I did my test. That is interesting about the setsid but I was not actually doing this checkout on an agent and was instead doing it on the master.

            Did you guys run it on the master as well?

            ataylor Alex Taylor added a comment - dnusbaum sorry missed this. I used ssh with private key when I did my test. That is interesting about the setsid but I was not actually doing this checkout on an agent and was instead doing it on the master. Did you guys run it on the master as well?
            markewaite Mark Waite added a comment -

            ataylor the crucial attribute is whether the process that invokes command line git (agent or master) has a controlling terminal attached. If you ran "jenkins.war" from the command line, then you were running in the foreground unless you took special steps like:

            • prefix command with `setsid`
            • redirect stdin from /dev/null (maybe, never tried this one, but think it should work)
            • run in the background and redirect stdin from /dev/null (maybe, never tried this one, but think it should work)
            markewaite Mark Waite added a comment - ataylor the crucial attribute is whether the process that invokes command line git (agent or master) has a controlling terminal attached. If you ran "jenkins.war" from the command line, then you were running in the foreground unless you took special steps like: prefix command with `setsid` redirect stdin from /dev/null (maybe, never tried this one, but think it should work) run in the background and redirect stdin from /dev/null (maybe, never tried this one, but think it should work)
            markewaite Mark Waite added a comment -

            I have not checked the behavior of a master running on Windows, just agents on Windows.

            markewaite Mark Waite added a comment - I have not checked the behavior of a master running on Windows, just agents on Windows.
            dnusbaum Devin Nusbaum added a comment -

            ataylor On my macOS master, it doesn't work for the reasons that Mark explained (the terminal is interactive, so it prompts for the key at the command line, which doesn't work). Once I used setsid and a local JNLP agent it worked fine.

            Are you getting a timeout, or a public key error?

            dnusbaum Devin Nusbaum added a comment - ataylor On my macOS master, it doesn't work for the reasons that Mark explained (the terminal is interactive, so it prompts for the key at the command line, which doesn't work). Once I used setsid and a local JNLP agent it worked fine. Are you getting a timeout, or a public key error?
            ataylor Alex Taylor added a comment -

            markewaite Yeah I think the setsid is probably affecting me because I am running Jenkins from the terminal.

            But my question would be how do I go about adding setsid on a master? I could normally add it onto the prefix to the agent but if I am doing a checkout on master then how would I add it?

            ataylor Alex Taylor added a comment - markewaite Yeah I think the setsid is probably affecting me because I am running Jenkins from the terminal. But my question would be how do I go about adding setsid on a master? I could normally add it onto the prefix to the agent but if I am doing a checkout on master then how would I add it?
            markewaite Mark Waite added a comment -

            If the master is running as a service (for example, as installed by the rpm file on CentOS, the deb file on Debian, the pkg file on FreeBSD), then setsid is not needed. Services are already running without a controlling terminal.

            If the master is running in the foreground and your computer has a command `setsid` in the PATH, then you could pass the java property:

            org.jenkinsci.plugins.gitclient.CliGitAPIImpl.useSETSID=true
            
            markewaite Mark Waite added a comment - If the master is running as a service (for example, as installed by the rpm file on CentOS, the deb file on Debian, the pkg file on FreeBSD), then setsid is not needed. Services are already running without a controlling terminal. If the master is running in the foreground and your computer has a command `setsid` in the PATH, then you could pass the java property: org.jenkinsci.plugins.gitclient.CliGitAPIImpl.useSETSID=true
            dnusbaum Devin Nusbaum added a comment -

            ataylor There is no setsid shell command on macOS, so you have to compile https://github.com/jerrykuch/ersatz-setsid locally (make setsid), but once you do that I would expect that you could just run `/path/to/setsid java -jar jenkins.war` and then run a build on on the master.

            dnusbaum Devin Nusbaum added a comment - ataylor There is no setsid shell command on macOS, so you have to compile https://github.com/jerrykuch/ersatz-setsid locally ( make setsid ), but once you do that I would expect that you could just run `/path/to/setsid java -jar jenkins.war` and then run a build on on the master.
            markewaite Mark Waite added a comment -

            ataylor the conversation seems to indicate that the problem you're detecting is related to executing Jenkins in the foreground on a machine that does not have the setsid command (not available on macOS or FreeBSD). I've marked this as resolved as a duplicate of JENKINS-20879.

            If I've misunderstood, please reopen the bug with more description of the failure mode.

            markewaite Mark Waite added a comment - ataylor the conversation seems to indicate that the problem you're detecting is related to executing Jenkins in the foreground on a machine that does not have the setsid command (not available on macOS or FreeBSD). I've marked this as resolved as a duplicate of JENKINS-20879 . If I've misunderstood, please reopen the bug with more description of the failure mode.

            People

              Unassigned Unassigned
              ataylor Alex Taylor
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: