Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-51968

sshagent plugin broken on Windows 10 version 1803

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • ssh-agent-plugin
    • None

      In Windows 10 version 1803, Microsoft started shipping OpenSSH in Windows, which was previously an optional beta feature. This causes problems for the sshagent step in Windows, which will produce a stacktrace resembling the following:
      java.lang.RuntimeException: [ssh-agent] Could not find a suitable ssh-agent provider.
      at com.cloudbees.jenkins.plugins.sshagent.SSHAgentStepExecution.initRemoteAgent(SSHAgentStepExecution.java:175)
      at com.cloudbees.jenkins.plugins.sshagent.SSHAgentStepExecution.start(SSHAgentStepExecution.java:63)
      at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:229)
      at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:153)
      (snip!)
      The problem is that Windows' OpenSSH binaries appear in the PATH ahead of those from Git, which means ssh-agent will be executed from C:\Windows\System32\OpenSSH.

      There are two obvious workarounds for this – first, to uninstall Windows' OpenSSH (which can be done in the new Settings panel under "Manage optional features”). The other solution would be to tweak the system path so that Git's paths appear first, but this is likely to cause other problems in Windows. I also tried to set the GIT_SSH environment variable, but this did not seem to work.

      It would be nice if the ssh-plugin either supported OpenSSH on Windows natively, or was smart enough to use the mingw binary provided on the system.

      I had a quick look at the ssh-agent source code, and to support OpenSSH on Windows a fix would need to be made in the ExecRemoteAgent class. The OpenSSH agent service (which is disabled by default in Windows 10) doesn't produce the output that is expected by the ExecRemoteAgent class. Specifically, this class expects that ssh-agent will produce output resembling the following:

      SSH_AUTH_SOCK=/tmp/whatever/agent.12283; export SSH_AUTH_SOCK;
      SSH_AGENT_PID=12284; export SSH_AGENT_PID;
      echo Agent pid 12284;

      The class then attempts to extract the agent PID by parsing the text in the parseAgentEnv method. However, the ssh-agent implementation on Windows' OpenSSH client doesn't produce any output at all, which causes this exception to be thrown:
      [ssh-agent] FATAL: Could not find a suitable ssh-agent provider
      [ssh-agent] Diagnostic report
      [ssh-agent] * Exec ssh-agent (binary ssh-agent on a remote machine)
      [ssh-agent] java.lang.StringIndexOutOfBoundsException: String index out of range: -14
      [ssh-agent] at java.lang.String.substring(String.java:1967)
      [ssh-agent] at com.cloudbees.jenkins.plugins.sshagent.exec.ExecRemoteAgent.getAgentValue(ExecRemoteAgent.java:154)
      [ssh-agent] at com.cloudbees.jenkins.plugins.sshagent.exec.ExecRemoteAgent.parseAgentEnv(ExecRemoteAgent.java:138)
      (snip!)
      I didn't dig any deeper into this, but my armchair opinion is that this class needs to find the PID in some other way than parsing stdout. I'm not sure exactly what way this would be. Microsoft is aware of this issue

       

      see https://github.com/PowerShell/Win32-OpenSSH/issues/1145 but it is unclear if they will change the implementation just yet.

          [JENKINS-51968] sshagent plugin broken on Windows 10 version 1803

          Jesse Glick added a comment -

          I did in fact update my test installation of Windows 10 to 1803 and indeed saw an ssh-agent command appear, which prints nothing when run. (I did not have one before although I have some Git distribution installed, via Chocolatey.) If some S.gpg-agent.ssh file is being created, I could not find it. Nor does the command respond to /? etc.

          Jesse Glick added a comment - I did in fact update my test installation of Windows 10 to 1803 and indeed saw an ssh-agent command appear, which prints nothing when run. (I did not have one before although I have some Git distribution installed, via Chocolatey.) If some S.gpg-agent.ssh file is being created, I could not find it. Nor does the command respond to /? etc.

          Andreas Meyer added a comment -

          Just a suggestion:

          Read the path of ssh-agent from a dedicated environment variable (say JENKINS_SSH_AGENT) instead of solely relying on PATH. If JENKINS_SSH_AGENT isn't defined, fallback to the old behaviour. This way we could avoid fiddling with PATH altogether.

          Andreas Meyer added a comment - Just a suggestion: Read the path of ssh-agent from a dedicated environment variable (say JENKINS_SSH_AGENT) instead of solely relying on PATH. If JENKINS_SSH_AGENT isn't defined, fallback to the old behaviour. This way we could avoid fiddling with PATH altogether.

          ethorsa added a comment -

          Is there any workaround known?

          ethorsa added a comment - Is there any workaround known?

          Jesse Glick added a comment -

          You can skip this plugin and just use the withCredentials step to bind an SSH private key to a local file, which can then be passed to various commands typically with a -i option. In a CI context I would not consider this any less secure than using ssh-agent, and it is certainly easier to troubleshoot.

          Jesse Glick added a comment - You can skip this plugin and just use the withCredentials step to bind an SSH private key to a local file, which can then be passed to various commands typically with a -i option. In a CI context I would not consider this any less secure than using ssh-agent , and it is certainly easier to troubleshoot.

          Carel Combrink added a comment -

          Any progress on this, I also have this issue using the Windows OpenSSH

          I get the following

          [ssh-agent] Looking for ssh-agent implementation...
[ssh-agent]   Exec ssh-agent (binary ssh-agent on a remote machine)
$ ssh-agent
[ssh-agent] FATAL: Could not find a suitable ssh-agent provider

          If I run my container with: "PowerShell -c "Get-Command ssh-agent""
          I get

          CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Application     ssh-agent.exe                                      8.1.0.1    C:\Windows\System32\OpenSSH\ssh-agent.exe

           

          Carel Combrink added a comment - Any progress on this, I also have this issue using the Windows OpenSSH I get the following [ssh-agent] Looking for ssh-agent implementation... [ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine) $ ssh-agent [ssh-agent] FATAL: Could not find a suitable ssh-agent provider If I run my container with: "PowerShell -c "Get-Command ssh-agent"" I get CommandType Name Version Source ----------- ---- ------- ------ Application ssh-agent.exe 8.1.0.1 C:\Windows\System32\OpenSSH\ssh-agent.exe  

          Muzammil Mohammad added a comment -

          carelc i am also facing the same issue, did you resolved it?

          Muzammil Mohammad added a comment - carelc i am also facing the same issue, did you resolved it?

          Carel Combrink added a comment -

          muzammil_6701
          I installed the windows git command line tools, that include ssh-agent and removed the Windows ssh-agent from my path,

          The Powershell commands in my Dockerfile that does this:

          RUN ${tmp_path} = ${env:PATH} `
  ; ${tmp_path} = (${tmp_path}.Split(';') | Where-Object { $_ -ne 'C:\Windows\System32\OpenSSH\' }) -join ';' `
  ; ${tmp_path} = 'C:\Program Files\Git\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Git\mingw64\bin\;' + ${tmp_path} `
  ; [System.Environment]::SetEnvironmentVariable('PATH', ${tmp_path},[System.EnvironmentVariableTarget]::Machine) `

          Carel Combrink added a comment - muzammil_6701 I installed the windows git command line tools, that include ssh-agent and removed the Windows ssh-agent from my path, The Powershell commands in my Dockerfile that does this: RUN ${tmp_path} = ${env:PATH} ` ; ${tmp_path} = (${tmp_path}.Split( ';' ) | Where- Object { $_ -ne 'C:\Windows\System32\OpenSSH\' }) -join ';' ` ; ${tmp_path} = 'C:\Program Files\Git\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Git\mingw64\bin\;' + ${tmp_path} ` ; [ System .Environment]::SetEnvironmentVariable( 'PATH' , ${tmp_path},[ System .EnvironmentVariableTarget]::Machine) `

          Zaitcev Peter added a comment -

          I've changed agent PATH so it uses Git's SSH-Agent.
          However, it still fails with the same error.

          00:00:02.890  locadmin@WIN10-VM-TEST d:\Jenkins_AppCTS\workspace\ssh-agent-test-pipeline>where ssh-agent 
00:00:02.890  C:\Git\usr\bin\ssh-agent.exe
00:00:04.465  [Pipeline] sshagent
00:00:04.470  FATAL: [ssh-agent] Could not find specified credentials
00:00:04.470  [ssh-agent] Looking for ssh-agent implementation...
00:00:04.521  Could not find ssh-agent: IOException: Cannot run program "ssh-agent": CreateProcess error=2, The system cannot find the file specified
00:00:04.521  Check if ssh-agent is installed and in PATH
00:00:04.521  [ssh-agent] FATAL: Could not find a suitable ssh-agent provider
00:00:04.521  [ssh-agent] Diagnostic report

          Zaitcev Peter added a comment - I've changed agent PATH so it uses Git's SSH-Agent. However, it still fails with the same error. 00:00:02.890 locadmin@WIN10-VM-TEST d:\Jenkins_AppCTS\workspace\ssh-agent-test-pipeline>where ssh-agent 00:00:02.890 C:\Git\usr\bin\ssh-agent.exe 00:00:04.465 [Pipeline] sshagent 00:00:04.470 FATAL: [ssh-agent] Could not find specified credentials 00:00:04.470 [ssh-agent] Looking for ssh-agent implementation... 00:00:04.521 Could not find ssh-agent: IOException: Cannot run program "ssh-agent" : CreateProcess error=2, The system cannot find the file specified 00:00:04.521 Check if ssh-agent is installed and in PATH 00:00:04.521 [ssh-agent] FATAL: Could not find a suitable ssh-agent provider 00:00:04.521 [ssh-agent] Diagnostic report

          aab added a comment -

          I uninstalled the openssh client (some 8.1 version) and rebooted the jenkins slave (windows 10) then the machine started using the openssh-client from /usr/bin/ssh (MSYS_NT-10.0-WOW) it's version is OpenSSH_7.6p1, OpenSSL 1.0.2n  7 Dec 2017, now if I type the ssh-agent I get the following info

          PS C:\Users\jenkins> ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-RbgKCjtDtgXm/agent.7016; export SSH_AUTH_SOCK;
SSH_AGENT_PID=1872; export SSH_AGENT_PID;
echo Agent pid 1872;
PS C:\Users\jenkins>

          With the version installed earlier, I used to get nothing when I entered command ssh-agent.

           

           

          aab added a comment - I uninstalled the openssh client (some 8.1 version) and rebooted the jenkins slave (windows 10) then the machine started using the openssh-client from /usr/bin/ssh (MSYS_NT-10.0-WOW) it's version is OpenSSH_7.6p1, OpenSSL 1.0.2n  7 Dec 2017, now if I type the ssh-agent I get the following info PS C:\Users\jenkins> ssh-agent SSH_AUTH_SOCK=/tmp/ssh-RbgKCjtDtgXm/agent.7016; export SSH_AUTH_SOCK; SSH_AGENT_PID=1872; export SSH_AGENT_PID; echo Agent pid 1872; PS C:\Users\jenkins> With the version installed earlier, I used to get nothing when I entered command ssh-agent.    

          Julie Heard added a comment - - edited

          I have just encountered the same issue and wanted to confirm a fix that worked for me (this is the fix listed above in the ticket):

          • git for windows includes an ssh-agent.exe which windows needs to know where it is for jenkins ssh-agent to be able to pick it up. If this is not in the PATH environment variables, you see the error listed in this ticket.
          • To fix this you could try adding %ProgramFiles%\Git\usr\bin\ssh-agent.exe (yours might be somewhere else) to your PATH environment variable
          • An alternative to try, in a command window run echo %PATH% copy the output, add %ProgramFiles%\Git\usr\bin\ssh-agent.exe at the beginning and then the a SET "PATH=%ProgramFiles%\Git\usr\bin\ssh-agent.exe;... with the rest of the PATH variable you copied 

          The ticket does say ‘this is likely to cause other problems in Windows’ so proceed with caution.

          Julie Heard added a comment - - edited I have just encountered the same issue and wanted to confirm a fix that worked for me (this is the fix listed above in the ticket) : git for windows includes an ssh-agent.exe which windows needs to know where it is for jenkins ssh-agent to be able to pick it up. If this is not in the PATH environment variables, you see the error listed in this ticket. To fix this you could try adding %ProgramFiles%\Git\usr\bin\ssh-agent.exe (yours might be somewhere else) to your PATH environment variable An alternative to try, in a command window run echo %PATH% copy the output, add %ProgramFiles%\Git\usr\bin\ssh-agent.exe at the beginning and then the a SET "PATH=%ProgramFiles%\Git\usr\bin\ssh-agent.exe;... with the rest of the PATH variable you copied  The ticket does say ‘this is likely to cause other problems in Windows’ so proceed with caution.

            Unassigned Unassigned
            nre_ableton Nik Reiman
            Votes:
            8 Vote for this issue
            Watchers:
            18 Start watching this issue

              Created:
              Updated: