Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50181

ssh-agent/ssh-credentials-plugin failing because ssh-add expects a newline in the keyfile

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Minor
    • Resolution: Fixed
    • None
    • ssh-credentials-1.17.1

    Description

      Repro:

      • Add Credentials
          - set Kind to "SSH Username with private key"
          - tick "enter directly"
          - paste a password-less private key without a trailing newline
      • Attempt to use credentials (I used ssh-agent from a Jenkinsfile)
      • Observe that ssh-add will prompt for a passphrase in the logs and the ssh-add has failed.

      The relevant part of my logs looked like this:

      ```
      [Pipeline] sshagent
      [ssh-agent] Using credentials jenkins (Github SSH key)
      [ssh-agent] Looking for ssh-agent implementation...
      [ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine)
      $ ssh-agent
      SSH_AUTH_SOCK=/tmp/ssh-rEGjLSRTHULl/agent.3927
      SSH_AGENT_PID=3929
      [ssh-agent] started an agent
      $ ssh-add /var/lib/jenkins/workspace/job@tmp/private_key_2980200938951827942.key
      Enter passphrase for /var/lib/jenkins/workspace/job@tmp/private_key_2980200938951827942.key: [Pipeline] // sshagent
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // withEnv
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      ERROR: Failed to run ssh-add
      Finished: FAILURE

      ```

      Adding the trailing newline to input in the web-ui resolves this issue. Adding multiple newlines didn't seem have any adverse effect so Jenkins should probably just add a newline when it writes the keyfile.

      Attachments

        Issue Links

          Activity

            liath John Jones created issue -
            dnusbaum Devin Nusbaum added a comment - - edited

            Thanks for reporting the issue!

            Adding multiple newlines didn't seem have any adverse effect so Jenkins should probably just add a newline when it writes the keyfile.

            Sounds reasonable to me, although probably a newline should only be added if there isn't one already so that resaving the credentials doesn't keep adding newlines. Feel free to submit a pull request to the repository (ideally with a regression test); here is the class that I think would need to be modified.

            dnusbaum Devin Nusbaum added a comment - - edited Thanks for reporting the issue! Adding multiple newlines didn't seem have any adverse effect so Jenkins should probably just add a newline when it writes the keyfile. Sounds reasonable to me, although probably a newline should only be added if there isn't one already so that resaving the credentials doesn't keep adding newlines. Feel free to submit a pull request to the repository (ideally with a regression test); here is the class that I think would need to be modified.
            liath John Jones added a comment - PR created here:  https://github.com/jenkinsci/ssh-credentials-plugin/pull/33
            eltusha Ellen Tushar added a comment -

            I've tried the adding a new line after the private key to no avail.  I've also tried adding a few lines and a # sign on one line.  I still get the ssh-add error about the passphrase. 

            Jenkins 2.164.3, SSH-agent 1.17 SSH-credentials 1.16

            After downgrading these plugins, I'm able to use the credentials with the trailing new line.

            SSH-agent 1.13  SSH-credentials 1.12

            Has anyone been able to use the trailing new line trick with these plugin versions?  SSH-agent 1.17 SSH-credentials 1.16

            eltusha Ellen Tushar added a comment - I've tried the adding a new line after the private key to no avail.  I've also tried adding a few lines and a # sign on one line.  I still get the ssh-add error about the passphrase.  Jenkins 2.164.3, SSH-agent 1.17 SSH-credentials 1.16 After downgrading these plugins, I'm able to use the credentials with the trailing new line. SSH-agent 1.13  SSH-credentials 1.12 Has anyone been able to use the trailing new line trick with these plugin versions?  SSH-agent 1.17 SSH-credentials 1.16
            liath John Jones made changes -
            Field Original Value New Value
            Description Repro:
            - Add Credentials
              - set Kind to "SSH Username with private key"
              - tick "enter directly"
              - paste a password-less private key without a trailing newline
            - Attempt to use credentials (I used ssg-agent from a Jenkinsfile)
            - Observe that ssh-add will prompt for a passphrase in the logs and the ssh-add has failed.

            The relevant part of my logs looked like this:

            ```
            [Pipeline] sshagent
            [ssh-agent] Using credentials jenkins (Github SSH key)
            [ssh-agent] Looking for ssh-agent implementation...
            [ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine)
            $ ssh-agent
            SSH_AUTH_SOCK=/tmp/ssh-rEGjLSRTHULl/agent.3927
            SSH_AGENT_PID=3929
            [ssh-agent] started an agent
            $ ssh-add /var/lib/jenkins/workspace/job@tmp/private_key_2980200938951827942.key
            Enter passphrase for /var/lib/jenkins/workspace/job@tmp/private_key_2980200938951827942.key: [Pipeline] // sshagent
            [Pipeline] }
            [Pipeline] // stage
            [Pipeline] }
            [Pipeline] // withEnv
            [Pipeline] }
            [Pipeline] // node
            [Pipeline] End of Pipeline
            ERROR: Failed to run ssh-add
            Finished: FAILURE

            ```

            Adding the trailing newline to input in the web-ui resolves this issue. Adding multiple newlines didn't seem have any adverse effect so Jenkins should probably just add a newline when it writes the keyfile.
            Repro:
             - Add Credentials
               - set Kind to "SSH Username with private key"
               - tick "enter directly"
               - paste a password-less private key without a trailing newline
             - Attempt to use credentials (I used ssh-agent from a Jenkinsfile)
             - Observe that ssh-add will prompt for a passphrase in the logs and the ssh-add has failed.

            The relevant part of my logs looked like this:

            ```
             [Pipeline] sshagent
             [ssh-agent] Using credentials jenkins (Github SSH key)
             [ssh-agent] Looking for ssh-agent implementation...
             [ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine)
             $ ssh-agent
             SSH_AUTH_SOCK=/tmp/ssh-rEGjLSRTHULl/agent.3927
             SSH_AGENT_PID=3929
             [ssh-agent] started an agent
             $ ssh-add /var/lib/jenkins/workspace/job@tmp/private_key_2980200938951827942.key
             Enter passphrase for /var/lib/jenkins/workspace/job@tmp/private_key_2980200938951827942.key: [Pipeline] // sshagent
             [Pipeline] }
             [Pipeline] // stage
             [Pipeline] }
             [Pipeline] // withEnv
             [Pipeline] }
             [Pipeline] // node
             [Pipeline] End of Pipeline
             ERROR: Failed to run ssh-add
             Finished: FAILURE

            ```

            Adding the trailing newline to input in the web-ui resolves this issue. Adding multiple newlines didn't seem have any adverse effect so Jenkins should probably just add a newline when it writes the keyfile.
            jvz Matt Sicker added a comment - Fixed in PR https://github.com/jenkinsci/ssh-credentials-plugin/pull/33 which was just merged.
            jvz Matt Sicker made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Fixed but Unreleased [ 10203 ]
            jvz Matt Sicker added a comment -

            Released in 1.17.1.

            jvz Matt Sicker added a comment - Released in 1.17.1.
            jvz Matt Sicker made changes -
            Released As ssh-credentials-1.17.1
            Status Fixed but Unreleased [ 10203 ] Resolved [ 5 ]

            jvz I am still experiencing this issue with ssh-credentials-plugin 1.17.2 / Jenkins 2.176.2, can this be re-opened?

            When I paste an SSH key without a password and without a newline after ----END OPENSSH PRIVATE KEY----, I am getting this:

            [EnvInject] - Loading node environment variables.
            Building in workspace /opt/jenkins/workspace/tests/testssh
            [ssh-agent] Looking for ssh-agent implementation...
            [ssh-agent]   Exec ssh-agent (binary ssh-agent on a remote machine)
            $ ssh-agent
            SSH_AUTH_SOCK=/tmp/ssh-LjVbJNMcagCy/agent.130927
            SSH_AGENT_PID=130929
            [ssh-agent] Started.
            Running ssh-add (command line suppressed)
            Enter passphrase for /opt/jenkins/workspace/tests/testssh@tmp/private_key_4661922093191141579.key: ERROR: Failed to run ssh-add
            Finished: FAILURE 

            When I update the key and put a newline, it works fine:

            Building in workspace /opt/jenkins/workspace/tests/testssh
            [ssh-agent] Looking for ssh-agent implementation...
            [ssh-agent]   Exec ssh-agent (binary ssh-agent on a remote machine)
            $ ssh-agent
            SSH_AUTH_SOCK=/tmp/ssh-LHBjsdVD6N7X/agent.130956
            SSH_AGENT_PID=130958
            [ssh-agent] Started.
            Running ssh-add (command line suppressed)
            Identity added: /opt/jenkins/workspace/tests/testssh@tmp/private_key_1585458568929474760.key (SSH CI key)
            [ssh-agent] Using credentials jenkins (SSH key used for Tomcat restarts)
            [testssh] $ /bin/sh -xe /tmp/jenkins2172743275862346783.sh
            + hostname
            ci
            $ ssh-agent -k
            unset SSH_AUTH_SOCK;
            unset SSH_AGENT_PID;
            echo Agent pid 130958 killed;
            [ssh-agent] Stopped.
            Finished: SUCCESS
            
            warden Radek Antoniuk added a comment - jvz I am still experiencing this issue with ssh-credentials-plugin 1.17.2 / Jenkins 2.176.2, can this be re-opened? When I paste an SSH key without a password and without a newline after ---- END OPENSSH PRIVATE KEY ---- , I am getting this: [EnvInject] - Loading node environment variables. Building in workspace /opt/jenkins/workspace/tests/testssh [ssh-agent] Looking for ssh-agent implementation... [ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine) $ ssh-agent SSH_AUTH_SOCK=/tmp/ssh-LjVbJNMcagCy/agent.130927 SSH_AGENT_PID=130929 [ssh-agent] Started. Running ssh-add (command line suppressed) Enter passphrase for /opt/jenkins/workspace/tests/testssh@tmp/private_key_4661922093191141579.key: ERROR: Failed to run ssh-add Finished: FAILURE When I update the key and put a newline, it works fine: Building in workspace /opt/jenkins/workspace/tests/testssh [ssh-agent] Looking for ssh-agent implementation... [ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine) $ ssh-agent SSH_AUTH_SOCK=/tmp/ssh-LHBjsdVD6N7X/agent.130956 SSH_AGENT_PID=130958 [ssh-agent] Started. Running ssh-add (command line suppressed) Identity added: /opt/jenkins/workspace/tests/testssh@tmp/private_key_1585458568929474760.key (SSH CI key) [ssh-agent] Using credentials jenkins (SSH key used for Tomcat restarts) [testssh] $ /bin/sh -xe /tmp/jenkins2172743275862346783.sh + hostname ci $ ssh-agent -k unset SSH_AUTH_SOCK; unset SSH_AGENT_PID; echo Agent pid 130958 killed; [ssh-agent] Stopped. Finished: SUCCESS
            jvz Matt Sicker added a comment -

            Is that a separate issue? And can you reproduce this with a unit test? See the original PR for example.

            jvz Matt Sicker added a comment - Is that a separate issue? And can you reproduce this with a unit test? See the original PR for example.
            warden Radek Antoniuk added a comment - - edited

            OK, I debugged it a little and I think I know what the problem is:
            even though I modified the ssh-credentials plugin to save with a newline, it still didn't work for me.
            It turns out that the ssh-agent plugin that writes the private key to the file, strips it out of the newline. Then... ssh-add asks for a password. When I add a newline, it adds it to the agent fine. The question is now, what is stripping this newline... I'll continue to debug but any hints appreciated.

            warden Radek Antoniuk added a comment - - edited OK, I debugged it a little and I think I know what the problem is: even though I modified the ssh-credentials plugin to save with a newline, it still didn't work for me. It turns out that the ssh-agent plugin that writes the private key to the file, strips it out of the newline. Then... ssh-add asks for a password. When I add a newline, it adds it to the agent fine. The question is now, what is stripping this newline... I'll continue to debug but any hints appreciated.
            terracota Terracota Pz added a comment -

            I'm having the same issue as initially described (it seems ssh-add prompts for passphrase and fails), but workaround of adding newline in credentials' key does not solve it.

            Versions:

            • Jenkins 2.190.3
            • "ssh agent" plugin 1.17
            • "ssh credentials" plugin 1.18

            Logs:
            [ssh-agent] Using credentials testcreds
            [ssh-agent] Looking for ssh-agent implementation...
            [ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine)
            $ ssh-agent
            SSH_AUTH_SOCK=/tmp/ssh-HmEucFexUrOg/agent.19128
            SSH_AGENT_PID=19129
            Running ssh-add (command line suppressed)Enter passphrase for /opt/jenkins/workspace/roduct-service-read-build_master@tmp/private_key_1675109103121607963.key: [Pipeline] // sshagent[Pipeline] }[Pipeline] // stage

            terracota Terracota Pz added a comment - I'm having the same issue as initially described (it seems ssh-add prompts for passphrase and fails), but workaround of adding newline in credentials' key does not solve it. Versions: Jenkins 2.190.3 "ssh agent" plugin 1.17 "ssh credentials" plugin 1.18 Logs: [ssh-agent] Using credentials testcreds [ssh-agent] Looking for ssh-agent implementation... [ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine) $ ssh-agent SSH_AUTH_SOCK=/tmp/ssh-HmEucFexUrOg/agent.19128 SSH_AGENT_PID=19129 Running ssh-add (command line suppressed)Enter passphrase for /opt/jenkins/workspace/roduct-service-read-build_master@tmp/private_key_1675109103121607963.key: [Pipeline] // sshagent [Pipeline] } [Pipeline] // stage
            terracota Terracota Pz added a comment -

            UPDATE

            I'm not having the issue anymore, it was my fault:

            terracota Terracota Pz added a comment - UPDATE I'm not having the issue anymore, it was my fault: I was using github token as the private key when I created the "SSH Username with private key" credential. Instead, an RSA private key should be used (eg: as per https://help.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh ). With this, it works fine.
            dnusbaum Devin Nusbaum made changes -
            Remote Link This issue links to "jenkinsci/ssh-credentials-plugin#46 (Web Link)" [ 25009 ]

            After upgrading to ssh-agent 1.21 we had a job hang at a git command.

            We terminated it after more than 30 minutes. Tried again and again with same result.

            After reverting to 1.20 it worked again.

            Versions:

            • Jenkins 2.280
            • ssh-agent 1.21
            • ssh-credentials 1.18.1

            Log:

             

            ...
            [Pipeline] sshagent
            15:27:55  [ssh-agent] Using credentials jenkins (jenkins ssh key)
            15:27:55  [ssh-agent] Looking for ssh-agent implementation...
            15:27:55  [ssh-agent]   Exec ssh-agent (binary ssh-agent on a remote machine)
            15:27:55  $ docker exec bb2595d46378e3aa4279cba0a20d8e3026f79d84bc9d2f2b5b856bbfba3ab1f5 ssh-agent
            15:27:55  [ssh-agent]   Java/JNR ssh-agent
            15:27:55  [ssh-agent] Registered BouncyCastle on the remote agent
            15:27:55  [ssh-agent] Started.
            [Pipeline] {
            [Pipeline] sh
            15:27:55  + git ls-remote --heads origin
            

             

             

            snago Robin Karlsson added a comment - After upgrading to ssh-agent 1.21 we had a job hang at a git command. We terminated it after more than 30 minutes. Tried again and again with same result. After reverting to 1.20 it worked again. Versions: Jenkins 2.280 ssh-agent 1.21 ssh-credentials 1.18.1 Log:   ... [Pipeline] sshagent 15:27:55 [ssh-agent] Using credentials jenkins (jenkins ssh key) 15:27:55 [ssh-agent] Looking for ssh-agent implementation... 15:27:55 [ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine) 15:27:55 $ docker exec bb2595d46378e3aa4279cba0a20d8e3026f79d84bc9d2f2b5b856bbfba3ab1f5 ssh-agent 15:27:55 [ssh-agent] Java/JNR ssh-agent 15:27:55 [ssh-agent] Registered BouncyCastle on the remote agent 15:27:55 [ssh-agent] Started. [Pipeline] { [Pipeline] sh 15:27:55 + git ls-remote --heads origin    
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-43564 [ JENKINS-43564 ]

            People

              dnusbaum Devin Nusbaum
              liath John Jones
              Votes:
              4 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: