Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60714

Cannot create jenkins ssh username with private key credential via rest xml api

      Basically, I am trying to create a credential on jenkins via Rest API. Using xml data below:

      ```
      <?xml version='1.0' encoding='UTF-8'?>
      <com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey>
      <scope>GLOBAL</scope>
      <id>jenkins-github-ssh</id>
      <description>jenkins-github-ssh</description>
      <username>username</username>
      <directEntryPrivateKeySource>
      <privateKey>----BEGIN OPENSSH PRIVATE KEY----
      *****************************************
      ----END OPENSSH PRIVATE KEY----</privateKey>
      </directEntryPrivateKeySource>
      </com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey>
      ```

      I can see the [credential][1] after calling REST post request. But when I use this credential for a GitHub repository, Jenkins says:

      >Failed to connect to repository : Command "git ls-remote -h – git@github.com:***.git HEAD" returned status code 128:
      stdout:
      stderr: Load key "/tmp/ssh3978703187838467164.key": invalid format
      git@github.com: Permission denied (publickey).
      fatal: Could not read from remote repository.

      But If I update the credential which is created by rest api with same private key above manually. It works. Somehow key is broken while posting, which seems to be a bug.

       

      Do you guys have any idea to point me to a workaround?

      Jenkins 2.198 & SSH Credentials Plugin 1.17.3

      Thanks

      [1]: https://i.stack.imgur.com/j7Ncg.png

          [JENKINS-60714] Cannot create jenkins ssh username with private key credential via rest xml api

          Matt Sicker added a comment -

          Is that a PKCS#12 private key? If so, that's a known limitation. To work around that, you can convert it to either the old PEM format or to the OpenSSH format.

          Matt Sicker added a comment - Is that a PKCS#12 private key? If so, that's a known limitation. To work around that, you can convert it to either the old PEM format or to the OpenSSH format.

          Sam K added a comment -

          I am facing the same issue and I am using pem file. Has anyone found any work around for this ?

          Sam K added a comment - I am facing the same issue and I am using pem file. Has anyone found any work around for this ?

          Matt Sicker added a comment -

          saimak: can you try adding a newline at the end of your public key in the request body? Also, what key file format are you using? The old PEM format, the new OpenSSH format, or that PKCS12 format (more portable than OpenSSH but not supported in this plugin).

          Matt Sicker added a comment - saimak : can you try adding a newline at the end of your public key in the request body? Also, what key file format are you using? The old PEM format, the new OpenSSH format, or that PKCS12 format (more portable than OpenSSH but not supported in this plugin).

          Ramachandran added a comment -

          I happened to bump in to this ticket when I search for resolving a similar issue. I am posting how I resolved the issue here, so it might help someone. I have created a test pipeline job to view the working key (which was added manually) and error key (which was added via REST call) and observed that there was some extra spaces in the beginning of my error key. I have tweaked the xml input in the REST api code and resolved the issue.

           

          pipeline {
            agent any  
            
            environment {
              ERROR_KEY = credentials('error-key-id')
              WORKING_KEY = credentials('working-key-id')
            }
            
             stages {
                stage('Test') {
                   steps {
                      sh '''
                          set +x
                          echo "-------------------------------------------"
                          cat $ERROR_KEY
                          echo "-------------------------------------------"
                          cat $WORKING_KEY
                          echo "-------------------------------------------"
                      '''
                   }
                }
             }
          }
          
          

           

           

          Ramachandran added a comment - I happened to bump in to this ticket when I search for resolving a similar issue. I am posting how I resolved the issue here, so it might help someone. I have created a test pipeline job to view the working key (which was added manually) and error key (which was added via REST call) and observed that there was some extra spaces in the beginning of my error key. I have tweaked the xml input in the REST api code and resolved the issue.   pipeline { agent any environment { ERROR_KEY = credentials( 'error-key-id' ) WORKING_KEY = credentials( 'working-key-id' ) } stages { stage( 'Test' ) { steps { sh ''' set +x echo "-------------------------------------------" cat $ERROR_KEY echo "-------------------------------------------" cat $WORKING_KEY echo "-------------------------------------------" ''' } } } }    

          Allan BURDAJEWICZ added a comment - - edited

          While helping a user recently, I discovered that the persisted private key does not contains the CRNL (newlines). The decrypted value is a one line string. This seems to happen since the release of 2.277.1 and JEP 228 XStream Update.

          A workaround / solution for this would be to use the carriage return character in the XML to ensure that it is preserved:

          	<privateKey>-----BEGIN PRIVATE KEY-----&#xd;
          .............&#xd;
          .............&#xd;
          [...]
          -----END PRIVATE KEY-----&#xd;
                   </privateKey>
          

          samthebest saimak Would that solve your problem ?

          Allan BURDAJEWICZ added a comment - - edited While helping a user recently, I discovered that the persisted private key does not contains the CRNL (newlines). The decrypted value is a one line string. This seems to happen since the release of 2.277.1 and JEP 228 XStream Update . A workaround / solution for this would be to use the carriage return character in the XML to ensure that it is preserved: <privateKey>-----BEGIN PRIVATE KEY-----&#xd; .............&#xd; .............&#xd; [...] -----END PRIVATE KEY-----&#xd; </privateKey> samthebest saimak Would that solve your problem ?

          Jesse Glick added a comment -

          If so, the problem is in github-branch-source, not credentials.

          Jesse Glick added a comment - If so, the problem is in github-branch-source , not credentials .

          Ray Kivisto added a comment -

          Based on my testing, a SSH keypair being imported using:

          curl -X POST -u USER:API_TOKEN -H 'content-type:application/xml' -d @credential.xml "JENKINS_URL/credentials/store/system/domain/_/createCredentials" 

          currently gets stripped of all newlines, as Allan mentioned above, this seems to be the real problem based on:

          1. I tried using this credential to connect a ssh agent using SSH Build Agents plugin and this failed:

          Caused by: java.io.IOException: PEM problem: it is of unknown type. Supported algorithms are :[ssh-ed25519, ecdsa-sha2-nistp521, ecdsa-sha2-nistp384, ecdsa-sha2-nistp256, rsa-sha2-256, rsa-sha2-512, ssh-rsa, ssh-dss] at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:482)

          Even though this keypair was generated with the command `ssh-keygen -t rsa -m PEM -C "Jenkins agent key" -f "jenkinsAgent_rsa"`, and if I manually update my credential using the web UI in Jenkins, using the private key without newlines, my build agent connects successfully.

          2. If I save this ssh credential to a file (saved-key) using a Pipeline job, then I try to manually use that to ssh to a machine that accepts that keypair, ssh even complains about the format:

          ssh -i saved-key $HOSTNAME
          Load key "saved-key": invalid format 

          If I use the original SSH key that ssh-keygen created (jenkinsAgent_rsa), it works;

          ssh -i jenkinsAgent_rsa $HOSTNAME
          Last login: Wed Jun  1 22:04:01 2022 from REDACTED
          [rkivisto@rkivisto-centos ~]$ 

          So I believe the change would be when the createCredentials endpoint is called, and the credential XML contains a "<privateKey>" element, everything in that element should be preserved with the newlines.

           

           

          Ray Kivisto added a comment - Based on my testing, a SSH keypair being imported using: curl -X POST -u USER:API_TOKEN -H 'content-type:application/xml' -d @credential.xml "JENKINS_URL/credentials/store/system/domain/_/createCredentials" currently gets stripped of all newlines, as Allan mentioned above, this seems to be the real problem based on: 1. I tried using this credential to connect a ssh agent using SSH Build Agents plugin and this failed: Caused by: java.io.IOException: PEM problem: it is of unknown type. Supported algorithms are :[ssh-ed25519, ecdsa-sha2-nistp521, ecdsa-sha2-nistp384, ecdsa-sha2-nistp256, rsa-sha2-256, rsa-sha2-512, ssh-rsa, ssh-dss] at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:482) Even though this keypair was generated with the command `ssh-keygen -t rsa -m PEM -C "Jenkins agent key" -f "jenkinsAgent_rsa"`, and if I manually update my credential using the web UI in Jenkins, using the private key without newlines, my build agent connects successfully. 2. If I save this ssh credential to a file (saved-key) using a Pipeline job, then I try to manually use that to ssh to a machine that accepts that keypair, ssh even complains about the format: ssh -i saved-key $HOSTNAME Load key "saved-key" : invalid format If I use the original SSH key that ssh-keygen created (jenkinsAgent_rsa), it works; ssh -i jenkinsAgent_rsa $HOSTNAME Last login: Wed Jun  1 22:04:01 2022 from REDACTED [rkivisto@rkivisto-centos ~]$ So I believe the change would be when the createCredentials endpoint is called, and the credential XML contains a "<privateKey>" element, everything in that element should be preserved with the newlines.    

          Jesse Glick added a comment -

          Did you try using a CDATA block?

          XML element text is not designed to perfectly preserve whitespace. For normal credentials usage this is no issue since the field is a Secret and so encrypted into another format.

          We could use SecretBytes in lieu of these multiline text fields, though that would be even more awkward for manual upload via REST since you would need to Base64-encode the file first.

          Or the credentials implementation could automatically convert a private key with missing whitespace into the requisite format, I guess using some pattern matching, splitting lines at a certain length, etc.

          Jesse Glick added a comment - Did you try using a CDATA block? XML element text is not designed to perfectly preserve whitespace. For normal credentials usage this is no issue since the field is a Secret and so encrypted into another format. We could use SecretBytes in lieu of these multiline text fields, though that would be even more awkward for manual upload via REST since you would need to Base64-encode the file first. Or the credentials implementation could automatically convert a private key with missing whitespace into the requisite format, I guess using some pattern matching, splitting lines at a certain length, etc.

          Ray Kivisto added a comment -

          Great idea about CDATA, I tried it using the following syntax:

           

          <com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey plugin="ssh-credentials@277.v95c2fec1c047">
            <scope>GLOBAL</scope>
            <id>6c35d44f-aead-49ca-a70b-a1cf540dea82</id>
            <description></description>
            <username>ray</username>
            <usernameSecret>false</usernameSecret>
            <privateKeySource class="com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$DirectEntryPrivateKeySource">
              <privateKey><![CDATA[
          -----BEGIN RSA PRIVATE KEY-----
          ...REDACTED...
          -----END RSA PRIVATE KEY-----
          ]]>
              </privateKey>
            </privateKeySource>
          </com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey> 

          unfortunately the newlines were still not preserved. Yes, those both sound like viable options, the second one being more easy to use for the end user.

          For now as a workaround, we can import SSH credentials between instances using the web UI, or maybe also using the credential migration scripts here https://github.com/cloudbees/jenkins-scripts/tree/master/credentials-migration

           

          Ray Kivisto added a comment - Great idea about CDATA, I tried it using the following syntax:   <com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey plugin= "ssh-credentials@277.v95c2fec1c047" >   <scope>GLOBAL</scope>   <id>6c35d44f-aead-49ca-a70b-a1cf540dea82</id>   <description></description>   <username>ray</username>   <usernameSecret> false </usernameSecret>   <privateKeySource class= "com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$DirectEntryPrivateKeySource" >     <privateKey><![CDATA[ -----BEGIN RSA PRIVATE KEY----- ...REDACTED... -----END RSA PRIVATE KEY----- ]]>     </privateKey>   </privateKeySource> </com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey> unfortunately the newlines were still not preserved. Yes, those both sound like viable options, the second one being more easy to use for the end user. For now as a workaround, we can import SSH credentials between instances using the web UI, or maybe also using the credential migration scripts here https://github.com/cloudbees/jenkins-scripts/tree/master/credentials-migration  

            Unassigned Unassigned
            samthebest Sam Savage
            Votes:
            1 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: