• Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • ssh-credentials-plugin
    • None
    • Jenkins ver. 2.138.1

      After I installed the latest LTS I cannot connect to the Linux slave/agent anymore. The error is:

      SSHLauncher{host='[OMISSIS]', port=22, credentialsId='jenkins-cred', jvmOptions='', javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.KnownHostsFileKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}SSHLauncher{host='[OMISSIS]', port=22, credentialsId='jenkins-cred', jvmOptions='', javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.KnownHostsFileKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}[10/09/18 10:14:36] [SSH] Opening SSH connection to [OMISSIS]:22.[10/09/18 10:14:36] [SSH] SSH host key matches key in Known Hosts file. Connection will be allowed.ERROR: Server rejected the 1 private key(s) for jenkins (credentialId:jenkins-cred/method:publickey)ERROR: Failed to authenticate as jenkins with credential=jenkins-credjava.io.IOException: Publickey authentication failed. at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:291) at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:483) at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.doAuthenticate(TrileadSSHPublicKeyAuthenticator.java:109) at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:436) at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:455) at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1214) at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:844) at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:831) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748)Caused by: java.io.IOException: PEM problem: it is of unknown type at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:500) at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:225) ... 11 more[10/09/18 10:14:36] [SSH] Authentication failed.Authentication failed.[10/09/18 10:14:36] Launch failed - cleaning up connection[10/09/18 10:14:36] [SSH] Connection closed.
      

      As a workaround, I had to re-enter the private key in https://$HOST/credentials/store/system/domain/_/credential/jenkins-cred/update using the format:

      -----BEGIN RSA PRIVATE KEY-----
       […]
      -----END RSA PRIVATE KEY-----

      instead of the previous one (I don't know how I entered it the first time years ago):

      {[…]}

          [JENKINS-53954] Cannot connect to slave/agent Linux anymore

          MICHAEL LUPO added a comment -

          This is plaguing a large jenkins install in my company. This needs to be fixed PRONTO!

           

          MICHAEL LUPO added a comment - This is plaguing a large jenkins install in my company. This needs to be fixed PRONTO!  

          Mark Waite added a comment - - edited

          mikelupo the bug report describes a work around which can be applied immediately and requires no change to any software anywhere. Have you tried that work around?

          My interpretation of this report is that it is reporting that previously the private key credential field would accept an invalid format and then would assume that the invalid format was an RSA private key. The private key credential field no longer accepts that invalid format. It now requires a valid format for the private key.

          My understanding of the standard definition of a private key definition is that it must include the prologue

          -----BEGIN RSA PRIVATE KEY-----
          

          and the epilogue

          -----END RSA PRIVATE KEY-----
          

          That information defines the format of the private key. When I use an ed25519 private key, the prologue is different. When I use a dsa private key, the prologue is different as well.

          Mark Waite added a comment - - edited mikelupo the bug report describes a work around which can be applied immediately and requires no change to any software anywhere. Have you tried that work around? My interpretation of this report is that it is reporting that previously the private key credential field would accept an invalid format and then would assume that the invalid format was an RSA private key. The private key credential field no longer accepts that invalid format. It now requires a valid format for the private key. My understanding of the standard definition of a private key definition is that it must include the prologue -----BEGIN RSA PRIVATE KEY----- and the epilogue -----END RSA PRIVATE KEY----- That information defines the format of the private key. When I use an ed25519 private key, the prologue is different. When I use a dsa private key, the prologue is different as well.

          MICHAEL LUPO added a comment -

          Thanks for your replies.

          I’ve been using these keys with these slaves for over three years. All of a sudden this problem surfaces after updating jenkins.

          Funny thing is that it only complains on the slaves with the user as jenkins_1.
          Same Mac-mini hosts multiple slaves. jenkins_n.
          It’s the same on all my six Mac-minis. Only the jenkins_1 is affected. I reimported the key last week with a unique description (so I can pick it with confidence) and the problem went away until I restarted the jenkins master.

          All master and slaves are running Oracle java 1.8.xx

          Thanks,
          Mike

          MICHAEL LUPO added a comment - Thanks for your replies. I’ve been using these keys with these slaves for over three years. All of a sudden this problem surfaces after updating jenkins. Funny thing is that it only complains on the slaves with the user as jenkins_1. Same Mac-mini hosts multiple slaves. jenkins_n. It’s the same on all my six Mac-minis. Only the jenkins_1 is affected. I reimported the key last week with a unique description (so I can pick it with confidence) and the problem went away until I restarted the jenkins master. All master and slaves are running Oracle java 1.8.xx Thanks, Mike

          Mark Waite added a comment -

          I didn't understand from your description if the private key which was installed earlier included the expected prologue or not.

          If it did not include the expected prologue, then that means your installation was relying on a bug in the handling of private keys. Incorrectly formatted private keys appear to have been treated by default as though they were RSA private keys. That bug appears to have been fixed in a recent update.

          If it did include the expected prologue, then that is a different bug than is reported in this case.

          Mark Waite added a comment - I didn't understand from your description if the private key which was installed earlier included the expected prologue or not. If it did not include the expected prologue, then that means your installation was relying on a bug in the handling of private keys. Incorrectly formatted private keys appear to have been treated by default as though they were RSA private keys. That bug appears to have been fixed in a recent update. If it did include the expected prologue, then that is a different bug than is reported in this case.

          MICHAEL LUPO added a comment -

          We just noticed yet another anomaly. When the Jenkins server is restarted, any DEFAULT password parameters set in jobs are getting clobbered.

          As a workaround, we have to reset the default password parameter in the job's config, then re-save it. It will work till the Jenkins server is restarted again. Then we have to rinse-repeat. 

          markewaite where do SSH keys & passwords get stored? I would think they are from different plugins, but if their storage is similar, then this might be a hint at the root cause.

           

           

          MICHAEL LUPO added a comment - We just noticed yet another anomaly. When the Jenkins server is restarted, any DEFAULT password parameters set in jobs are getting clobbered. As a workaround, we have to reset the default password parameter in the job's config, then re-save it. It will work till the Jenkins server is restarted again. Then we have to rinse-repeat.  markewaite where do SSH keys & passwords get stored? I would think they are from different plugins, but if their storage is similar, then this might be a hint at the root cause.    

          Mark Waite added a comment - - edited

          mikelupo I don't understand from your description if the private key which was installed earlier included the expected prologue (and epilogue) or not. Did it include the expected beginning string and ending string?

          -----BEGIN RSA PRIVATE KEY-----
          

          If not, then once that prologue is included, does the private key persist across Jenkins server restarts?

          If the prologue string is included and the agent is unable to connect after server restart, then we'll need much more detail to duplicate the problem. I restart my Jenkins server frequently and it uses private keys to connect to agents with no issue.

          Please describe further in a new bug report what you mean when you say:

          When the Jenkins server is restarted, any DEFAULT password parameters set in jobs are getting clobbered.

          Does that mean that a job which took a password as a parameter and had a default assigned to that parameter will be run without using the default? If so, please provide a numbered step by step description of the process you use to define that job type so that others can duplicate the behavior.

          The private key for an ssh connection to an agent is stored by Jenkins itself or by the Jenkins credentials plugin. I'm confident the same basic storage mechanism is used for both secret parameters and private keys.

          Mark Waite added a comment - - edited mikelupo I don't understand from your description if the private key which was installed earlier included the expected prologue (and epilogue) or not. Did it include the expected beginning string and ending string? -----BEGIN RSA PRIVATE KEY----- If not, then once that prologue is included, does the private key persist across Jenkins server restarts? If the prologue string is included and the agent is unable to connect after server restart, then we'll need much more detail to duplicate the problem. I restart my Jenkins server frequently and it uses private keys to connect to agents with no issue. Please describe further in a new bug report what you mean when you say: When the Jenkins server is restarted, any DEFAULT password parameters set in jobs are getting clobbered. Does that mean that a job which took a password as a parameter and had a default assigned to that parameter will be run without using the default? If so, please provide a numbered step by step description of the process you use to define that job type so that others can duplicate the behavior. The private key for an ssh connection to an agent is stored by Jenkins itself or by the Jenkins credentials plugin. I'm confident the same basic storage mechanism is used for both secret parameters and private keys.

          MICHAEL LUPO added a comment -

          markewaite, the initial import was over three years ago, so I can't comment on that. But the second time I imported that key (last week) I definitely imported the key correctly with the entire prolog. It worked fine on all jenkins_1 users until the server restart.

          It sounds like I might be hitting a different defect. How can I help dig in to find the root cause?

           

          MICHAEL LUPO added a comment - markewaite , the initial import was over three years ago, so I can't comment on that. But the second time I imported that key (last week) I definitely imported the key correctly with the entire prolog. It worked fine on all jenkins_1 users until the server restart. It sounds like I might be hitting a different defect. How can I help dig in to find the root cause?  

          Mark Waite added a comment - - edited

          The best path is to submit a separate bug report that describes the precise conditions of the problem you're seeing. There are so many Jenkins servers in the world and so many agents in the world that I expect the case you're seeing will have some specific details which make it more uncommon than typical.

          Some of the attributes that might make your environment different could include:

          • operating system of the Jenkins master
          • operating system of the agent that has the issue
          • Jenkins master version
          • Java version on Jenkins master and Jenkins agent
          • Private key type (ed25519, rsa, ecdsa, dsa, ...)
          • Is FIPS mode enabled on any of the machines involved in the environment
          • Locale of the master and the agent
          • Is the private key protected by a passphrase?
          • Does the passphrase include shell special characters?
          • Plugin versions in your installation (Jenkins support pllugin can generate a bundle that includes this info)

          Mark Waite added a comment - - edited The best path is to submit a separate bug report that describes the precise conditions of the problem you're seeing. There are so many Jenkins servers in the world and so many agents in the world that I expect the case you're seeing will have some specific details which make it more uncommon than typical. Some of the attributes that might make your environment different could include: operating system of the Jenkins master operating system of the agent that has the issue Jenkins master version Java version on Jenkins master and Jenkins agent Private key type (ed25519, rsa, ecdsa, dsa, ...) Is FIPS mode enabled on any of the machines involved in the environment Locale of the master and the agent Is the private key protected by a passphrase? Does the passphrase include shell special characters? Plugin versions in your installation (Jenkins support pllugin can generate a bundle that includes this info)

          Matt Sicker added a comment -

          Were you using the "uploaded file on Jenkins master" for storing your key? That option was removed a while ago due to security issues.

          Matt Sicker added a comment - Were you using the "uploaded file on Jenkins master" for storing your key? That option was removed a while ago due to security issues.

            Unassigned Unassigned
            jhack Giacomo Boccardo
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: