Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53954

Cannot connect to slave/agent Linux anymore

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      After I installed the latest LTS I cannot connect to the Linux slave/agent anymore. The error is:

      SSHLauncher{host='[OMISSIS]', port=22, credentialsId='jenkins-cred', jvmOptions='', javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.KnownHostsFileKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}SSHLauncher{host='[OMISSIS]', port=22, credentialsId='jenkins-cred', jvmOptions='', javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.KnownHostsFileKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}[10/09/18 10:14:36] [SSH] Opening SSH connection to [OMISSIS]:22.[10/09/18 10:14:36] [SSH] SSH host key matches key in Known Hosts file. Connection will be allowed.ERROR: Server rejected the 1 private key(s) for jenkins (credentialId:jenkins-cred/method:publickey)ERROR: Failed to authenticate as jenkins with credential=jenkins-credjava.io.IOException: Publickey authentication failed. at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:291) at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:483) at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.doAuthenticate(TrileadSSHPublicKeyAuthenticator.java:109) at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:436) at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:455) at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1214) at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:844) at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:831) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748)Caused by: java.io.IOException: PEM problem: it is of unknown type at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:500) at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:225) ... 11 more[10/09/18 10:14:36] [SSH] Authentication failed.Authentication failed.[10/09/18 10:14:36] Launch failed - cleaning up connection[10/09/18 10:14:36] [SSH] Connection closed.
      

      As a workaround, I had to re-enter the private key in https://$HOST/credentials/store/system/domain/_/credential/jenkins-cred/update using the format:

      -----BEGIN RSA PRIVATE KEY-----
       […]
      -----END RSA PRIVATE KEY-----

      instead of the previous one (I don't know how I entered it the first time years ago):

      {[…]}

        Attachments

          Activity

          Hide
          mikelupo MICHAEL LUPO added a comment -

          We just noticed yet another anomaly. When the Jenkins server is restarted, any DEFAULT password parameters set in jobs are getting clobbered.

          As a workaround, we have to reset the default password parameter in the job's config, then re-save it. It will work till the Jenkins server is restarted again. Then we have to rinse-repeat. 

          Mark Waite where do SSH keys & passwords get stored? I would think they are from different plugins, but if their storage is similar, then this might be a hint at the root cause.

           

           

          Show
          mikelupo MICHAEL LUPO added a comment - We just noticed yet another anomaly. When the Jenkins server is restarted, any DEFAULT password parameters set in jobs are getting clobbered. As a workaround, we have to reset the default password parameter in the job's config, then re-save it. It will work till the Jenkins server is restarted again. Then we have to rinse-repeat.  Mark Waite where do SSH keys & passwords get stored? I would think they are from different plugins, but if their storage is similar, then this might be a hint at the root cause.    
          Hide
          markewaite Mark Waite added a comment - - edited

          MICHAEL LUPO I don't understand from your description if the private key which was installed earlier included the expected prologue (and epilogue) or not. Did it include the expected beginning string and ending string?

          -----BEGIN RSA PRIVATE KEY-----
          

          If not, then once that prologue is included, does the private key persist across Jenkins server restarts?

          If the prologue string is included and the agent is unable to connect after server restart, then we'll need much more detail to duplicate the problem. I restart my Jenkins server frequently and it uses private keys to connect to agents with no issue.

          Please describe further in a new bug report what you mean when you say:

          When the Jenkins server is restarted, any DEFAULT password parameters set in jobs are getting clobbered.

          Does that mean that a job which took a password as a parameter and had a default assigned to that parameter will be run without using the default? If so, please provide a numbered step by step description of the process you use to define that job type so that others can duplicate the behavior.

          The private key for an ssh connection to an agent is stored by Jenkins itself or by the Jenkins credentials plugin. I'm confident the same basic storage mechanism is used for both secret parameters and private keys.

          Show
          markewaite Mark Waite added a comment - - edited MICHAEL LUPO I don't understand from your description if the private key which was installed earlier included the expected prologue (and epilogue) or not. Did it include the expected beginning string and ending string? -----BEGIN RSA PRIVATE KEY----- If not, then once that prologue is included, does the private key persist across Jenkins server restarts? If the prologue string is included and the agent is unable to connect after server restart, then we'll need much more detail to duplicate the problem. I restart my Jenkins server frequently and it uses private keys to connect to agents with no issue. Please describe further in a new bug report what you mean when you say: When the Jenkins server is restarted, any DEFAULT password parameters set in jobs are getting clobbered. Does that mean that a job which took a password as a parameter and had a default assigned to that parameter will be run without using the default? If so, please provide a numbered step by step description of the process you use to define that job type so that others can duplicate the behavior. The private key for an ssh connection to an agent is stored by Jenkins itself or by the Jenkins credentials plugin. I'm confident the same basic storage mechanism is used for both secret parameters and private keys.
          Hide
          mikelupo MICHAEL LUPO added a comment -

          Mark Waite, the initial import was over three years ago, so I can't comment on that. But the second time I imported that key (last week) I definitely imported the key correctly with the entire prolog. It worked fine on all jenkins_1 users until the server restart.

          It sounds like I might be hitting a different defect. How can I help dig in to find the root cause?

           

          Show
          mikelupo MICHAEL LUPO added a comment - Mark Waite , the initial import was over three years ago, so I can't comment on that. But the second time I imported that key (last week) I definitely imported the key correctly with the entire prolog. It worked fine on all jenkins_1 users until the server restart. It sounds like I might be hitting a different defect. How can I help dig in to find the root cause?  
          Hide
          markewaite Mark Waite added a comment - - edited

          The best path is to submit a separate bug report that describes the precise conditions of the problem you're seeing. There are so many Jenkins servers in the world and so many agents in the world that I expect the case you're seeing will have some specific details which make it more uncommon than typical.

          Some of the attributes that might make your environment different could include:

          • operating system of the Jenkins master
          • operating system of the agent that has the issue
          • Jenkins master version
          • Java version on Jenkins master and Jenkins agent
          • Private key type (ed25519, rsa, ecdsa, dsa, ...)
          • Is FIPS mode enabled on any of the machines involved in the environment
          • Locale of the master and the agent
          • Is the private key protected by a passphrase?
          • Does the passphrase include shell special characters?
          • Plugin versions in your installation (Jenkins support pllugin can generate a bundle that includes this info)
          Show
          markewaite Mark Waite added a comment - - edited The best path is to submit a separate bug report that describes the precise conditions of the problem you're seeing. There are so many Jenkins servers in the world and so many agents in the world that I expect the case you're seeing will have some specific details which make it more uncommon than typical. Some of the attributes that might make your environment different could include: operating system of the Jenkins master operating system of the agent that has the issue Jenkins master version Java version on Jenkins master and Jenkins agent Private key type (ed25519, rsa, ecdsa, dsa, ...) Is FIPS mode enabled on any of the machines involved in the environment Locale of the master and the agent Is the private key protected by a passphrase? Does the passphrase include shell special characters? Plugin versions in your installation (Jenkins support pllugin can generate a bundle that includes this info)
          Hide
          jvz Matt Sicker added a comment -

          Were you using the "uploaded file on Jenkins master" for storing your key? That option was removed a while ago due to security issues.

          Show
          jvz Matt Sicker added a comment - Were you using the "uploaded file on Jenkins master" for storing your key? That option was removed a while ago due to security issues.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            jhack Giacomo Boccardo
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated: