Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55971

SSH Host key matches, Authentication failed, Slaves failed to reconnect

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • core, ssh-credentials-plugin
    • None
    • Jenkins ver 2.163
      Linux
      Chrome

      SSHLauncher{host='myhost', port=22, credentialsId='aaa-bbb-ccc-myRealCredentials', jvmOptions='', javaPath='/home/bla/jenkins/bin/java', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.ManuallyTrustedKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
      [02/04/19 22:09:16] [SSH] Opening SSH connection to ligenr10.swb.siemens.de:22.
      [02/04/19 22:09:16] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed.
      [02/04/19 22:09:16] [SSH] Authentication failed.

      Manage old data complained on startup - ( didn't take real notice of message )

      Credentials, private key, was corrupted when starting Jenkins after updating to 2.163

      Eventually found and resolved.

      ( Jenkins- Credentials->System->Global Credentials -> name )

      ( updated Private Key - entered directly )

      Might not be a bug but user error. However, caused 8 hour loss of Jenkins until problem found.

       

          [JENKINS-55971] SSH Host key matches, Authentication failed, Slaves failed to reconnect

          Steve Graham created issue -
          Steve Graham made changes -
          Description Original: SSHLauncher\{host='myhost', port=22, credentialsId='aaa-bbb-ccc-myRealCredentials', jvmOptions='', javaPath='/home/bla/jenkins/bin/java', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.ManuallyTrustedKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
          [02/04/19 22:09:16] [SSH] Opening SSH connection to ligenr10.swb.siemens.de:22.
          [02/04/19 22:09:16] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed.
          [02/04/19 22:09:16] [SSH] {color:#FF0000}Authentication failed.{color}

          Manage old data complained on startup - ( didn't take real notice of message )

          Credentials, private key, was corrupted when starting Jenkins after updating to 2.163

          Eventually found and corrected.

                      		New: SSHLauncher\{host='myhost', port=22, credentialsId='aaa-bbb-ccc-myRealCredentials', jvmOptions='', javaPath='/home/bla/jenkins/bin/java', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.ManuallyTrustedKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
           [02/04/19 22:09:16] [SSH] Opening SSH connection to ligenr10.swb.siemens.de:22.
           [02/04/19 22:09:16] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed.
           [02/04/19 22:09:16] [SSH] {color:#ff0000}Authentication failed.{color}

          Manage old data complained on startup - ( didn't take real notice of message )

          Credentials, private key, was corrupted when starting Jenkins after updating to 2.163

          Eventually found and corrected.

          Might not be a bug but user error. However, caused 8 hour loss of Jenkins until problem found.

           
          Steve Graham made changes -
          Description Original: SSHLauncher\{host='myhost', port=22, credentialsId='aaa-bbb-ccc-myRealCredentials', jvmOptions='', javaPath='/home/bla/jenkins/bin/java', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.ManuallyTrustedKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
           [02/04/19 22:09:16] [SSH] Opening SSH connection to ligenr10.swb.siemens.de:22.
           [02/04/19 22:09:16] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed.
           [02/04/19 22:09:16] [SSH] {color:#ff0000}Authentication failed.{color}

          Manage old data complained on startup - ( didn't take real notice of message )

          Credentials, private key, was corrupted when starting Jenkins after updating to 2.163

          Eventually found and corrected.

          Might not be a bug but user error. However, caused 8 hour loss of Jenkins until problem found.

                      		New: SSHLauncher\{host='myhost', port=22, credentialsId='aaa-bbb-ccc-myRealCredentials', jvmOptions='', javaPath='/home/bla/jenkins/bin/java', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.ManuallyTrustedKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
           [02/04/19 22:09:16] [SSH] Opening SSH connection to ligenr10.swb.siemens.de:22.
           [02/04/19 22:09:16] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed.
           [02/04/19 22:09:16] [SSH] {color:#ff0000}Authentication failed.{color}

          Manage old data complained on startup - ( didn't take real notice of message )

          Credentials, private key, was corrupted when starting Jenkins after updating to 2.163

          Eventually found and corrected.

          ( Jenkins- Credentials->System->Global Credentials -> name )

          ( updated Private Key - entered directly )

          Might not be a bug but user error. However, caused 8 hour loss of Jenkins until problem found.

           
          Steve Graham made changes -
          Description Original: SSHLauncher\{host='myhost', port=22, credentialsId='aaa-bbb-ccc-myRealCredentials', jvmOptions='', javaPath='/home/bla/jenkins/bin/java', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.ManuallyTrustedKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
           [02/04/19 22:09:16] [SSH] Opening SSH connection to ligenr10.swb.siemens.de:22.
           [02/04/19 22:09:16] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed.
           [02/04/19 22:09:16] [SSH] {color:#ff0000}Authentication failed.{color}

          Manage old data complained on startup - ( didn't take real notice of message )

          Credentials, private key, was corrupted when starting Jenkins after updating to 2.163

          Eventually found and corrected.

          ( Jenkins- Credentials->System->Global Credentials -> name )

          ( updated Private Key - entered directly )

          Might not be a bug but user error. However, caused 8 hour loss of Jenkins until problem found.

                      		New: SSHLauncher\{host='myhost', port=22, credentialsId='aaa-bbb-ccc-myRealCredentials', jvmOptions='', javaPath='/home/bla/jenkins/bin/java', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.ManuallyTrustedKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
           [02/04/19 22:09:16] [SSH] Opening SSH connection to ligenr10.swb.siemens.de:22.
           [02/04/19 22:09:16] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed.
           [02/04/19 22:09:16] [SSH] {color:#ff0000}Authentication failed.{color}

          Manage old data complained on startup - ( didn't take real notice of message )

          Credentials, private key, was corrupted when starting Jenkins after updating to 2.163

          *+Eventually found and resolved.+*

          ( Jenkins- Credentials->System->Global Credentials -> name )

          ( updated Private Key - entered directly )

          Might not be a bug but user error. However, caused 8 hour loss of Jenkins until problem found.

           

          Devin Nusbaum added a comment -

          Perhaps another case of a migration issue with https://jenkins.io/security/advisory/2018-06-25/#SECURITY-440? CC wfollonier, but AFAIR we have never been able to reproduce this from scratch (the migration is supposed to be handled automatically). What version of Jenkins and plugins did you update from/to?

          Devin Nusbaum added a comment - Perhaps another case of a migration issue with https://jenkins.io/security/advisory/2018-06-25/#SECURITY-440? CC wfollonier , but AFAIR we have never been able to reproduce this from scratch (the migration is supposed to be handled automatically). What version of Jenkins and plugins did you update from/to?
          Devin Nusbaum made changes -
          Assignee Original: Devin Nusbaum [ dnusbaum ]

          Steve Graham added a comment -

          Thanks for looking into it.

          I upgraded from 2.162 -> 2.163

          All plugins are the latest available ( and when 2.163 was working I checked all plugins again ).

          My test system with only 2 slaves (the two machines are also in my live system, same master) worked which is why I applied the update to my live system. The two systems are however not identical, I don't need all plugins on my test system, but are up to date regarding ssh and credentials plugins.

           

          Steve Graham added a comment - Thanks for looking into it. I upgraded from 2.162 -> 2.163 All plugins are the latest available ( and when 2.163 was working I checked all plugins again ). My test system with only 2 slaves (the two machines are also in my live system, same master) worked which is why I applied the update to my live system. The two systems are however not identical, I don't need all plugins on my test system, but are up to date regarding ssh and credentials plugins.  

          Devin Nusbaum added a comment -

          If you didn't update any plugins, then I really don't know what could have happened. Are there lines in your logs at the time of the upgrade containing "SECURITY-440: Migrating UsersPrivateKeySource to DirectEntryPrivateKeySource"?

          Devin Nusbaum added a comment - If you didn't update any plugins, then I really don't know what could have happened. Are there lines in your logs at the time of the upgrade containing "SECURITY-440: Migrating UsersPrivateKeySource to DirectEntryPrivateKeySource"?

          Steve Graham added a comment -

          Yes - hundreds since at least Jun 27 2018, the last one :

          Feb 03, 2019 4:52:13 PM com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource readResolve
          INFO: SECURITY-440: Migrating UsersPrivateKeySource to DirectEntryPrivateKeySource

          I can't see what causes these messages. It does not seem to be associated to a build, maybe only with a new start ?

          Actually, there are no such messages in the log from yesterday ( 5th ) or today.  My upgrade was on the 4th. Finally worked at approx 22:30 ( CET ) 

           

          Steve Graham added a comment - Yes - hundreds since at least Jun 27 2018, the last one : Feb 03, 2019 4:52:13 PM com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource readResolve INFO: SECURITY-440: Migrating UsersPrivateKeySource to DirectEntryPrivateKeySource I can't see what causes these messages. It does not seem to be associated to a build, maybe only with a new start ? Actually, there are no such messages in the log from yesterday ( 5th ) or today.  My upgrade was on the 4th. Finally worked at approx 22:30 ( CET )   

          Devin Nusbaum added a comment - - edited

          Well, I'm glad it's working for you now. See similar reports of issues with that security fix in JENKINS-54746 and JENKINS-52232. Looking at JENKINS-54746 I think the exception in this comment (this is probably what you saw in the old data monitor) is the root cause. The migration is gated by the RunScripts permission being active when that code runs, and that code is expected to be run as ACL.System, but for some reason it ran as ACL.anonymous. When the exception is thrown in readResolve, perhaps the effect is that the serialized data is totally lost and it is as if no private keys were ever entered. Perhaps this code should be modified to instead just return if that permission is not found to avoid the conversion exception.

          Devin Nusbaum added a comment - - edited Well, I'm glad it's working for you now. See similar reports of issues with that security fix in JENKINS-54746 and JENKINS-52232 . Looking at JENKINS-54746 I think the exception in this comment (this is probably what you saw in the old data monitor) is the root cause. The migration is gated by the RunScripts permission being active when that code runs, and that code is expected to be run as ACL.System , but for some reason it ran as ACL.anonymous . When the exception is thrown in readResolve , perhaps the effect is that the serialized data is totally lost and it is as if no private keys were ever entered. Perhaps this code should be modified to instead just return if that permission is not found to avoid the conversion exception.

            Unassigned Unassigned
            sgjenkins Steve Graham
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: