-
Bug
-
Resolution: Duplicate
-
Major
-
None
-
Jenkins 1.138.3
ssh-slaves-plugin 1.29.1
Java 1.8.0_171-8u171
OS: Debian Stretch (Server and Jenkins Nodes)
-
-
ssh-slaves-1.29.2
Starting with ssh-slaves-plugin 1.29.1, the SSH agent connection with all nodes fail.
Logs on Jenkins (logfile):
[11/21/18 09:38:38] SSH Launch of HOSTNAME on HOSTNAME failed in 391 ms
Logs on Jenkins (UI):
SSHLauncher{host='HOSTNAME', port=22, credentialsId='b1f94972-dfce-45d0-9cb1-f14e3dc324b9', jvmOptions='', javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.ManuallyTrustedKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
[11/21/18 09:40:05] [SSH] Opening SSH connection to HOSTNAME:22.
[11/21/18 09:40:05] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed.
[11/21/18 09:40:05] [SSH] Authentication failed.
Authentication failed.
[11/21/18 09:40:05] Launch failed - cleaning up connection
[11/21/18 09:40:05] [SSH] Connection closed.
The auth.log of a node just contains:
Closed due to user request. [preauth]
Downgrading to 1.28.1 fixes the issue.
- duplicates
-
-
- Reopened
-
- is related to
-
-
- Resolved
-
-
-
- Reopened
-
- links to
[JENKINS-54746] Can't connect via SSH on 1.29.1
Matthew Stewart
added a comment - I was also experiencing this issue and can confirm that the SECURITY-440 fix was the source: My ssh credential was referencing a key file on master. Directly entering the key resolved the issue.
Matthew Stewart
added a comment - I was also experiencing this issue and can confirm that the SECURITY-440 fix was the source: My ssh credential was referencing a key file on master. Directly entering the key resolved the issue. 
Michael Mrozek
added a comment - I'll look into manually changing my credentials as well, but SECURITY-440 claims:
Existing SSH credentials of these kinds are migrated to "directly entered" SSH credentials.
So that doesn't seem to be working.
Michael Mrozek
added a comment - I'll look into manually changing my credentials as well, but SECURITY-440 claims:
Existing SSH credentials of these kinds are migrated to "directly entered" SSH credentials.
So that doesn't seem to be working. 
Johannes Rudolph
added a comment - We had a similar problem. For some reason, access to some node didn't work any more (with a log similar to the one on top) but to one node access still worked. I tried to find the differences. All the nodes had SSH private keys stored in jenkins. However, for the broken slaves, there were old entries in credentials.xml that still used UsersPrivateKeySource.
I noticed that once 1.29.4 was installed the private key data wasn't shown any more on the credential update screen for those keys (and also it seems they wouldn't be used any more when accessing the agent). I solved the problem by downgrading to 1.26 again and going to the update page of each of the affected keys and just clicked "save" which seemed to upgrade those keys to DirectEntryPrivateKeySource. Then upgrading to 1.29.4 and then everything worked.
Johannes Rudolph
added a comment - We had a similar problem. For some reason, access to some node didn't work any more (with a log similar to the one on top) but to one node access still worked. I tried to find the differences. All the nodes had SSH private keys stored in jenkins. However, for the broken slaves, there were old entries in credentials.xml that still used UsersPrivateKeySource.
I noticed that once 1.29.4 was installed the private key data wasn't shown any more on the credential update screen for those keys (and also it seems they wouldn't be used any more when accessing the agent). I solved the problem by downgrading to 1.26 again and going to the update page of each of the affected keys and just clicked "save" which seemed to upgrade those keys to DirectEntryPrivateKeySource. Then upgrading to 1.29.4 and then everything worked. I was able to workaround the problem by manually pasting the key from ~jenkins/.ssh/id_rsa into the GUI under Credentials -> (Selected One) -> Update. It had the (only) radio button "Enter directly" already selected, but then it was blank. It was previously something like "from a file" I swear. Once I did that, all my remote nodes came back up; no need to change any configuration in them (to address fix from dpogue above; they already had "Known Host").
https://jenkins/credentials/store/system/domain/_/credential/<UUID>/update
Aaron D. Marasco
added a comment - - edited I was able to workaround the problem by manually pasting the key from ~jenkins/.ssh/id_rsa into the GUI under Credentials -> (Selected One) -> Update. It had the (only) radio button "Enter directly" already selected, but then it was blank. It was previously something like "from a file" I swear. Once I did that, all my remote nodes came back up; no need to change any configuration in them (to address fix from dpogue above; they already had "Known Host").
https://jenkins/credentials/store/system/domain/_/credential/ <UUID>/update
I have tested some configuration to try to replicate the issue, I could not. These are the step I made:
- Start vanilla Jenkins core 2.157 + ssh-slaves 1.26 + ssh-credential 1.12
- Create 3 SSH credentials "From the Jenkins master ~/.ssh", "From a file on Jenkins master", and "Enter directly"
- Create 3 SSH agents using the credential created before
- Check that every agent connect
- Upgrade to ssh-slaves 1.29.4 + ssh-credential 1.14 (it is a required dependency)
- Restart Jenkins
- Check every agent connect
- Check every credential is migrated to use DirectEntryPrivateKeySource ("Enter directly")
I check the code of ssh-credentials, all the credential source type have the proper method readResolve to migrate to "Enter directly", so there is something I do not see. In any case, the issue is related to ssh-credentials and a deprecated type of credentials, the workaround it is to recreate the credential with the same ID using "Enter directly" for the key, probably if you only save again the credential it will be migrated.
I will put a note in the troubleshooting guide and on changelog breaking changes.
Beat Guggisberg
added a comment - I had the same problem today after a server reboot.
Plugin was on 1.29.1 from an earlier update, but the server never rebooted since.
Now it was not able to connect to the slaves and i got the Old Data warning.
I was able solve the problem by updating all plugins to new version (Slaves plugin upgraded to 1.29.4). Then downgrading the SSH Slaves plugin back to 1.29.1 and after a reboot again upgrading it to 1.29.4.
Now the slaves work again.
Beat Guggisberg
added a comment - I had the same problem today after a server reboot.
Plugin was on 1.29.1 from an earlier update, but the server never rebooted since.
Now it was not able to connect to the slaves and i got the Old Data warning.
I was able solve the problem by updating all plugins to new version (Slaves plugin upgraded to 1.29.4). Then downgrading the SSH Slaves plugin back to 1.29.1 and after a reboot again upgrading it to 1.29.4.
Now the slaves work again.
I think this is a dupe of JENKINS-52232. The exception from the old data monitor posted here gave me an idea of a possible mitigation, although the fact that that code is running as anonymous at all is a big question and is probably the root cause.
From SECURITY-440 / CVE-2018-1000601 fix:
SSH Credentials Plugin no longer supports SSH credentials from files on the Jenkins master file system, neither user-specified file paths nor ~/.ssh. Existing SSH credentials of these kinds are migrated to "directly entered" SSH credentials.
Okay, that bit about the SECURITY-440 issue was helpful. I changed my agents to use "Known Host" verification instead of "Manually Authorized Key" verification and now it's working!