notizblock I had the same problem and I could solve it by rebooting the machine (restarting the jenkins service did not solve the problem).

Does not make much sense to me but – problem solved.

After updating the ssh slaves plugin to 1.29.1, Jenkins will not connect to any of my build agents, and it complains about data stored in an old format:

ConversionException: Could not call com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource.readResolve() : anonymous is missing the Overall/RunScripts permission : Could not call com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource.readResolve() : anonymous is missing the Overall/RunScripts permission ---- Debugging information ---- message : Could not call com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource.readResolve() : anonymous is missing the Overall/RunScripts permission cause-exception : com.thoughtworks.xstream.converters.reflection.ObjectAccessException cause-message : Could not call com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource.readResolve() : anonymous is missing the Overall/RunScripts permission class : com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource required-type : com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource converter-type : hudson.util.RobustReflectionConverter path : /com.cloudbees.plugins.credentials.SystemCredentialsProvider/domainCredentialsMap/entry/java.util.concurrent.CopyOnWriteArrayList/com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey/privateKeySource line number : 21 -------------------------------

Rebooting the Jenkins machine did not help, and neither has rebooting the build agent machines.

Also for me rebooting did not definitely solve the problem, it stopped working again.

Downgraded to 1.28.1, problem seems to be solved (for now). Sorry for the noise.

I'm running Jenkins on CentOS7 with openjdk 1.8.0_191.

I had the same issue. SSH connection worked fine, upgrade from 1.28.1 to 1.29.1 and SSH connection failed with exactly the same output as in the bug description above:

• [SSH] SSH host key matches key seen previously for this host. Connection will be allowed.
• [SSH] Authentication failed.

Downgraded to 1.28.1 and connection immediately works again.

we have the same issue too.

Hi all,

Please note that update to 1.29.1 also causes Subversion Plugin (2.12.1) to fail on svn+ssh checkouts:

org.tmatesoft.svn.core.SVNCancelException: svn: E200015: authentication cancelled
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.cancel(SVNErrorManager.java:39)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.cancel(SVNErrorManager.java:34)
	at org.tmatesoft.svn.core.internal.io.svn.SVNSSHConnector.open(SVNSSHConnector.java:149)
	at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:77)
	at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:1274)
	at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.testConnection(SVNRepositoryImpl.java:99)
	at org.tmatesoft.svn.core.io.SVNRepository.getRepositoryUUID(SVNRepository.java:268)
	at jenkins.scm.impl.subversion.SVNRepositoryView.<init>(SVNRepositoryView.java:92)
	at jenkins.scm.impl.subversion.SubversionSCMSource.openSession(SubversionSCMSource.java:336)
	at jenkins.scm.impl.subversion.SubversionSCMSource.retrieve(SubversionSCMSource.java:272)
Caused: java.io.IOException
	at jenkins.scm.impl.subversion.SubversionSCMSource.retrieve(SubversionSCMSource.java:278)
	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:583)
	at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:95)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:303)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429) 

Thank you for taking a look at this!

With best regards,
Tom.

I've been working around this by temporarily giving anonymous admin access and rebooting Jenkins once so it can convert the old data, then revoking the admin access. But I need to do it on every Jenkins upgrade, so it's not a permanent solution

mrozekma What do you have to make on every upgrade?

I could not replicate the issue but the cause seems clear, so I changed to use Trilead ssh2 from Jenkins Core and use Trilead API plugin only for test, you have to uninstall the Trilead API plugin is not used.

I tried updating to 1.29.2, but I'm still seeing the same problem.

After the upgrade, I have the warning that data is stored in an old format, and the SSH build agents fail to connect with "Authentication failed".

It is weird 1.29.4 does not have any ssh connection/authentication class all are in the trilead-ssh2 in the core now, Did you uninstall the Trilead API plugin (trilead-api)? it also needs to restart Jenkins

I've updated to 1.29.4, uninstalled the Trilead API plugin, and I'm still getting the same result. Downgrading the plugin fixes it.

Unreadable data page says:

ConversionException: Could not call com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource.readResolve() : anonymous is missing the Overall/RunScripts permission : Could not call com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource.readResolve() : anonymous is missing the Overall/RunScripts permission ---- Debugging information ---- message : Could not call com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource.readResolve() : anonymous is missing the Overall/RunScripts permission cause-exception : com.thoughtworks.xstream.converters.reflection.ObjectAccessException cause-message : Could not call com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource.readResolve() : anonymous is missing the Overall/RunScripts permission class : com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource required-type : com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource converter-type : hudson.util.RobustReflectionConverter path : /com.cloudbees.plugins.credentials.SystemCredentialsProvider/domainCredentialsMap/entry/java.util.concurrent.CopyOnWriteArrayList/com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey/privateKeySource line number : 21 -------------------------------

dpogue I need more info in your case,

• This exception is from Agent logs or from Jenkins instance logs?
• What ssh-slaves version you upgrade from?
• What Jenkins version do you use?
• What ssh-credentials plugin version you have installed?
• What credentials plugin version you have installed?
• Are your agents Cloud agents (Docker/EC2/...)?
• What kind of old data report Jenkins?
• Do you have some monitors activated when you go to Manage Jenkins (the alerts that you show in this page)?

Also, the complete exception could be nice to have

Jenkins version: 2.156

(Working) SSH Slaves plugin version: 1.28.1

SSH Credentials plugin version: 1.14

Credentials plugin version: 2.1.18

Jenkins is running on an ArchLinux box in my office. The build agents are two Mac Mini machines running High Sierra, also in the office.

After updating the SSH Slaves plugin to 1.29.4 and restarting Jenkins, both my build agents are listed as offline, and there's 1 problem indicated in the Jenkins header bar. It's the "data stored in an older format and/or unreadable data" message.

Clicking Manage there takes me to the Manage Old Data page, where the only thing listed is the exception mentioned earlier, in the Unreadable Data section:

The only alert on the Manage Jenkins page is a yellow banner saying that data is stored in an old or unreadable format.

There are no errors in the Jenkins System log page.

Clicking into one of the build agents says "This agent is offline because Jenkins failed to launch the agent process on it."

Looking at the build agent log, all it says is:

[12/29/18 14:19:55] [SSH] Opening SSH connection to XX.XX.XX.XX:22.
[12/29/18 14:19:55] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed.
[12/29/18 14:19:55] [SSH] Authentication failed.
Authentication failed.
[12/29/18 14:19:55] Launch failed - cleaning up connection
[12/29/18 14:19:55] [SSH] Connection closed.

I think that I know what happens, Could you attach the JENKINS_HOME/nodes/ONE_OF_THE_AGENTS/config.xml file? change any sensible info I will replace it with valid data to replicate the issue.

I have the same problem. It works for all but one slave, which is of course the one with the job I wanted to run over the holidays. No luck so far...

This is the config of the failing node:

<?xml version='1.1' encoding='UTF-8'?>
<slave>
  <name>build-rhel6-v2</name>
  <description>RHEL6 Build Maschine mit CentOS 6.9</description>
  <remoteFS>/home/jenkins</remoteFS>
  <numExecutors>1</numExecutors>
  <mode>EXCLUSIVE</mode>
  <retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>
  <launcher class="hudson.plugins.sshslaves.SSHLauncher" plugin="ssh-slaves@1.29.1">
    <host>build-rhel6-v2</host>
    <port>22</port>
    <credentialsId>da89b575-fd00-4ec1-8040-2849d29f332b</credentialsId>
    <launchTimeoutSeconds>210</launchTimeoutSeconds>
    <maxNumRetries>10</maxNumRetries>
    <retryWaitTime>15</retryWaitTime>
    <sshHostKeyVerificationStrategy class="hudson.plugins.sshslaves.verifiers.ManuallyProvidedKeyVerificationStrategy">
      <key>
        <algorithm>ssh-rsa</algorithm>
        <key>AAAAB3NzaC1yc2EAAAABIwAAAQEApV/XRNwYycIKNMv+D5J6mVy72XZdBiu9ULIZJ0u+VDSGYZmr
UPur2HWsB15TkZWiuxSBENJsKM7fvMABP/jAQ8HXEWBuPsSUmMJIVbZqhfhpd28og7JH8edbJiFA
nDyOy5AUKSvM91mkYkl86HI3jJ1i6pe5UdOc4YYdF3lN5vh6jn0phBvIsNy8Z85uZb7Fof/5XVxw
a3KHh+8zKrHgKAM75eSvZNTIwVcw6suhcudDb8PPnlNQaoH1WISpMe/yLuhURg+nq5fNuWP9zg6q
Hl87EJ1s+Qh4KUZj6PPsxkIENye5eQKpiJcuhyLTUVgKPxfYS9Syi3ZbsIQTtsOWcw==</key>
      </key>
    </sshHostKeyVerificationStrategy>
    <tcpNoDelay>true</tcpNoDelay>
  </launcher>
  <label>linux rhel rhel6 rhel6.9 scalesdk x11utils rhel centos centos6 centos6.9 loco dynasdk zip</label>
  <nodeProperties/>
</slave>
 

The system has similar agents configured with the same configuration (apart from description, name etc.) which works fine. No idea how to solve this.

Never mind, in my case somebody else changed the credentials - Jenkins now uses a new node key to authenticate, which was not in the authorized_keys of this particular host.

The sshd logs didn't help in this case though - Jenkins claims that the key was rejected while the sshd reports that the connection was closed on client request - probably, because it has run out of keys to try...

ifernandezcalvo Sorry for the delay, had to wait 'til I was back in the office to grab the agent config.

config.xml

I check the config.xml against the last version and it works so should be something related with the credential configuration, Could you attach the snippet related with the credential used by the Agents (in JENKINS_HOME/credentials.xml)? remove the key and the passphrase if it has, I will replace them with other values. I need something like the following snippet

        <com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey plugin="ssh-credentials@1.14">
          <scope>GLOBAL</scope>
          <id>ayogo-build-agent-ssh</id>
          <description>ssh key</description>
          <username>USER</username>
          <privateKeySource class="com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$DirectEntryPrivateKeySource">
            <privateKey>{XXXX}</privateKey>
          </privateKeySource>
        </com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey>

I think is related to SECURITY-440 changes, the use of ssh keys from files was deprecated due to a potential security leak os files, but I am not sure why it works with ssh-slaves 1.28.1 with the last versions of ssh-credentials and credentials.

https://jenkins.io/security/advisory/2018-06-25/#SECURITY-440
https://github.com/jenkinsci/ssh-credentials-plugin/blob/master/src/main/java/com/cloudbees/jenkins/plugins/sshcredentials/impl/BasicSSHUserPrivateKey.java#L522-L529
https://github.com/jenkinsci/ssh-credentials-plugin/commit/18b3121fa94a174064447d637dc11539e33b3a76#diff-5b0528d739d5ca8914126dc568017ddd

Okay, that bit about the SECURITY-440 issue was helpful. I changed my agents to use "Known Host" verification instead of "Manually Authorized Key" verification and now it's working!

I was also experiencing this issue and can confirm that the SECURITY-440 fix was the source: My ssh credential was referencing a key file on master.  Directly entering the key resolved the issue.

We had a similar problem. For some reason, access to some node didn't work any more (with a log similar to the one on top) but to one node access still worked. I tried to find the differences. All the nodes had SSH private keys stored in jenkins. However, for the broken slaves, there were old entries in credentials.xml that still used UsersPrivateKeySource.

I noticed that once 1.29.4 was installed the private key data wasn't shown any more on the credential update screen for those keys (and also it seems they wouldn't be used any more when accessing the agent). I solved the problem by downgrading to 1.26 again and going to the update page of each of the affected keys and just clicked "save" which seemed to upgrade those keys to DirectEntryPrivateKeySource. Then upgrading to 1.29.4 and then everything worked.

I was able to workaround the problem by manually pasting the key from ~jenkins/.ssh/id_rsa into the GUI under Credentials -> (Selected One) -> Update. It had the (only) radio button "Enter directly" already selected, but then it was blank. It was previously something like "from a file" I swear. Once I did that, all my remote nodes came back up; no need to change any configuration in them (to address fix from dpogue above; they already had "Known Host").

https://jenkins/credentials/store/system/domain/_/credential/<UUID>/update
 

I have tested some configuration to try to replicate the issue, I could not. These are the step I made:

• Start vanilla Jenkins core 2.157 + ssh-slaves 1.26 + ssh-credential 1.12
• Create 3 SSH credentials "From the Jenkins master ~/.ssh", "From a file on Jenkins master", and "Enter directly"
• Create 3 SSH agents using the credential created before
• Check that every agent connect
• Upgrade to ssh-slaves 1.29.4 + ssh-credential 1.14 (it is a required dependency)
• Restart Jenkins
• Check every agent connect
• Check every credential is migrated to use DirectEntryPrivateKeySource ("Enter directly")

I check the code of ssh-credentials, all the credential source type have the proper method readResolve to migrate to "Enter directly", so there is something I do not see. In any case, the issue is related to ssh-credentials and a deprecated type of credentials, the workaround it is to recreate the credential with the same ID using "Enter directly" for the key, probably if you only save again the credential it will be migrated.

I will put a note in the troubleshooting guide and on changelog breaking changes.

I had the same problem today after a server reboot.

Plugin was on 1.29.1 from an earlier update, but the server never rebooted since.

Now it was not able to connect to the slaves and i got the Old Data warning.

I was able solve the problem by updating all plugins to new version (Slaves plugin upgraded to 1.29.4). Then downgrading the SSH Slaves plugin back to 1.29.1 and after a reboot again upgrading it to 1.29.4.

Now the slaves work again.

I think this is a dupe of JENKINS-52232. The exception from the old data monitor posted here gave me an idea of a possible mitigation, although the fact that that code is running as anonymous at all is a big question and is probably the root cause.

I'll close this one, then we continue in JENKINS-52232.

From SECURITY-440 / CVE-2018-1000601 fix:

SSH Credentials Plugin no longer supports SSH credentials from files on the Jenkins master file system, neither user-specified file paths nor ~/.ssh. Existing SSH credentials of these kinds are migrated to "directly entered" SSH credentials.