Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54746

Can't connect via SSH on 1.29.1

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Duplicate
    • Labels:
      None
    • Environment:
      Jenkins 1.138.3
      ssh-slaves-plugin 1.29.1
      Java 1.8.0_171-8u171
      OS: Debian Stretch (Server and Jenkins Nodes)
    • Similar Issues:
    • Released As:
      ssh-slaves-1.29.2

      Description

      Starting with ssh-slaves-plugin 1.29.1, the SSH agent connection with all nodes fail.

      Logs on Jenkins (logfile):

      [11/21/18 09:38:38] SSH Launch of HOSTNAME on HOSTNAME failed in 391 ms
      

      Logs on Jenkins (UI):

      SSHLauncher{host='HOSTNAME', port=22, credentialsId='b1f94972-dfce-45d0-9cb1-f14e3dc324b9', jvmOptions='', javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.ManuallyTrustedKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
      [11/21/18 09:40:05] [SSH] Opening SSH connection to HOSTNAME:22.
      [11/21/18 09:40:05] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed.
      [11/21/18 09:40:05] [SSH] Authentication failed.
      Authentication failed.
      [11/21/18 09:40:05] Launch failed - cleaning up connection
      [11/21/18 09:40:05] [SSH] Connection closed.
      

      The auth.log of a node just contains:

      Closed due to user request. [preauth]
      

      Downgrading to 1.28.1 fixes the issue.

        Attachments

        1. config.xml
          2 kB
        2. jenkins1.png
          jenkins1.png
          19 kB
        3. jenkins2.png
          jenkins2.png
          143 kB

          Issue Links

            Activity

            Hide
            fbaeuerle Florian Bäuerle added a comment -

            Florian P I had the same problem and I could solve it by rebooting the machine (restarting the jenkins service did not solve the problem).

            Does not make much sense to me but – problem solved.

            Show
            fbaeuerle Florian Bäuerle added a comment - Florian P I had the same problem and I could solve it by rebooting the machine (restarting the jenkins service did not solve the problem). Does not make much sense to me but – problem solved.
            Hide
            dpogue Darryl Pogue added a comment -

            After updating the ssh slaves plugin to 1.29.1, Jenkins will not connect to any of my build agents, and it complains about data stored in an old format:

            ConversionException: Could not call com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource.readResolve() : anonymous is missing the Overall/RunScripts permission : Could not call com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource.readResolve() : anonymous is missing the Overall/RunScripts permission ---- Debugging information ---- message : Could not call com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource.readResolve() : anonymous is missing the Overall/RunScripts permission cause-exception : com.thoughtworks.xstream.converters.reflection.ObjectAccessException cause-message : Could not call com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource.readResolve() : anonymous is missing the Overall/RunScripts permission class : com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource required-type : com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource converter-type : hudson.util.RobustReflectionConverter path : /com.cloudbees.plugins.credentials.SystemCredentialsProvider/domainCredentialsMap/entry/java.util.concurrent.CopyOnWriteArrayList/com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey/privateKeySource line number : 21 -------------------------------

             

            Rebooting the Jenkins machine did not help, and neither has rebooting the build agent machines.

            Show
            dpogue Darryl Pogue added a comment - After updating the ssh slaves plugin to 1.29.1, Jenkins will not connect to any of my build agents, and it complains about data stored in an old format: ConversionException: Could not call com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource.readResolve() : anonymous is missing the Overall/RunScripts permission : Could not call com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource.readResolve() : anonymous is missing the Overall/RunScripts permission ---- Debugging information ---- message : Could not call com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource.readResolve() : anonymous is missing the Overall/RunScripts permission cause-exception : com.thoughtworks.xstream.converters.reflection.ObjectAccessException cause-message : Could not call com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource.readResolve() : anonymous is missing the Overall/RunScripts permission class : com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource required-type : com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource converter-type : hudson.util.RobustReflectionConverter path : /com.cloudbees.plugins.credentials.SystemCredentialsProvider/domainCredentialsMap/entry/java.util.concurrent.CopyOnWriteArrayList/com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey/privateKeySource line number : 21 -------------------------------   Rebooting the Jenkins machine did not help, and neither has rebooting the build agent machines.
            Hide
            fbaeuerle Florian Bäuerle added a comment -

            Also for me rebooting did not definitely solve the problem, it stopped working again.

            Downgraded to 1.28.1, problem seems to be solved (for now). Sorry for the noise.

            I'm running Jenkins on CentOS7 with openjdk 1.8.0_191.

            Show
            fbaeuerle Florian Bäuerle added a comment - Also for me rebooting did not definitely solve the problem, it stopped working again. Downgraded to 1.28.1, problem seems to be solved (for now). Sorry for the noise. I'm running Jenkins on CentOS7 with openjdk 1.8.0_191.
            Hide
            pjaytycy Pieter-Jan Busschaert added a comment -

            I had the same issue. SSH connection worked fine, upgrade from 1.28.1 to 1.29.1 and SSH connection failed with exactly the same output as in the bug description above:

            • [SSH] SSH host key matches key seen previously for this host. Connection will be allowed.
            • [SSH] Authentication failed.

            Downgraded to 1.28.1 and connection immediately works again.

            Show
            pjaytycy Pieter-Jan Busschaert added a comment - I had the same issue. SSH connection worked fine, upgrade from 1.28.1 to 1.29.1 and SSH connection failed with exactly the same output as in the bug description above: [SSH] SSH host key matches key seen previously for this host. Connection will be allowed. [SSH] Authentication failed. Downgraded to 1.28.1 and connection immediately works again.
            Hide
            maksonlee Makson Lee added a comment -

            we have the same issue too.

            Show
            maksonlee Makson Lee added a comment - we have the same issue too.
            Hide
            tom_ghyselinck Tom Ghyselinck added a comment -

            Hi all,

            Please note that update to 1.29.1 also causes Subversion Plugin (2.12.1) to fail on svn+ssh checkouts:

            org.tmatesoft.svn.core.SVNCancelException: svn: E200015: authentication cancelled
            	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.cancel(SVNErrorManager.java:39)
            	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.cancel(SVNErrorManager.java:34)
            	at org.tmatesoft.svn.core.internal.io.svn.SVNSSHConnector.open(SVNSSHConnector.java:149)
            	at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:77)
            	at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:1274)
            	at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.testConnection(SVNRepositoryImpl.java:99)
            	at org.tmatesoft.svn.core.io.SVNRepository.getRepositoryUUID(SVNRepository.java:268)
            	at jenkins.scm.impl.subversion.SVNRepositoryView.<init>(SVNRepositoryView.java:92)
            	at jenkins.scm.impl.subversion.SubversionSCMSource.openSession(SubversionSCMSource.java:336)
            	at jenkins.scm.impl.subversion.SubversionSCMSource.retrieve(SubversionSCMSource.java:272)
            Caused: java.io.IOException
            	at jenkins.scm.impl.subversion.SubversionSCMSource.retrieve(SubversionSCMSource.java:278)
            	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:583)
            	at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:95)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:303)
            	at hudson.model.ResourceController.execute(ResourceController.java:97)
            	at hudson.model.Executor.run(Executor.java:429) 

            Thank you for taking a look at this!

            With best regards,
            Tom.

            Show
            tom_ghyselinck Tom Ghyselinck added a comment - Hi all, Please note that update to 1.29.1 also causes Subversion Plugin (2.12.1) to fail on  svn+ssh checkouts: org.tmatesoft.svn.core.SVNCancelException: svn: E200015: authentication cancelled at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.cancel(SVNErrorManager.java:39) at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.cancel(SVNErrorManager.java:34) at org.tmatesoft.svn.core.internal.io.svn.SVNSSHConnector.open(SVNSSHConnector.java:149) at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:77) at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:1274) at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.testConnection(SVNRepositoryImpl.java:99) at org.tmatesoft.svn.core.io.SVNRepository.getRepositoryUUID(SVNRepository.java:268) at jenkins.scm.impl.subversion.SVNRepositoryView.<init>(SVNRepositoryView.java:92) at jenkins.scm.impl.subversion.SubversionSCMSource.openSession(SubversionSCMSource.java:336) at jenkins.scm.impl.subversion.SubversionSCMSource.retrieve(SubversionSCMSource.java:272) Caused: java.io.IOException at jenkins.scm.impl.subversion.SubversionSCMSource.retrieve(SubversionSCMSource.java:278) at jenkins.scm.api.SCMSource.fetch(SCMSource.java:583) at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:95) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:303) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) Thank you for taking a look at this! With best regards, Tom.
            Hide
            mrozekma Michael Mrozek added a comment -

            I've been working around this by temporarily giving anonymous admin access and rebooting Jenkins once so it can convert the old data, then revoking the admin access. But I need to do it on every Jenkins upgrade, so it's not a permanent solution

            Show
            mrozekma Michael Mrozek added a comment - I've been working around this by temporarily giving anonymous admin access and rebooting Jenkins once so it can convert the old data, then revoking the admin access. But I need to do it on every Jenkins upgrade, so it's not a permanent solution
            Hide
            ifernandezcalvo Ivan Fernandez Calvo added a comment -

            Michael Mrozek What do you have to make on every upgrade?

            Show
            ifernandezcalvo Ivan Fernandez Calvo added a comment - Michael Mrozek What do you have to make on every upgrade?
            Hide
            ifernandezcalvo Ivan Fernandez Calvo added a comment -

            I could not replicate the issue but the cause seems clear, so I changed to use Trilead ssh2 from Jenkins Core and use Trilead API plugin only for test, you have to uninstall the Trilead API plugin is not used.

            Show
            ifernandezcalvo Ivan Fernandez Calvo added a comment - I could not replicate the issue but the cause seems clear, so I changed to use Trilead ssh2 from Jenkins Core and use Trilead API plugin only for test, you have to uninstall the Trilead API plugin is not used.
            Hide
            dpogue Darryl Pogue added a comment -

            I tried updating to 1.29.2, but I'm still seeing the same problem.

            After the upgrade, I have the warning that data is stored in an old format, and the SSH build agents fail to connect with "Authentication failed".

            Show
            dpogue Darryl Pogue added a comment - I tried updating to 1.29.2, but I'm still seeing the same problem. After the upgrade, I have the warning that data is stored in an old format, and the SSH build agents fail to connect with "Authentication failed".
            Hide
            ifernandezcalvo Ivan Fernandez Calvo added a comment - - edited

            It is weird 1.29.4 does not have any ssh connection/authentication class all are in the trilead-ssh2 in the core now, Did you uninstall the Trilead API plugin (trilead-api)? it also needs to restart Jenkins

            Show
            ifernandezcalvo Ivan Fernandez Calvo added a comment - - edited It is weird 1.29.4 does not have any ssh connection/authentication class all are in the trilead-ssh2 in the core now, Did you uninstall the Trilead API plugin (trilead-api)? it also needs to restart Jenkins
            Hide
            dpogue Darryl Pogue added a comment -

            I've updated to 1.29.4, uninstalled the Trilead API plugin, and I'm still getting the same result. Downgrading the plugin fixes it.

             

            Unreadable data page says:

            ConversionException: Could not call com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource.readResolve() : anonymous is missing the Overall/RunScripts permission : Could not call com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource.readResolve() : anonymous is missing the Overall/RunScripts permission ---- Debugging information ---- message : Could not call com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource.readResolve() : anonymous is missing the Overall/RunScripts permission cause-exception : com.thoughtworks.xstream.converters.reflection.ObjectAccessException cause-message : Could not call com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource.readResolve() : anonymous is missing the Overall/RunScripts permission class : com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource required-type : com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource converter-type : hudson.util.RobustReflectionConverter path : /com.cloudbees.plugins.credentials.SystemCredentialsProvider/domainCredentialsMap/entry/java.util.concurrent.CopyOnWriteArrayList/com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey/privateKeySource line number : 21 -------------------------------
            Show
            dpogue Darryl Pogue added a comment - I've updated to 1.29.4, uninstalled the Trilead API plugin, and I'm still getting the same result. Downgrading the plugin fixes it.   Unreadable data page says: ConversionException: Could not call com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource.readResolve() : anonymous is missing the Overall/RunScripts permission : Could not call com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource.readResolve() : anonymous is missing the Overall/RunScripts permission ---- Debugging information ---- message : Could not call com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource.readResolve() : anonymous is missing the Overall/RunScripts permission cause-exception : com.thoughtworks.xstream.converters.reflection.ObjectAccessException cause-message : Could not call com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource.readResolve() : anonymous is missing the Overall/RunScripts permission class : com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource required-type : com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource converter-type : hudson.util.RobustReflectionConverter path : /com.cloudbees.plugins.credentials.SystemCredentialsProvider/domainCredentialsMap/entry/java.util.concurrent.CopyOnWriteArrayList/com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey/privateKeySource line number : 21 -------------------------------
            Hide
            ifernandezcalvo Ivan Fernandez Calvo added a comment - - edited

            Darryl Pogue I need more info in your case,

            • This exception is from Agent logs or from Jenkins instance logs?
            • What ssh-slaves version you upgrade from?
            • What Jenkins version do you use?
            • What ssh-credentials plugin version you have installed?
            • What credentials plugin version you have installed?
            • Are your agents Cloud agents (Docker/EC2/...)?
            • What kind of old data report Jenkins?
            • Do you have some monitors activated when you go to Manage Jenkins (the alerts that you show in this page)?

            Also, the complete exception could be nice to have

            Show
            ifernandezcalvo Ivan Fernandez Calvo added a comment - - edited Darryl Pogue I need more info in your case, This exception is from Agent logs or from Jenkins instance logs? What ssh-slaves version you upgrade from? What Jenkins version do you use? What ssh-credentials plugin version you have installed? What credentials plugin version you have installed? Are your agents Cloud agents (Docker/EC2/...)? What kind of old data report Jenkins? Do you have some monitors activated when you go to Manage Jenkins (the alerts that you show in this page)? Also, the complete exception could be nice to have
            Hide
            dpogue Darryl Pogue added a comment -

            Jenkins version: 2.156

            (Working) SSH Slaves plugin version: 1.28.1

            SSH Credentials plugin version: 1.14

            Credentials plugin version: 2.1.18

            Jenkins is running on an ArchLinux box in my office. The build agents are two Mac Mini machines running High Sierra, also in the office.

             

            After updating the SSH Slaves plugin to 1.29.4 and restarting Jenkins, both my build agents are listed as offline, and there's 1 problem indicated in the Jenkins header bar. It's the "data stored in an older format and/or unreadable data" message.

             

            Clicking Manage there takes me to the Manage Old Data page, where the only thing listed is the exception mentioned earlier, in the Unreadable Data section:

             

            The only alert on the Manage Jenkins page is a yellow banner saying that data is stored in an old or unreadable format.

            There are no errors in the Jenkins System log page.

             

            Clicking into one of the build agents says "This agent is offline because Jenkins failed to launch the agent process on it."

            Looking at the build agent log, all it says is:

            [12/29/18 14:19:55] [SSH] Opening SSH connection to XX.XX.XX.XX:22.
            [12/29/18 14:19:55] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed.
            [12/29/18 14:19:55] [SSH] Authentication failed.
            Authentication failed.
            [12/29/18 14:19:55] Launch failed - cleaning up connection
            [12/29/18 14:19:55] [SSH] Connection closed.
            Show
            dpogue Darryl Pogue added a comment - Jenkins version: 2.156 (Working) SSH Slaves plugin version: 1.28.1 SSH Credentials plugin version: 1.14 Credentials plugin version: 2.1.18 Jenkins is running on an ArchLinux box in my office. The build agents are two Mac Mini machines running High Sierra, also in the office.   After updating the SSH Slaves plugin to 1.29.4 and restarting Jenkins, both my build agents are listed as offline, and there's 1 problem indicated in the Jenkins header bar. It's the "data stored in an older format and/or unreadable data" message.   Clicking Manage there takes me to the Manage Old Data page, where the only thing listed is the exception mentioned earlier, in the Unreadable Data section:   The only alert on the Manage Jenkins page is a yellow banner saying that data is stored in an old or unreadable format. There are no errors in the Jenkins System log page.   Clicking into one of the build agents says "This agent is offline because Jenkins failed to launch the agent process on it." Looking at the build agent log, all it says is: [12/29/18 14:19:55] [SSH] Opening SSH connection to XX.XX.XX.XX:22. [12/29/18 14:19:55] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed. [12/29/18 14:19:55] [SSH] Authentication failed. Authentication failed. [12/29/18 14:19:55] Launch failed - cleaning up connection [12/29/18 14:19:55] [SSH] Connection closed.
            Hide
            ifernandezcalvo Ivan Fernandez Calvo added a comment -

            I think that I know what happens, Could you attach the JENKINS_HOME/nodes/ONE_OF_THE_AGENTS/config.xml file? change any sensible info I will replace it with valid data to replicate the issue.

            Show
            ifernandezcalvo Ivan Fernandez Calvo added a comment - I think that I know what happens, Could you attach the JENKINS_HOME/nodes/ONE_OF_THE_AGENTS/config.xml file? change any sensible info I will replace it with valid data to replicate the issue.
            Hide
            bluehorn Torsten Landschoff added a comment -

            I have the same problem. It works for all but one slave, which is of course the one with the job I wanted to run over the holidays. No luck so far...

             

            This is the config of the failing node:

             

            <?xml version='1.1' encoding='UTF-8'?>
            <slave>
              <name>build-rhel6-v2</name>
              <description>RHEL6 Build Maschine mit CentOS 6.9</description>
              <remoteFS>/home/jenkins</remoteFS>
              <numExecutors>1</numExecutors>
              <mode>EXCLUSIVE</mode>
              <retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>
              <launcher class="hudson.plugins.sshslaves.SSHLauncher" plugin="ssh-slaves@1.29.1">
                <host>build-rhel6-v2</host>
                <port>22</port>
                <credentialsId>da89b575-fd00-4ec1-8040-2849d29f332b</credentialsId>
                <launchTimeoutSeconds>210</launchTimeoutSeconds>
                <maxNumRetries>10</maxNumRetries>
                <retryWaitTime>15</retryWaitTime>
                <sshHostKeyVerificationStrategy class="hudson.plugins.sshslaves.verifiers.ManuallyProvidedKeyVerificationStrategy">
                  <key>
                    <algorithm>ssh-rsa</algorithm>
                    <key>AAAAB3NzaC1yc2EAAAABIwAAAQEApV/XRNwYycIKNMv+D5J6mVy72XZdBiu9ULIZJ0u+VDSGYZmr
            UPur2HWsB15TkZWiuxSBENJsKM7fvMABP/jAQ8HXEWBuPsSUmMJIVbZqhfhpd28og7JH8edbJiFA
            nDyOy5AUKSvM91mkYkl86HI3jJ1i6pe5UdOc4YYdF3lN5vh6jn0phBvIsNy8Z85uZb7Fof/5XVxw
            a3KHh+8zKrHgKAM75eSvZNTIwVcw6suhcudDb8PPnlNQaoH1WISpMe/yLuhURg+nq5fNuWP9zg6q
            Hl87EJ1s+Qh4KUZj6PPsxkIENye5eQKpiJcuhyLTUVgKPxfYS9Syi3ZbsIQTtsOWcw==</key>
                  </key>
                </sshHostKeyVerificationStrategy>
                <tcpNoDelay>true</tcpNoDelay>
              </launcher>
              <label>linux rhel rhel6 rhel6.9 scalesdk x11utils rhel centos centos6 centos6.9 loco dynasdk zip</label>
              <nodeProperties/>
            </slave>
             

            The system has similar agents configured with the same configuration (apart from description, name etc.) which works fine. No idea how to solve this.

            Show
            bluehorn Torsten Landschoff added a comment - I have the same problem. It works for all but one slave, which is of course the one with the job I wanted to run over the holidays. No luck so far...   This is the config of the failing node:   <?xml version='1.1' encoding='UTF-8'?> <slave> <name>build-rhel6-v2</name> <description>RHEL6 Build Maschine mit CentOS 6.9</description> <remoteFS>/home/jenkins</remoteFS> <numExecutors>1</numExecutors> <mode>EXCLUSIVE</mode> <retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/> <launcher class="hudson.plugins.sshslaves.SSHLauncher" plugin="ssh-slaves@1.29.1"> <host>build-rhel6-v2</host> <port>22</port> <credentialsId>da89b575-fd00-4ec1-8040-2849d29f332b</credentialsId> <launchTimeoutSeconds>210</launchTimeoutSeconds> <maxNumRetries>10</maxNumRetries> <retryWaitTime>15</retryWaitTime> <sshHostKeyVerificationStrategy class="hudson.plugins.sshslaves.verifiers.ManuallyProvidedKeyVerificationStrategy"> <key> <algorithm>ssh-rsa</algorithm> <key>AAAAB3NzaC1yc2EAAAABIwAAAQEApV/XRNwYycIKNMv+D5J6mVy72XZdBiu9ULIZJ0u+VDSGYZmr UPur2HWsB15TkZWiuxSBENJsKM7fvMABP/jAQ8HXEWBuPsSUmMJIVbZqhfhpd28og7JH8edbJiFA nDyOy5AUKSvM91mkYkl86HI3jJ1i6pe5UdOc4YYdF3lN5vh6jn0phBvIsNy8Z85uZb7Fof/5XVxw a3KHh+8zKrHgKAM75eSvZNTIwVcw6suhcudDb8PPnlNQaoH1WISpMe/yLuhURg+nq5fNuWP9zg6q Hl87EJ1s+Qh4KUZj6PPsxkIENye5eQKpiJcuhyLTUVgKPxfYS9Syi3ZbsIQTtsOWcw==</key> </key> </sshHostKeyVerificationStrategy> <tcpNoDelay>true</tcpNoDelay> </launcher> <label>linux rhel rhel6 rhel6.9 scalesdk x11utils rhel centos centos6 centos6.9 loco dynasdk zip</label> <nodeProperties/> </slave> The system has similar agents configured with the same configuration (apart from description, name etc.) which works fine. No idea how to solve this.
            Hide
            bluehorn Torsten Landschoff added a comment -

            Never mind, in my case somebody else changed the credentials - Jenkins now uses a new node key to authenticate, which was not in the authorized_keys of this particular host.

            The sshd logs didn't help in this case though - Jenkins claims that the key was rejected while the sshd reports that the connection was closed on client request - probably, because it has run out of keys to try...

            Show
            bluehorn Torsten Landschoff added a comment - Never mind, in my case somebody else changed the credentials - Jenkins now uses a new node key to authenticate, which was not in the authorized_keys of this particular host. The sshd logs didn't help in this case though - Jenkins claims that the key was rejected while the sshd reports that the connection was closed on client request - probably, because it has run out of keys to try...
            Hide
            dpogue Darryl Pogue added a comment -

            Ivan Fernandez Calvo Sorry for the delay, had to wait 'til I was back in the office to grab the agent config.

            config.xml

            Show
            dpogue Darryl Pogue added a comment - Ivan Fernandez Calvo Sorry for the delay, had to wait 'til I was back in the office to grab the agent config. config.xml
            Hide
            ifernandezcalvo Ivan Fernandez Calvo added a comment - - edited

            I check the config.xml against the last version and it works so should be something related with the credential configuration, Could you attach the snippet related with the credential used by the Agents (in JENKINS_HOME/credentials.xml)? remove the key and the passphrase if it has, I will replace them with other values. I need something like the following snippet

                    <com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey plugin="ssh-credentials@1.14">
                      <scope>GLOBAL</scope>
                      <id>ayogo-build-agent-ssh</id>
                      <description>ssh key</description>
                      <username>USER</username>
                      <privateKeySource class="com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$DirectEntryPrivateKeySource">
                        <privateKey>{XXXX}</privateKey>
                      </privateKeySource>
                    </com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey>
            
            Show
            ifernandezcalvo Ivan Fernandez Calvo added a comment - - edited I check the config.xml against the last version and it works so should be something related with the credential configuration, Could you attach the snippet related with the credential used by the Agents (in JENKINS_HOME/credentials.xml)? remove the key and the passphrase if it has, I will replace them with other values. I need something like the following snippet <com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey plugin= "ssh-credentials@1.14" > <scope>GLOBAL</scope> <id>ayogo-build-agent-ssh</id> <description>ssh key</description> <username>USER</username> <privateKeySource class= "com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$DirectEntryPrivateKeySource" > <privateKey>{XXXX}</privateKey> </privateKeySource> </com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey>
            Hide
            ifernandezcalvo Ivan Fernandez Calvo added a comment -

            I think is related to SECURITY-440 changes, the use of ssh keys from files was deprecated due to a potential security leak os files, but I am not sure why it works with ssh-slaves 1.28.1 with the last versions of ssh-credentials and credentials.

            https://jenkins.io/security/advisory/2018-06-25/#SECURITY-440
            https://github.com/jenkinsci/ssh-credentials-plugin/blob/master/src/main/java/com/cloudbees/jenkins/plugins/sshcredentials/impl/BasicSSHUserPrivateKey.java#L522-L529
            https://github.com/jenkinsci/ssh-credentials-plugin/commit/18b3121fa94a174064447d637dc11539e33b3a76#diff-5b0528d739d5ca8914126dc568017ddd

            Show
            ifernandezcalvo Ivan Fernandez Calvo added a comment - I think is related to SECURITY-440 changes, the use of ssh keys from files was deprecated due to a potential security leak os files, but I am not sure why it works with ssh-slaves 1.28.1 with the last versions of ssh-credentials and credentials. https://jenkins.io/security/advisory/2018-06-25/#SECURITY-440 https://github.com/jenkinsci/ssh-credentials-plugin/blob/master/src/main/java/com/cloudbees/jenkins/plugins/sshcredentials/impl/BasicSSHUserPrivateKey.java#L522-L529 https://github.com/jenkinsci/ssh-credentials-plugin/commit/18b3121fa94a174064447d637dc11539e33b3a76#diff-5b0528d739d5ca8914126dc568017ddd
            Hide
            dpogue Darryl Pogue added a comment -

            Okay, that bit about the SECURITY-440 issue was helpful. I changed my agents to use "Known Host" verification instead of "Manually Authorized Key" verification and now it's working!

            Show
            dpogue Darryl Pogue added a comment - Okay, that bit about the SECURITY-440 issue was helpful. I changed my agents to use "Known Host" verification instead of "Manually Authorized Key" verification and now it's working!
            Hide
            k8wbkxwgtenhgnghfm9t Matthew Stewart added a comment -

            I was also experiencing this issue and can confirm that the SECURITY-440 fix was the source: My ssh credential was referencing a key file on master.  Directly entering the key resolved the issue.

            Show
            k8wbkxwgtenhgnghfm9t Matthew Stewart added a comment - I was also experiencing this issue and can confirm that the SECURITY-440 fix was the source: My ssh credential was referencing a key file on master.  Directly entering the key resolved the issue.
            Hide
            mrozekma Michael Mrozek added a comment -

            I'll look into manually changing my credentials as well, but SECURITY-440 claims:

            Existing SSH credentials of these kinds are migrated to "directly entered" SSH credentials.

            So that doesn't seem to be working.

            Show
            mrozekma Michael Mrozek added a comment - I'll look into manually changing my credentials as well, but SECURITY-440 claims: Existing SSH credentials of these kinds are migrated to "directly entered" SSH credentials. So that doesn't seem to be working.
            Hide
            jrudolph Johannes Rudolph added a comment -

            We had a similar problem. For some reason, access to some node didn't work any more (with a log similar to the one on top) but to one node access still worked. I tried to find the differences. All the nodes had SSH private keys stored in jenkins. However, for the broken slaves, there were old entries in credentials.xml that still used UsersPrivateKeySource.

             

            I noticed that once 1.29.4 was installed the private key data wasn't shown any more on the credential update screen for those keys (and also it seems they wouldn't be used any more when accessing the agent). I solved the problem by downgrading to 1.26 again and going to the update page of each of the affected keys and just clicked "save" which seemed to upgrade those keys to DirectEntryPrivateKeySource. Then upgrading to 1.29.4 and then everything worked.

            Show
            jrudolph Johannes Rudolph added a comment - We had a similar problem. For some reason, access to some node didn't work any more (with a log similar to the one on top) but to one node access still worked. I tried to find the differences. All the nodes had SSH private keys stored in jenkins. However, for the broken slaves, there were old entries in credentials.xml that still used UsersPrivateKeySource.   I noticed that once 1.29.4 was installed the private key data wasn't shown any more on the credential update screen for those keys (and also it seems they wouldn't be used any more when accessing the agent). I solved the problem by downgrading to 1.26 again and going to the update page of each of the affected keys and just clicked "save" which seemed to upgrade those keys to DirectEntryPrivateKeySource. Then upgrading to 1.29.4 and then everything worked.
            Hide
            aarondmarasco_vsi Aaron D. Marasco added a comment - - edited

            I was able to workaround the problem by manually pasting the key from ~jenkins/.ssh/id_rsa into the GUI under Credentials -> (Selected One) -> Update. It had the (only) radio button "Enter directly" already selected, but then it was blank. It was previously something like "from a file" I swear. Once I did that, all my remote nodes came back up; no need to change any configuration in them (to address fix from Darryl Pogue above; they already had "Known Host").

            https://jenkins/credentials/store/system/domain/_/credential/<UUID>/update
             

             

            Show
            aarondmarasco_vsi Aaron D. Marasco added a comment - - edited I was able to workaround the problem by manually pasting the key from ~jenkins/.ssh/id_rsa into the GUI under Credentials -> (Selected One) -> Update. It had the (only) radio button "Enter directly" already selected, but then it was blank. It was previously something like "from a file" I swear. Once I did that, all my remote nodes came back up; no need to change any configuration in them (to address fix from Darryl Pogue above; they already had "Known Host"). https://jenkins/credentials/store/system/domain/_/credential/ <UUID>/update    
            Hide
            ifernandezcalvo Ivan Fernandez Calvo added a comment -

            I have tested some configuration to try to replicate the issue, I could not. These are the step I made:

            • Start vanilla Jenkins core 2.157 + ssh-slaves 1.26 + ssh-credential 1.12
            • Create 3 SSH credentials "From the Jenkins master ~/.ssh", "From a file on Jenkins master", and "Enter directly"
            • Create 3 SSH agents using the credential created before
            • Check that every agent connect
            • Upgrade to ssh-slaves 1.29.4 + ssh-credential 1.14 (it is a required dependency)
            • Restart Jenkins
            • Check every agent connect
            • Check every credential is migrated to use DirectEntryPrivateKeySource ("Enter directly")

            I check the code of ssh-credentials, all the credential source type have the proper method readResolve to migrate to "Enter directly", so there is something I do not see. In any case, the issue is related to ssh-credentials and a deprecated type of credentials, the workaround it is to recreate the credential with the same ID using "Enter directly" for the key, probably if you only save again the credential it will be migrated.

            I will put a note in the troubleshooting guide and on changelog breaking changes.

            Show
            ifernandezcalvo Ivan Fernandez Calvo added a comment - I have tested some configuration to try to replicate the issue, I could not. These are the step I made: Start vanilla Jenkins core 2.157 + ssh-slaves 1.26 + ssh-credential 1.12 Create 3 SSH credentials "From the Jenkins master ~/.ssh", "From a file on Jenkins master", and "Enter directly" Create 3 SSH agents using the credential created before Check that every agent connect Upgrade to ssh-slaves 1.29.4 + ssh-credential 1.14 (it is a required dependency) Restart Jenkins Check every agent connect Check every credential is migrated to use DirectEntryPrivateKeySource ("Enter directly") I check the code of ssh-credentials, all the credential source type have the proper method readResolve to migrate to "Enter directly", so there is something I do not see. In any case, the issue is related to ssh-credentials and a deprecated type of credentials, the workaround it is to recreate the credential with the same ID using "Enter directly" for the key, probably if you only save again the credential it will be migrated. I will put a note in the troubleshooting guide and on changelog breaking changes.
            Hide
            sintbert Beat Guggisberg added a comment -

            I had the same problem today after a server reboot.

            Plugin was on 1.29.1 from an earlier update, but the server never rebooted since.

            Now it was not able to connect to the slaves and i got the Old Data warning.

             

            I was able solve the problem by updating all plugins to new version (Slaves plugin upgraded to 1.29.4). Then downgrading the SSH Slaves plugin back to 1.29.1 and after a reboot again upgrading it to 1.29.4.

            Now the slaves work again.

             

            Show
            sintbert Beat Guggisberg added a comment - I had the same problem today after a server reboot. Plugin was on 1.29.1 from an earlier update, but the server never rebooted since. Now it was not able to connect to the slaves and i got the Old Data warning.   I was able solve the problem by updating all plugins to new version (Slaves plugin upgraded to 1.29.4). Then downgrading the SSH Slaves plugin back to 1.29.1 and after a reboot again upgrading it to 1.29.4. Now the slaves work again.  
            Hide
            dnusbaum Devin Nusbaum added a comment -

            I think this is a dupe of JENKINS-52232. The exception from the old data monitor posted here gave me an idea of a possible mitigation, although the fact that that code is running as anonymous at all is a big question and is probably the root cause.

            Show
            dnusbaum Devin Nusbaum added a comment - I think this is a dupe of JENKINS-52232 . The exception from the old data monitor posted here gave me an idea of a possible mitigation, although the fact that that code is running as anonymous at all is a big question and is probably the root cause.
            Hide
            ifernandezcalvo Ivan Fernandez Calvo added a comment -

            I'll close this one, then we continue in JENKINS-52232.

            Show
            ifernandezcalvo Ivan Fernandez Calvo added a comment - I'll close this one, then we continue in JENKINS-52232 .
            Hide
            vazhnov Alexey Vazhnov added a comment -

            From SECURITY-440 / CVE-2018-1000601 fix:

            SSH Credentials Plugin no longer supports SSH credentials from files on the Jenkins master file system, neither user-specified file paths nor ~/.ssh. Existing SSH credentials of these kinds are migrated to "directly entered" SSH credentials.

            Show
            vazhnov Alexey Vazhnov added a comment - From SECURITY-440 / CVE-2018-1000601 fix: SSH Credentials Plugin no longer supports SSH credentials from files on the Jenkins master file system, neither user-specified file paths nor ~/.ssh. Existing SSH credentials of these kinds are migrated to "directly entered" SSH credentials.

              People

              Assignee:
              ifernandezcalvo Ivan Fernandez Calvo
              Reporter:
              notizblock Florian P
              Votes:
              9 Vote for this issue
              Watchers:
              19 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: