Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40092

slave.jar copy via SCP fails in 2.33+

XMLWordPrintable

      After upgrade to 2.34 I got this when trying to launch slave agent over ssh:

      [11/29/16 10:55:03] [SSH] Checking java version of java
[11/29/16 10:55:04] [SSH] java -version returned 1.8.0_65.
[11/29/16 10:55:04] [SSH] Starting sftp client.
[11/29/16 10:55:04] [SSH] SFTP failed. Copying via SCP.
[11/29/16 10:55:05] [SSH] Copying latest slave.jar...
ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
java.lang.NullPointerException
	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792)
	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
	at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)
	at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:462)
	at hudson.plugins.sshslaves.SSHLauncher.copySlaveJarUsingSCP(SSHLauncher.java:1098)
	at hudson.plugins.sshslaves.SSHLauncher.copySlaveJar(SSHLauncher.java:1059)
	at hudson.plugins.sshslaves.SSHLauncher.access$300(SSHLauncher.java:137)
	at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:723)
	at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:706)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
[11/29/16 10:55:05] Launch failed - cleaning up connection
[11/29/16 10:55:05] [SSH] Connection closed.

      Full log in attachment.

      After downgrade to 2.31 it works just fine.

      Any ideas?
      I've tried to create new node but it's the same. Not sure where to search for more details

      I'm using ssh and it works:

      [jenkins@master ~]$ ssh -l jenkins slave -v
OpenSSH_5.3p1, OpenSSL 1.0.0-fips 29 Mar 2010
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to slave [xxxx] port 22.
debug1: Connection established.
debug1: identity file /home/jenkins/.ssh/id_rsa type 1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host 'slave' is known and matches the DSA host key.
debug1: Found key in /home/jenkins/.ssh/known_hosts:13
debug1: ssh_dss_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
xxxx
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: publickey
debug1: Offering public key: /home/jenkins/.ssh/id_rsa
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: password
jenkins@slave's password:
debug1: Authentication succeeded (password).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
Last login: Tue Nov 29 11:05:09 2016 from master

      I'm running jenkins on:
      master:
      Red Hat Enterprise Linux Server release 6.3 (Santiago)
      java version "1.7.0_55"

      slave:
      Red Hat Enterprise Linux Server release 6.6 (Santiago)
      openjdk version "1.8.0_65"

        1. jenkins-slave.log
          86 kB

          [JENKINS-40092] slave.jar copy via SCP fails in 2.33+

          Jesse Glick added a comment -

          Found a noncritical usage in support-core; and LocalController in the ATH seems to have an unused method which would be affected.

          Jesse Glick added a comment - Found a noncritical usage in support-core ; and LocalController in the ATH seems to have an unused method which would be affected.

          Jesse Glick added a comment - - edited

          Huh, seems like the normal code path is to use SFTP—which uses the correct API. But if there is no SFTP service, it retries with SCP, which does not use the correct API. And it seems I am responsible for forgetting to update the second code path, in a change I made four years ago!

          Jesse Glick added a comment - - edited Huh, seems like the normal code path is to use SFTP—which uses the correct API. But if there is no SFTP service, it retries with SCP, which does not use the correct API. And it seems I am responsible for forgetting to update the second code path, in a change I made four years ago!

          Jesse Glick added a comment -

          According to this chart most SSH servers include SFTP, and stodorov claims to have it, so something else went wrong when trying to connect via SFTP; currently the plugin just ignores any such errors (not reported at all) before falling back to SCP.

          The fix should be trivial, but I am running into trouble running SshSlavesPluginTest to confirm it.

          Jesse Glick added a comment - According to this chart most SSH servers include SFTP, and stodorov claims to have it, so something else went wrong when trying to connect via SFTP; currently the plugin just ignores any such errors (not reported at all) before falling back to SCP. The fix should be trivial, but I am running into trouble running SshSlavesPluginTest to confirm it.

          Jesse Glick added a comment -

          Finally got the acceptance test to run. Verified that SFTP does work in the scenario tested, which explains why no test regression has been reported. So the users affected by this bug are encountering some error using SFTP which the plugin failed to specify, triggering a fallback to the rarely used and untested SCP mode, which is what was broken by the core change in 2.33. If I introduce an artificial failure in the SFTP code, I see the reported error message.

          Jesse Glick added a comment - Finally got the acceptance test to run. Verified that SFTP does work in the scenario tested, which explains why no test regression has been reported. So the users affected by this bug are encountering some error using SFTP which the plugin failed to specify, triggering a fallback to the rarely used and untested SCP mode, which is what was broken by the core change in 2.33. If I introduce an artificial failure in the SFTP code, I see the reported error message.

          Steve Todorov added a comment -

          jglick I managed to get our slaves to work by downgrading Jenkins from 2.34 to 2.31 and Yet Another Docker Plugin from 0.1.0-rc30 to 0.1.0-rc29. Now everything seems to be working as usual. I haven't made any changes to our docker slaves. The interesting thing is that our opensuse docker slave was working fine, but ubuntu/debian/centos were getting the exception above. If you like, I can give you a slightly modified version of our Dockerfile for centos/ubuntu/debian so you can use it to test sometime tomorrow?

          Steve Todorov added a comment - jglick I managed to get our slaves to work by downgrading Jenkins from 2.34 to 2.31 and Yet Another Docker Plugin from 0.1.0-rc30 to 0.1.0-rc29. Now everything seems to be working as usual. I haven't made any changes to our docker slaves. The interesting thing is that our opensuse docker slave was working fine, but ubuntu/debian/centos were getting the exception above. If you like, I can give you a slightly modified version of our Dockerfile for centos/ubuntu/debian so you can use it to test sometime tomorrow?

          Jesse Glick added a comment -

          stodorov I have already completed testing. Something is broken on your SSH server, I think. The original problem will be reported properly in the slave log after my fix.

          Jesse Glick added a comment - stodorov I have already completed testing. Something is broken on your SSH server, I think. The original problem will be reported properly in the slave log after my fix.

          SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Jesse Glick
          Path:
          src/main/java/hudson/plugins/sshslaves/SSHLauncher.java
          http://jenkins-ci.org/commit/ssh-slaves-plugin/20492c2c8d118d8a1fd826ba804690da8eea68c1
          Log:
          [FIXED JENKINS-40092] Make sure copySlaveJarUsingSCP calls the supported API for obtaining slave.jar contents.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/hudson/plugins/sshslaves/SSHLauncher.java http://jenkins-ci.org/commit/ssh-slaves-plugin/20492c2c8d118d8a1fd826ba804690da8eea68c1 Log: [FIXED JENKINS-40092] Make sure copySlaveJarUsingSCP calls the supported API for obtaining slave.jar contents.

          Jesse Glick added a comment -

          Released 1.12 with fix.

          Jesse Glick added a comment - Released 1.12 with fix.

          Steve Todorov added a comment -

          jglick this release seems to have fixed the issue. I've upgraded to jenkins 2.34 and the latest SSH slaves plugin and everything works fine. Thanks for your fast fix!

          Steve Todorov added a comment - jglick this release seems to have fixed the issue. I've upgraded to jenkins 2.34 and the latest SSH slaves plugin and everything works fine. Thanks for your fast fix!

          SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Jesse Glick
          Path:
          src/main/java/org/jenkinsci/test/acceptance/plugins/ssh_slaves/SshSlaveLauncher.java
          src/test/java/plugins/SshSlavesPluginTest.java
          http://jenkins-ci.org/commit/acceptance-test-harness/4c7a364b881a082ca7e715f9e15416ee93704fb4
          Log:
          Merge pull request #234 from jglick/SshSlavesPluginTest-JENKINS-40092

          JENKINS-40092 Ensuring SshSlavesPluginTest can be run

          Compare: https://github.com/jenkinsci/acceptance-test-harness/compare/97a28168c585...4c7a364b881a

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/test/acceptance/plugins/ssh_slaves/SshSlaveLauncher.java src/test/java/plugins/SshSlavesPluginTest.java http://jenkins-ci.org/commit/acceptance-test-harness/4c7a364b881a082ca7e715f9e15416ee93704fb4 Log: Merge pull request #234 from jglick/SshSlavesPluginTest- JENKINS-40092 JENKINS-40092 Ensuring SshSlavesPluginTest can be run Compare: https://github.com/jenkinsci/acceptance-test-harness/compare/97a28168c585...4c7a364b881a

            jglick Jesse Glick
            pchudzik Pawel Chudzik
            Votes:
            4 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: