Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19758

Jenkins: [reproducible] ssh slave log not available, spinning wheel after upgrading to SSH Credentials 1.4, SSH Slaves 1.2 and disconnecting/reconnecting

      With a brand new Jenkins 1.532 install, no plugins, I am able to setup a dumb ssh slave fine, and can see the slave logfile when I disconnect/reconnect and click Log. This is using Credentials Plugin 1.4, SSH Credentials Plugin 0.3, SSH Slaves Plugin 0.25

      When I update to use Credentials Plugin 1.8.2 and restart, this process continues working fine. So far so good.

      But when I update to use SSH Credentials Plugin 1.4, SSH Slaves Plugin 1.2 I see a regression – when I disconnect/reconnect, then click on Log to view the slave logfile, I just see a spinning wheel in the browser. I can tell the slave did successfully connect (since I see 1 executor available), but I cannot see any slave log output.

      I can consistently reproduce this issue.

      This is a pretty major issue when used in a virtualized slaves environment where slaves are disconnected, reset, and reconnected, since this makes it very difficult to understand the status of the slaves and to troubleshoot slave connection issues.

          [JENKINS-19758] Jenkins: [reproducible] ssh slave log not available, spinning wheel after upgrading to SSH Credentials 1.4, SSH Slaves 1.2 and disconnecting/reconnecting

          Steve Roth added a comment - - edited

          Forgot to mention – this is a (dumb) ssh slave. The SSH credentials are 'username with password' global credentials.

          Steve Roth added a comment - - edited Forgot to mention – this is a (dumb) ssh slave. The SSH credentials are 'username with password' global credentials.

          Steve Roth added a comment -

          The slave logfile on the master, at logs/slaves/<someslave> is also not written when the reconnect occurs. The original logfile is moved to slave.log.1, but there is no slave.log file.

          The master logfile does not report any error:
          Sep 25, 2013 10:08:39 AM hudson.slaves.SlaveComputer tryReconnect
          INFO: Attempting to reconnect someslavebox

          Steve Roth added a comment - The slave logfile on the master, at logs/slaves/<someslave> is also not written when the reconnect occurs. The original logfile is moved to slave.log.1, but there is no slave.log file. The master logfile does not report any error: Sep 25, 2013 10:08:39 AM hudson.slaves.SlaveComputer tryReconnect INFO: Attempting to reconnect someslavebox

          Steve Roth added a comment -

          It's interesting that (after updating these plugin versions), and doing an initial Jenkins restart, I am able to see the Slave logfile by clicking on Log. But if I then disconnect and reconnect the slave, I now see the spinning wheel and no logfile.

          So this implies to me that the slave connection process is slightly different between the initial startup and the manual disconnect/reconnect.

          Steve Roth added a comment - It's interesting that (after updating these plugin versions), and doing an initial Jenkins restart, I am able to see the Slave logfile by clicking on Log. But if I then disconnect and reconnect the slave, I now see the spinning wheel and no logfile. So this implies to me that the slave connection process is slightly different between the initial startup and the manual disconnect/reconnect.

          Steve Roth added a comment - - edited

          Possibly related – though I can repro this issue in the latest and greatest Jenkins (1.532), I am also seeing it if we update our 1.518 instance (with Credentials 1.8.2, SSH credentials 1.4, SSH slaves 1.2) to 1.519 or later.

          Using the latest Credentials, SSH Credentials, and SSH Slaves versions, I have tried many different versions of Jenkins, starting with 1.519, and all of them have the same issue. But if I switch back to 1.518, the issue goes away.

          So to me that implies that 1.518 works with the newer versions of the SSH Credentials and SSH slaves plugins, but 1.519 (and later) do not. Something must have changed in the 1.519 codebase.

          So we are currently stuck on 1.518 till this issue is resolved. This is the only issue preventing us from moving to more recent Jenkins versions.

          Steve Roth added a comment - - edited Possibly related – though I can repro this issue in the latest and greatest Jenkins (1.532), I am also seeing it if we update our 1.518 instance (with Credentials 1.8.2, SSH credentials 1.4, SSH slaves 1.2) to 1.519 or later. Using the latest Credentials, SSH Credentials, and SSH Slaves versions, I have tried many different versions of Jenkins, starting with 1.519, and all of them have the same issue. But if I switch back to 1.518, the issue goes away. So to me that implies that 1.518 works with the newer versions of the SSH Credentials and SSH slaves plugins, but 1.519 (and later) do not. Something must have changed in the 1.519 codebase. So we are currently stuck on 1.518 till this issue is resolved. This is the only issue preventing us from moving to more recent Jenkins versions.

          1.533 does not show slave log in web page.
          On jenkins/logs/slaves/<SLAVE>/ there is no slave.log only slave.log.NNN where NNN is some number like .10 or .2 or .5 it seems to be random.

          Corneil du Plessis added a comment - 1.533 does not show slave log in web page. On jenkins/logs/slaves/<SLAVE>/ there is no slave.log only slave.log.NNN where NNN is some number like .10 or .2 or .5 it seems to be random.

          Stephen Connolly added a comment - - edited

          My working theory is that https://github.com/jenkinsci/jenkins/commit/d10fbc5eace959d3a5928d40b1c1f8975b3be1d0 is to blame

          Update: this theory proved incorrect when bug not reproduced in Jenkins 1.519

          Stephen Connolly added a comment - - edited My working theory is that https://github.com/jenkinsci/jenkins/commit/d10fbc5eace959d3a5928d40b1c1f8975b3be1d0 is to blame Update : this theory proved incorrect when bug not reproduced in Jenkins 1.519

          Contrary to Steve Roth, I can reproduce this issue on 1.532 but not on 1.519... I will see if I can narrow the range a bit more

          Stephen Connolly added a comment - Contrary to Steve Roth, I can reproduce this issue on 1.532 but not on 1.519... I will see if I can narrow the range a bit more

          somewhere between 1.520 and 1.523... narrowing in

          Stephen Connolly added a comment - somewhere between 1.520 and 1.523... narrowing in

          Ok, I cannot reproduce this issue with Jenkins 1.520 but I can with Jenkins 1.521... now to hunt for the culprit commit

          Stephen Connolly added a comment - Ok, I cannot reproduce this issue with Jenkins 1.520 but I can with Jenkins 1.521... now to hunt for the culprit commit

          Something is closing the PrintStream of the TaskListener after disconnection and that is disconnecting the stream from the file

          Stephen Connolly added a comment - Something is closing the PrintStream of the TaskListener after disconnection and that is disconnecting the stream from the file

          Stephen Connolly added a comment - https://github.com/jenkinsci/ssh-slaves-plugin/commit/2006a55e549d3b3094b04f9bd694c2f1ef8e9283 is the culprit

          Code changed in jenkins
          User: Stephen Connolly
          Path:
          src/main/java/hudson/plugins/sshslaves/SSHLauncher.java
          http://jenkins-ci.org/commit/ssh-slaves-plugin/f6a1e98bfb33c6d8501a8221cbad2cc6e13e5f55
          Log:
          [FIXED JENKINS-19758] Trilead SSH was closing the logger stream on Jenkins 1.521+

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: src/main/java/hudson/plugins/sshslaves/SSHLauncher.java http://jenkins-ci.org/commit/ssh-slaves-plugin/f6a1e98bfb33c6d8501a8221cbad2cc6e13e5f55 Log: [FIXED JENKINS-19758] Trilead SSH was closing the logger stream on Jenkins 1.521+

          Fixed in SSH Slaves 1.4

          Stephen Connolly added a comment - Fixed in SSH Slaves 1.4

          Steve Roth added a comment -

          Thank-you, Stephen. I will try to verify with this new version as soon as possible.

          Steve Roth added a comment - Thank-you, Stephen. I will try to verify with this new version as soon as possible.

          Steve Roth added a comment -

          I can no longer repro this issue after upgrading to SSH-slaves 1.4, and after upgrading to Jenkins 1.534. Looks good!

          Thanks!
          -Steve

          Steve Roth added a comment - I can no longer repro this issue after upgrading to SSH-slaves 1.4, and after upgrading to Jenkins 1.534. Looks good! Thanks! -Steve

          Costin Caraivan added a comment - - edited

          Upgraded to Jenkins 1.534, SSH Slaves 1.4, Credentials 1.9, SSH credentials 1.5.1.
          Same error, still.
          Master + slave Ubuntu 12.04, Java versions on both:
          java version "1.6.0_27"
          OpenJDK Runtime Environment (IcedTea6 1.12.6) (6b27-1.12.6-1ubuntu0.12.04.2)
          OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

          Costin Caraivan added a comment - - edited Upgraded to Jenkins 1.534, SSH Slaves 1.4, Credentials 1.9, SSH credentials 1.5.1. Same error, still. Master + slave Ubuntu 12.04, Java versions on both: java version "1.6.0_27" OpenJDK Runtime Environment (IcedTea6 1.12.6) (6b27-1.12.6-1ubuntu0.12.04.2) OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

          Daniel Beck added a comment -

          ccaraivan: Are you still experiencing this issue on recent Jenkins + plugin versions?

          Daniel Beck added a comment - ccaraivan : Are you still experiencing this issue on recent Jenkins + plugin versions?

          Daniel Beck added a comment -

          No response to comment asking for updated information in three weeks, so resolving as Fixed again.

          Please file a new issue if a similar issue occurs again, it will likely have a different cause.

          Daniel Beck added a comment - No response to comment asking for updated information in three weeks, so resolving as Fixed again. Please file a new issue if a similar issue occurs again, it will likely have a different cause.

            stephenconnolly Stephen Connolly
            sroth Steve Roth
            Votes:
            4 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: