Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45410

SSH Slave Agent fails occasionally with java 8 / 9 mismatch between master and slave

      We run a number of jenkins builds on slave machines. Approximately 1 in 20 builds will complete successfully but then fails with the following stack trace - apparently just about as it's ready to report that success. We eventually tracked it down to using java-8-openjdk on the master and java-9-openjdk on the slave. When the slave was switch to java-8-openjdk as well, then the problem disappeared.

      Some apparently similar bugs which look different upon closer inspection:

      https://issues.jenkins-ci.org/browse/JENKINS-21341

      https://issues.jenkins-ci.org/browse/JENKINS-22492

       

      Possible solutions:

      1) Fix this particular incompatibility with java-9-openjdk

      2) Perhaps visibly prompt users to check when different java defaults on master and slave machine

       

      Stack trace for 1 in 20 failure:
      + exit 0
      FATAL: Remote call on ca-g2-01 failed
      java.lang.NoClassDefFoundError: Could not initialize class hudson.util.ProcessTree$UnixReflection
      {{ at hudson.util.ProcessTree$UnixProcess.kill(ProcessTree.java:647)}}
      {{ at hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:668)}}
      {{ at hudson.util.ProcessTree$Unix.killAll(ProcessTree.java:589)}}
      {{ at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:996)}}
      {{ at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:987)}}
      {{ at hudson.remoting.UserRequest.perform(UserRequest.java:153)}}
      {{ at hudson.remoting.UserRequest.perform(UserRequest.java:50)}}
      {{ at hudson.remoting.Request$2.run(Request.java:336)}}
      {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}}
      {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}}
      {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1158)}}
      {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:632)}}
      {{ at java.lang.Thread.run(Thread.java:804)}}
      {{ at ......remote call to ca-g2-01(Native Method)}}
      {{ at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545)}}
      {{ at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)}}
      {{ at hudson.remoting.Channel.call(Channel.java:830)}}
      Caused: java.io.IOException: Remote call on ca-g2-01 failed
      {{ at hudson.remoting.Channel.call(Channel.java:838)}}
      {{ at hudson.Launcher$RemoteLauncher.kill(Launcher.java:984)}}
      {{ at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:496)}}
      {{ at hudson.model.Run.execute(Run.java:1735)}}
      {{ at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)}}
      {{ at hudson.model.ResourceController.execute(ResourceController.java:97)}}
      {{ at hudson.model.Executor.run(Executor.java:405)}}
      Recording test results
      Finished: FAILURE

          [JENKINS-45410] SSH Slave Agent fails occasionally with java 8 / 9 mismatch between master and slave

          Oleg Nenashev added a comment -

          Java 9 is not supported in Jenkins

          Oleg Nenashev added a comment - Java 9 is not supported in Jenkins

          Bruno Bowden added a comment -

          Just a more visible warning of the Java 9 compatibility would be enough for now. Bonus points for pointing to documentation on how to switch Java versions.

          Bruno Bowden added a comment - Just a more visible warning of the Java 9 compatibility would be enough for now. Bonus points for pointing to documentation on how to switch Java versions.

          Oleg Nenashev added a comment -

          batmat I believe this use-case is addressed by the Version Column plugin, right?

          Maybe we can move some bits of it closer to the core, e.g. making the plugin recommended

           

          Oleg Nenashev added a comment - batmat I believe this use-case is addressed by the Version Column plugin, right? Maybe we can move some bits of it closer to the core, e.g. making the plugin recommended  

          Baptiste Mathus added a comment - - edited

          IIUC, yes. The plugin (https://wiki.jenkins.io/display/JENKINS/VersionColumn+Plugin) will allow you to automatically disconnect an agent whose Java or remoting version does not satisfy the requirements (can be configured the recommended way, I.e. major Java version matching, or paranoid = exact same version on both sides).

          Maybe we can move some bits of it closer to the core, e.g. making the plugin recommended

          Indeed, might be an idea. I'd need to check the current behaviors to refresh myself with the features (some preexisting and some I added).

          Baptiste Mathus added a comment - - edited IIUC, yes. The plugin ( https://wiki.jenkins.io/display/JENKINS/VersionColumn+Plugin ) will allow you to automatically disconnect an agent whose Java or remoting version does not satisfy the requirements (can be configured the recommended way, I.e. major Java version matching, or paranoid = exact same version on both sides). Maybe we can move some bits of it closer to the core, e.g. making the plugin recommended Indeed, might be an idea. I'd need to check the current behaviors to refresh myself with the features (some preexisting and some I added).

          I believe we cannot act on something here and this is indeed a use case for versioncolumn-plugin.

          Please follow JENKINS-54741 and install this plugin for checking/aligning versions betwen master and agents.

          We're eager to hear any feedback or concrete idea on how to surface this better though .

          Thanks!

          Baptiste Mathus added a comment - I believe we cannot act on something here and this is indeed a use case for versioncolumn-plugin. Please follow JENKINS-54741 and install this plugin for checking/aligning versions betwen master and agents. We're eager to hear any feedback or concrete idea on how to surface this better though . Thanks!

            Unassigned Unassigned
            brunobowden Bruno Bowden
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: