Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45410

SSH Slave Agent fails occasionally with java 8 / 9 mismatch between master and slave

      We run a number of jenkins builds on slave machines. Approximately 1 in 20 builds will complete successfully but then fails with the following stack trace - apparently just about as it's ready to report that success. We eventually tracked it down to using java-8-openjdk on the master and java-9-openjdk on the slave. When the slave was switch to java-8-openjdk as well, then the problem disappeared.

      Some apparently similar bugs which look different upon closer inspection:

      https://issues.jenkins-ci.org/browse/JENKINS-21341

      https://issues.jenkins-ci.org/browse/JENKINS-22492

       

      Possible solutions:

      1) Fix this particular incompatibility with java-9-openjdk

      2) Perhaps visibly prompt users to check when different java defaults on master and slave machine

       

      Stack trace for 1 in 20 failure:
      + exit 0
      FATAL: Remote call on ca-g2-01 failed
      java.lang.NoClassDefFoundError: Could not initialize class hudson.util.ProcessTree$UnixReflection
      {{ at hudson.util.ProcessTree$UnixProcess.kill(ProcessTree.java:647)}}
      {{ at hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:668)}}
      {{ at hudson.util.ProcessTree$Unix.killAll(ProcessTree.java:589)}}
      {{ at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:996)}}
      {{ at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:987)}}
      {{ at hudson.remoting.UserRequest.perform(UserRequest.java:153)}}
      {{ at hudson.remoting.UserRequest.perform(UserRequest.java:50)}}
      {{ at hudson.remoting.Request$2.run(Request.java:336)}}
      {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}}
      {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}}
      {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1158)}}
      {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:632)}}
      {{ at java.lang.Thread.run(Thread.java:804)}}
      {{ at ......remote call to ca-g2-01(Native Method)}}
      {{ at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545)}}
      {{ at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)}}
      {{ at hudson.remoting.Channel.call(Channel.java:830)}}
      Caused: java.io.IOException: Remote call on ca-g2-01 failed
      {{ at hudson.remoting.Channel.call(Channel.java:838)}}
      {{ at hudson.Launcher$RemoteLauncher.kill(Launcher.java:984)}}
      {{ at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:496)}}
      {{ at hudson.model.Run.execute(Run.java:1735)}}
      {{ at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)}}
      {{ at hudson.model.ResourceController.execute(ResourceController.java:97)}}
      {{ at hudson.model.Executor.run(Executor.java:405)}}
      Recording test results
      Finished: FAILURE

          [JENKINS-45410] SSH Slave Agent fails occasionally with java 8 / 9 mismatch between master and slave

          Bruno Bowden created issue -
          Bruno Bowden made changes -
          Description Original: We run a number of jenkins builds on slave machines. Approximately 1 in 20 builds will complete successfully but then fails with the following stack trace - apparently just about as it's ready to report that success. We eventually tracked it down to using java-8-openjdk on the master and java-9-openjdk on the slave. When the slave was switch to java-8-openjdk as well, then the problem disappeared.

          Some apparently similar bugs which look different upon closer inspection:

          https://issues.jenkins-ci.org/browse/JENKINS-21341

          https://issues.jenkins-ci.org/browse/JENKINS-22492

           

          Possible solutions:

          1) Fix this particular incompatibility with java-9-openjdk

          2) Perhaps visibly prompt users to check when different java defaults on master and slave machine

           

          Stack trace for 1 in 20 failure:
          ...
          + exit 0 <=== SUCCESSFUL BUILD!!!
          FATAL: Remote call on XXXX failed
          java.lang.NoClassDefFoundError: Could not initialize class hudson.util.ProcessTree$UnixReflection
          at hudson.util.ProcessTree$UnixProcess.kill(ProcessTree.java:647)
          at hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:668)
          at hudson.util.ProcessTree$Unix.killAll(ProcessTree.java:589)
          at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:996)
          at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:987)
          at hudson.remoting.UserRequest.perform(UserRequest.java:153)
          at hudson.remoting.UserRequest.perform(UserRequest.java:50)
          at hudson.remoting.Request$2.run(Request.java:336)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1158)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:632)
          at java.lang.Thread.run(Thread.java:804)
          at ......remote call to ca-g2-01(Native Method)
          at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545)
          at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
          at hudson.remoting.Channel.call(Channel.java:830)
          Caused: java.io.IOException: Remote call on ca-g2-01 failed
          at hudson.remoting.Channel.call(Channel.java:838)
          at hudson.Launcher$RemoteLauncher.kill(Launcher.java:984)
          at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:496)
          at hudson.model.Run.execute(Run.java:1735)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          at hudson.model.ResourceController.execute(ResourceController.java:97)
          at hudson.model.Executor.run(Executor.java:405)
          Recording test results
          Finished: FAILURE
          New: We run a number of jenkins builds on slave machines. Approximately 1 in 20 builds will complete successfully but then fails with the following stack trace - apparently just about as it's ready to report that success. We eventually tracked it down to using java-8-openjdk on the master and java-9-openjdk on the slave. When the slave was switch to java-8-openjdk as well, then the problem disappeared.

          Some apparently similar bugs which look different upon closer inspection:

          https://issues.jenkins-ci.org/browse/JENKINS-21341

          https://issues.jenkins-ci.org/browse/JENKINS-22492

           

          Possible solutions:

          1) Fix this particular incompatibility with java-9-openjdk

          2) Perhaps visibly prompt users to check when different java defaults on master and slave machine

           

          {{Stack trace for 1 in 20 failure:}}
          {{ ...}}
          {{ + exit 0 <=== SUCCESSFUL BUILD!!!}}
          {{ FATAL: Remote call on XXXX failed}}
          {{ java.lang.NoClassDefFoundError: Could not initialize class hudson.util.ProcessTree$UnixReflection}}
          {{ at hudson.util.ProcessTree$UnixProcess.kill(ProcessTree.java:647)}}
          {{ at hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:668)}}
          {{ at hudson.util.ProcessTree$Unix.killAll(ProcessTree.java:589)}}
          {{ at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:996)}}
          {{ at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:987)}}
          {{ at hudson.remoting.UserRequest.perform(UserRequest.java:153)}}
          {{ at hudson.remoting.UserRequest.perform(UserRequest.java:50)}}
          {{ at hudson.remoting.Request$2.run(Request.java:336)}}
          {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}}
          {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}}
          {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1158)}}
          {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:632)}}
          {{ at java.lang.Thread.run(Thread.java:804)}}
          {{ at ......remote call to ca-g2-01(Native Method)}}
          {{ at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545)}}
          {{ at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)}}
          {{ at hudson.remoting.Channel.call(Channel.java:830)}}
          {{ Caused: java.io.IOException: Remote call on ca-g2-01 failed}}
          {{ at hudson.remoting.Channel.call(Channel.java:838)}}
          {{ at hudson.Launcher$RemoteLauncher.kill(Launcher.java:984)}}
          {{ at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:496)}}
          {{ at hudson.model.Run.execute(Run.java:1735)}}
          {{ at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)}}
          {{ at hudson.model.ResourceController.execute(ResourceController.java:97)}}
          {{ at hudson.model.Executor.run(Executor.java:405)}}
          {{ Recording test results}}
          {{ Finished: FAILURE}}
          Bruno Bowden made changes -
          Description Original: We run a number of jenkins builds on slave machines. Approximately 1 in 20 builds will complete successfully but then fails with the following stack trace - apparently just about as it's ready to report that success. We eventually tracked it down to using java-8-openjdk on the master and java-9-openjdk on the slave. When the slave was switch to java-8-openjdk as well, then the problem disappeared.

          Some apparently similar bugs which look different upon closer inspection:

          https://issues.jenkins-ci.org/browse/JENKINS-21341

          https://issues.jenkins-ci.org/browse/JENKINS-22492

           

          Possible solutions:

          1) Fix this particular incompatibility with java-9-openjdk

          2) Perhaps visibly prompt users to check when different java defaults on master and slave machine

           

          {{Stack trace for 1 in 20 failure:}}
          {{ ...}}
          {{ + exit 0 <=== SUCCESSFUL BUILD!!!}}
          {{ FATAL: Remote call on XXXX failed}}
          {{ java.lang.NoClassDefFoundError: Could not initialize class hudson.util.ProcessTree$UnixReflection}}
          {{ at hudson.util.ProcessTree$UnixProcess.kill(ProcessTree.java:647)}}
          {{ at hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:668)}}
          {{ at hudson.util.ProcessTree$Unix.killAll(ProcessTree.java:589)}}
          {{ at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:996)}}
          {{ at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:987)}}
          {{ at hudson.remoting.UserRequest.perform(UserRequest.java:153)}}
          {{ at hudson.remoting.UserRequest.perform(UserRequest.java:50)}}
          {{ at hudson.remoting.Request$2.run(Request.java:336)}}
          {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}}
          {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}}
          {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1158)}}
          {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:632)}}
          {{ at java.lang.Thread.run(Thread.java:804)}}
          {{ at ......remote call to ca-g2-01(Native Method)}}
          {{ at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545)}}
          {{ at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)}}
          {{ at hudson.remoting.Channel.call(Channel.java:830)}}
          {{ Caused: java.io.IOException: Remote call on ca-g2-01 failed}}
          {{ at hudson.remoting.Channel.call(Channel.java:838)}}
          {{ at hudson.Launcher$RemoteLauncher.kill(Launcher.java:984)}}
          {{ at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:496)}}
          {{ at hudson.model.Run.execute(Run.java:1735)}}
          {{ at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)}}
          {{ at hudson.model.ResourceController.execute(ResourceController.java:97)}}
          {{ at hudson.model.Executor.run(Executor.java:405)}}
          {{ Recording test results}}
          {{ Finished: FAILURE}}
          New: We run a number of jenkins builds on slave machines. Approximately 1 in 20 builds will complete successfully but then fails with the following stack trace - apparently just about as it's ready to report that success. We eventually tracked it down to using java-8-openjdk on the master and java-9-openjdk on the slave. When the slave was switch to java-8-openjdk as well, then the problem disappeared.

          Some apparently similar bugs which look different upon closer inspection:

          https://issues.jenkins-ci.org/browse/JENKINS-21341

          https://issues.jenkins-ci.org/browse/JENKINS-22492

           

          Possible solutions:

          1) Fix this particular incompatibility with java-9-openjdk

          2) Perhaps visibly prompt users to check when different java defaults on master and slave machine

           

          {{Stack trace for 1 in 20 failure:}}
          {{+ exit 0}}
          {{FATAL: Remote call on ca-g2-01 failed}}
          {{java.lang.NoClassDefFoundError: Could not initialize class hudson.util.ProcessTree$UnixReflection}}
          {{ at hudson.util.ProcessTree$UnixProcess.kill(ProcessTree.java:647)}}
          {{ at hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:668)}}
          {{ at hudson.util.ProcessTree$Unix.killAll(ProcessTree.java:589)}}
          {{ at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:996)}}
          {{ at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:987)}}
          {{ at hudson.remoting.UserRequest.perform(UserRequest.java:153)}}
          {{ at hudson.remoting.UserRequest.perform(UserRequest.java:50)}}
          {{ at hudson.remoting.Request$2.run(Request.java:336)}}
          {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}}
          {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}}
          {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1158)}}
          {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:632)}}
          {{ at java.lang.Thread.run(Thread.java:804)}}
          {{ at ......remote call to ca-g2-01(Native Method)}}
          {{ at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545)}}
          {{ at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)}}
          {{ at hudson.remoting.Channel.call(Channel.java:830)}}
          {{Caused: java.io.IOException: Remote call on ca-g2-01 failed}}
          {{ at hudson.remoting.Channel.call(Channel.java:838)}}
          {{ at hudson.Launcher$RemoteLauncher.kill(Launcher.java:984)}}
          {{ at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:496)}}
          {{ at hudson.model.Run.execute(Run.java:1735)}}
          {{ at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)}}
          {{ at hudson.model.ResourceController.execute(ResourceController.java:97)}}
          {{ at hudson.model.Executor.run(Executor.java:405)}}
          {{Recording test results}}
          {{Finished: FAILURE}}
          Oleg Nenashev made changes -
          Epic Link New: JENKINS-40689 [ 177392 ]
          Oleg Nenashev made changes -
          Assignee Original: Kohsuke Kawaguchi [ kohsuke ]
          Ashton Treadway made changes -
          Link New: This issue is duplicated by JENKINS-42413 [ JENKINS-42413 ]
          Oleg Nenashev made changes -
          Labels New: java11
          Oleg Nenashev made changes -
          Labels Original: java11 New: java11 java11-compatibility
          Oleg Nenashev made changes -
          Component/s New: versioncolumn-plugin [ 15927 ]
          Baptiste Mathus made changes -
          Labels Original: java11 java11-compatibility New: java11 java11-compatibility triaged
          Baptiste Mathus made changes -
          Resolution New: Not A Defect [ 7 ]
          Status Original: Open [ 1 ] New: Closed [ 6 ]

            Unassigned Unassigned
            brunobowden Bruno Bowden
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: