Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19453

Slave launcher fails after NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins$MasterComputer

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • core
    • Core versions: Jenkins-1.509.2, Jenkins-1.509.3, Jenkins-1.509.4-RC
      Master: Windows server 2007 SP1; RHEL6
      Slave: Linux Mint 64 bits; Windows 7 (JNLP)

      I am able to start slave.jar via ssh or via jnlp.
      The log of the connection shows that the node is correctly started.
      In a project, I have 2 main parts:

      • a svn checkout
      • a shell command to start cmake + compilation.

      If I launch my build, it hangs. Then, I disabled the svn checkout and leave only the shell command with a "echo hello".
      If I start the build, it hangs with the following error message:

      Started by user anonymous
      [EnvInject] - Loading node environment variables.
      Building remotely on slave_linux_64 in workspace /home/crystal/jenkins/workspace/TESTLinux
      FATAL: command execution failed
      java.io.IOException: Remote call on slave_linux_64 failed
      at hudson.remoting.Channel.call(Channel.java:723)
      at hudson.Launcher$RemoteLauncher.launch(Launcher.java:862)
      at hudson.Launcher$ProcStarter.start(Launcher.java:353)
      at hudson.Launcher$ProcStarter.join(Launcher.java:360)
      at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:91)
      at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:60)
      at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
      at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
      at hudson.model.Build$BuildExecution.build(Build.java:199)
      at hudson.model.Build$BuildExecution.doRun(Build.java:160)
      at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:586)
      at hudson.model.Run.execute(Run.java:1603)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:247)
      Caused by: java.lang.NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins$MasterComputer
      at hudson.Launcher$LocalLauncher.<init>(Launcher.java:755)
      at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:991)
      at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:965)
      at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      at hudson.remoting.Request$2.run(Request.java:326)
      at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at hudson.remoting.Engine$1$1.run(Engine.java:63)
      at java.lang.Thread.run(Thread.java:724)
      Build step 'Execute shell' marked build as failure
      Finished: FAILURE

      When I looked at the jenkins directory I configured on the slave, all the required jar are present, the workspace directory is present and the build directory is correctly created by the jenkins master. But nothing in this directory.

          [JENKINS-19453] Slave launcher fails after NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins$MasterComputer

          yann collette created issue -

          Oleg Nenashev added a comment - - edited

          We experience this issue since the migration to 1.509.2. It is being reproduced in 1.509.3 as well.
          Seems that Windows JNLP slaves periodically fail into the death loop

          [EnvInject] - Loading node environment variables.
          Building remotely on XXX in workspace C:\....
          FATAL: command execution failed
          java.io.IOException: Remote call on ru20-custom-codecs-farm09 failed
          at hudson.remoting.Channel.call(Channel.java:681)
          at hudson.Launcher$RemoteLauncher.launch(Launcher.java:862)
          at hudson.Launcher$ProcStarter.start(Launcher.java:353)
          at hudson.Launcher$ProcStarter.join(Launcher.java:360)
          ...
          at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
          at hudson.model.ResourceController.execute(ResourceController.java:88)
          at hudson.model.Executor.run(Executor.java:237)
          Caused by: java.lang.NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins$MasterComputer
          at hudson.Launcher$LocalLauncher.<init>(Launcher.java:755)
          at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:991)
          at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:965)
          at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:326)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at hudson.remoting.Engine$1$1.run(Engine.java:58)
          at java.lang.Thread.run(Unknown Source)
          Build step 'Execute Windows batch command' marked build as failure
          Finished: FAILURE

          Oleg Nenashev added a comment - - edited We experience this issue since the migration to 1.509.2. It is being reproduced in 1.509.3 as well. Seems that Windows JNLP slaves periodically fail into the death loop [EnvInject] - Loading node environment variables. Building remotely on XXX in workspace C:\.... FATAL: command execution failed java.io.IOException: Remote call on ru20-custom-codecs-farm09 failed at hudson.remoting.Channel.call(Channel.java:681) at hudson.Launcher$RemoteLauncher.launch(Launcher.java:862) at hudson.Launcher$ProcStarter.start(Launcher.java:353) at hudson.Launcher$ProcStarter.join(Launcher.java:360) ... at hudson.matrix.MatrixRun.run(MatrixRun.java:146) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:237) Caused by: java.lang.NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins$MasterComputer at hudson.Launcher$LocalLauncher.<init>(Launcher.java:755) at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:991) at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:965) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:58) at java.lang.Thread.run(Unknown Source) Build step 'Execute Windows batch command' marked build as failure Finished: FAILURE
          Oleg Nenashev made changes -
          Component/s New: core [ 15593 ]
          Component/s New: slave-setup [ 15929 ]
          Description Original: I am able to start slave.jar via ssh or via jnlp.
          The log of the connection shows that the node is correctly started.
          In a project, I have 2 main parts:
          - a svn checkout
          - a shell command to start cmake + compilation.

          If I launch my build, it hangs. Then, I disabled the svn checkout and leave only the shell command with a "echo hello".
          If I start the build, it hangs with the following error message:


          Started by user anonymous
          [EnvInject] - Loading node environment variables.
          Building remotely on slave_linux_64 in workspace /home/crystal/jenkins/workspace/TESTLinux
          FATAL: command execution failed
          java.io.IOException: Remote call on slave_linux_64 failed
          at hudson.remoting.Channel.call(Channel.java:723)
          at hudson.Launcher$RemoteLauncher.launch(Launcher.java:862)
          at hudson.Launcher$ProcStarter.start(Launcher.java:353)
          at hudson.Launcher$ProcStarter.join(Launcher.java:360)
          at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:91)
          at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:60)
          at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
          at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
          at hudson.model.Build$BuildExecution.build(Build.java:199)
          at hudson.model.Build$BuildExecution.doRun(Build.java:160)
          at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:586)
          at hudson.model.Run.execute(Run.java:1603)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
          at hudson.model.ResourceController.execute(ResourceController.java:88)
          at hudson.model.Executor.run(Executor.java:247)
          Caused by: java.lang.NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins$MasterComputer
          at hudson.Launcher$LocalLauncher.<init>(Launcher.java:755)
          at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:991)
          at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:965)
          at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:326)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
          at java.util.concurrent.FutureTask.run(FutureTask.java:166)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at hudson.remoting.Engine$1$1.run(Engine.java:63)
          at java.lang.Thread.run(Thread.java:724)
          Build step 'Execute shell' marked build as failure
          Finished: FAILURE


          When I looked at the jenkins directory I configured on the slave, all the required jar are present, the workspace directory is present and the build directory is correctly created by the jenkins master. But nothing in this directory.
          New: I am able to start slave.jar via ssh or via jnlp.
          The log of the connection shows that the node is correctly started.
          In a project, I have 2 main parts:
          - a svn checkout
          - a shell command to start cmake + compilation.

          If I launch my build, it hangs. Then, I disabled the svn checkout and leave only the shell command with a "echo hello".
          If I start the build, it hangs with the following error message:

          {quote}
          Started by user anonymous
          [EnvInject] - Loading node environment variables.
          Building remotely on slave_linux_64 in workspace /home/crystal/jenkins/workspace/TESTLinux
          FATAL: command execution failed
          java.io.IOException: Remote call on slave_linux_64 failed
          at hudson.remoting.Channel.call(Channel.java:723)
          at hudson.Launcher$RemoteLauncher.launch(Launcher.java:862)
          at hudson.Launcher$ProcStarter.start(Launcher.java:353)
          at hudson.Launcher$ProcStarter.join(Launcher.java:360)
          at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:91)
          at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:60)
          at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
          at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
          at hudson.model.Build$BuildExecution.build(Build.java:199)
          at hudson.model.Build$BuildExecution.doRun(Build.java:160)
          at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:586)
          at hudson.model.Run.execute(Run.java:1603)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
          at hudson.model.ResourceController.execute(ResourceController.java:88)
          at hudson.model.Executor.run(Executor.java:247)
          Caused by: java.lang.NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins$MasterComputer
          at hudson.Launcher$LocalLauncher.<init>(Launcher.java:755)
          at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:991)
          at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:965)
          at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:326)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
          at java.util.concurrent.FutureTask.run(FutureTask.java:166)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at hudson.remoting.Engine$1$1.run(Engine.java:63)
          at java.lang.Thread.run(Thread.java:724)
          Build step 'Execute shell' marked build as failure
          Finished: FAILURE
          {quote}

          When I looked at the jenkins directory I configured on the slave, all the required jar are present, the workspace directory is present and the build directory is correctly created by the jenkins master. But nothing in this directory.
          Environment Original: Master: Windows server 2007 SP1
          Slave: Linux Mint 64 bits
          New: Core versions: Jenkins-1.509.2, Jenkins-1.509.3
          Master: Windows server 2007 SP1; RHEL6
          Slave: Linux Mint 64 bits; Windows 7 (JNLP)
          Oleg Nenashev made changes -
          Labels New: remoting

          Oleg Nenashev added a comment -

          Update to remoting-2.32 in 1.509.4 does not solve the issue.
          Seems that is is being reproduced even more frequently.

          Oleg Nenashev added a comment - Update to remoting-2.32 in 1.509.4 does not solve the issue. Seems that is is being reproduced even more frequently.
          Oleg Nenashev made changes -
          Environment Original: Core versions: Jenkins-1.509.2, Jenkins-1.509.3
          Master: Windows server 2007 SP1; RHEL6
          Slave: Linux Mint 64 bits; Windows 7 (JNLP)
          New: Core versions: Jenkins-1.509.2, Jenkins-1.509.3, Jenkins-1.509.4-RC
          Master: Windows server 2007 SP1; RHEL6
          Slave: Linux Mint 64 bits; Windows 7 (JNLP)
          Labels Original: remoting New: class remoting
          Priority Original: Major [ 3 ] New: Critical [ 2 ]
          Oleg Nenashev made changes -
          Labels Original: class remoting New: class core remoting

          Oleg Nenashev added a comment -

          According to monitoring statistics, issue has became twice more frequent after migration to 1.509.4

          Seems that issue is caused by initialization of static channel instance, but I cannot catch it in the debugger.

          Oleg Nenashev added a comment - According to monitoring statistics, issue has became twice more frequent after migration to 1.509.4 Seems that issue is caused by initialization of static channel instance, but I cannot catch it in the debugger.
          Oleg Nenashev made changes -
          Priority Original: Critical [ 2 ] New: Blocker [ 1 ]

          Oleg Nenashev added a comment - - edited

          I’ve found a reproducible scenario for the issue:

          1. Create a node, which has big network delays / small throughput to a master
          2. Reconnect a node to have a fresh classloader
          3. Run the job, which invokes shell/batch script
          4. When the jobs performs a classloading for “Execute shell/batch”, abort the job (you will see the log below)
          5. After that, each job utilizing remote calls will fail with a “java.lang.NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins$MasterComputer” error

          Seems that issue is in incorrect behavior of remote call's during a termination of classloading processes

          A stacktrace from the step #4

          Run condition [Regular expression match] preventing perform for step [Inject environment variables]
          Regular expression run condition: Expression=[.*linux.*], Label=[64bit cn60 cn60-custom-archw-benchmarking custom haps51 win7]
          Run condition [Regular expression match] preventing perform for step [Inject environment variables]
          FATAL: command execution failed
          java.io.InterruptedIOException
          at hudson.Launcher$RemoteLauncher.launch(Launcher.java:864)
          at com.cloudbees.jenkins.plugins.customtools.DecoratedLauncher.launch(DecoratedLauncher.java:49)
          at com.cloudbees.jenkins.plugins.customtools.CustomToolInstallWrapper$2.launch(CustomToolInstallWrapper.java:218)
          at hudson.Launcher$ProcStarter.start(Launcher.java:353)
          at hudson.Launcher$ProcStarter.join(Launcher.java:360)
          at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:91)
          at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:60)
          at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
          at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:780)
          at hudson.model.Build$BuildExecution.build(Build.java:199)
          at hudson.model.Build$BuildExecution.doRun(Build.java:160)
          at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:565)
          at hudson.model.Run.execute(Run.java:1592)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
          at hudson.model.ResourceController.execute(ResourceController.java:88)
          at hudson.model.Executor.run(Executor.java:237)
          Caused by: java.lang.InterruptedException
          at java.lang.Object.wait(Native Method)
          at hudson.remoting.Request.call(Request.java:146)
          at hudson.remoting.Channel.call(Channel.java:714)
          at hudson.Launcher$RemoteLauncher.launch(Launcher.java:862)
          ... 15 more
          Build step 'Execute shell' marked build as failure
          Archiving artifacts
          Finished: FAILURE

          Oleg Nenashev added a comment - - edited I’ve found a reproducible scenario for the issue: Create a node, which has big network delays / small throughput to a master Reconnect a node to have a fresh classloader Run the job, which invokes shell/batch script When the jobs performs a classloading for “Execute shell/batch”, abort the job (you will see the log below) After that, each job utilizing remote calls will fail with a “java.lang.NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins$MasterComputer” error Seems that issue is in incorrect behavior of remote call's during a termination of classloading processes A stacktrace from the step #4 Run condition [Regular expression match] preventing perform for step [Inject environment variables] Regular expression run condition: Expression= [.*linux.*] , Label= [64bit cn60 cn60-custom-archw-benchmarking custom haps51 win7] Run condition [Regular expression match] preventing perform for step [Inject environment variables] FATAL: command execution failed java.io.InterruptedIOException at hudson.Launcher$RemoteLauncher.launch(Launcher.java:864) at com.cloudbees.jenkins.plugins.customtools.DecoratedLauncher.launch(DecoratedLauncher.java:49) at com.cloudbees.jenkins.plugins.customtools.CustomToolInstallWrapper$2.launch(CustomToolInstallWrapper.java:218) at hudson.Launcher$ProcStarter.start(Launcher.java:353) at hudson.Launcher$ProcStarter.join(Launcher.java:360) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:91) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:60) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:780) at hudson.model.Build$BuildExecution.build(Build.java:199) at hudson.model.Build$BuildExecution.doRun(Build.java:160) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:565) at hudson.model.Run.execute(Run.java:1592) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:237) Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:146) at hudson.remoting.Channel.call(Channel.java:714) at hudson.Launcher$RemoteLauncher.launch(Launcher.java:862) ... 15 more Build step 'Execute shell' marked build as failure Archiving artifacts Finished: FAILURE

            kohsuke Kohsuke Kawaguchi
            ycollet yann collette
            Votes:
            5 Vote for this issue
            Watchers:
            17 Start watching this issue

              Created:
              Updated:
              Resolved: