Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-70267

Internal pipeline crashes leave stuck executors

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Critical Critical
    • workflow-cps-plugin, workflow-job-plugin
    • None

      If a pipeline crashes due to some unexpected internal errors, jobs get stuck and block the executors from running other things.

      I have built the following minimal example for a pipeline (Taken from another issue: JENKINS-70080. This is not the main problem, just an easy way to cause the issue described later):

      class TestRunner {
  private def testCases = []

  @NonCPS
  void add(newTests) {
    testCases += newTests
  }
}


node('test'){
    stage('a'){
        script {
            def t = new TestRunner()
        }
        echo "Hello"   
    }
}

      This code produces the following crash:

      12:33:04  [Pipeline] Start of Pipeline
12:33:04  [Pipeline] node
12:33:05  Running on test in /home/jenkins/workspace/test pipeline
12:33:05  [Pipeline] End of Pipeline
12:33:05  java.lang.VerifyError: (class: TestRunner, method: add signature: (Ljava/lang/Object;)V) Unable to pop operand off an empty stack
12:33:05  	at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
12:33:05  	at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3166)
12:33:05  	at java.base/java.lang.Class.getDeclaredMethod(Class.java:2473)
12:33:05  	at java.base/jdk.internal.reflect.ReflectionFactory.findReadWriteObjectForSerialization(ReflectionFactory.java:556)
12:33:05  	at java.base/jdk.internal.reflect.ReflectionFactory.readObjectForSerialization(ReflectionFactory.java:537)
12:33:05  	at jdk.unsupported/sun.reflect.ReflectionFactory.readObjectForSerialization(ReflectionFactory.java:144)
12:33:05  	at org.jboss.marshalling.reflect.JDKSpecific$SerMethods.<init>(JDKSpecific.java:61)
12:33:05  	at org.jboss.marshalling.reflect.SerializableClass.<init>(SerializableClass.java:84)
12:33:05  	at org.jboss.marshalling.reflect.SerializableClassRegistry$1.computeValue(SerializableClassRegistry.java:62)
12:33:05  	at org.jboss.marshalling.reflect.SerializableClassRegistry$1.computeValue(SerializableClassRegistry.java:59)
12:33:05  	at java.base/java.lang.ClassValue.getFromHashMap(ClassValue.java:228)
12:33:05  	at java.base/java.lang.ClassValue.getFromBackup(ClassValue.java:210)
12:33:05  	at java.base/java.lang.ClassValue.get(ClassValue.java:116)
12:33:05  	at org.jboss.marshalling.reflect.SerializableClassRegistry.lookup(SerializableClassRegistry.java:83)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.writeNewSerializableClass(RiverMarshaller.java:1514)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.writeNewClass(RiverMarshaller.java:1417)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.writeClass(RiverMarshaller.java:1268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.writeClassClass(RiverMarshaller.java:1256)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:166)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.writeArrayObject(RiverMarshaller.java:312)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:222)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.writeArrayObject(RiverMarshaller.java:312)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:222)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1080)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1080)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
12:33:05  	at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
12:33:05  	at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
12:33:05  	at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
12:33:05  	at java.base/java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:341)
12:33:05  	at java.base/java.util.HashMap.internalWriteEntries(HashMap.java:1858)
12:33:05  	at java.base/java.util.HashMap.writeObject(HashMap.java:1412)
12:33:05  	at org.jboss.marshalling.reflect.JDKSpecific$SerMethods.callWriteObject(JDKSpecific.java:89)
12:33:05  	at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:199)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1089)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
12:33:05  	at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:116)
12:33:05  	at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.lambda$writeObject$1(RiverWriter.java:144)
12:33:05  	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:331)
12:33:05  	at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.writeObject(RiverWriter.java:143)
12:33:05  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:577)
12:33:05  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:554)
12:33:05  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgramIfPossible(CpsThreadGroup.java:537)
12:33:05  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:461)
12:33:05  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:330)
12:33:05  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:294)
12:33:05  	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
12:33:05  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
12:33:05  	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
12:33:05  	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:30)
12:33:05  	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:70)
12:33:05  	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
12:33:05  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
12:33:05  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
12:33:05  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
12:33:05  	at java.base/java.lang.Thread.run(Thread.java:829)
12:33:05  Finished: FAILURE

      The stack trace according to the monitoring looks like this:

      Stack-trace
java.lang.IllegalStateException: trying to open a build log on test pipeline #53 after it has completed
      at org.jenkinsci.plugins.workflow.job.WorkflowRun.getListener(WorkflowRun.java:232)
      at org.jenkinsci.plugins.workflow.job.WorkflowRun$NodePrintListener.onNewHead(WorkflowRun.java:1079)
      at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1556)
      at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.notifyNewHead(CpsThreadGroup.java:492)
      at org.jenkinsci.plugins.workflow.cps.FlowHead.setNewHead(FlowHead.java:158)
      at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution.launch(CpsBodyExecution.java:125)
      at org.jenkinsci.plugins.workflow.cps.CpsBodyInvoker.launch(CpsBodyInvoker.java:188)
      at org.jenkinsci.plugins.workflow.cps.CpsBodyInvoker.launch(CpsBodyInvoker.java:183)
      at org.jenkinsci.plugins.workflow.cps.CpsBodyInvoker$1.onSuccess(CpsBodyInvoker.java:159)
      at org.jenkinsci.plugins.workflow.cps.CpsBodyInvoker$1.onSuccess(CpsBodyInvoker.java:154)
      at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$5$1.run(CpsFlowExecution.java:930)
      at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38)
      at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
      at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:30)
      at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:70)
      at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      at java.base/java.lang.Thread.run(Thread.java:829)

      After the crash, one of the executors for a given agent will be permanently blocked. There is no way of aborting the build via UI through the common means. While the red X button is there, it has no effect instead just prompts a popup saying:

      Are you sure you want to abort null?

      We were able to find a workaround by explicitly removing the executor from the computer via CLI:

      // Example Code, partially based on https://github.com/cloudbees/jenkins-scripts/blob/master/ProperlyStopOnlyRunningPipelines.groovy

jenkins.model.Jenkins.instanceOrNull.getComputers().each { computer ->
  computer.executors.findAll { exec -> exec.isBusy() && exec.currentExecutable }.each { exec ->
    if (exec.currentExecutable.getFullDisplayName().contains("test pipeline")) {
      println "Stopping ${exec.currentExecutable.getFullDisplayName()}"
      computer.removeExecutor(exec)
    }
  }
}

      This issue keeps reappearing in our systems within different build jobs. This example was based on our Test System, where we reproduced the issue. However we have the same error in production (see attached environment).

        1. prod_environment.txt
          6 kB

            Unassigned Unassigned
            degelma Marian Degel
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: