Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52966

Two sequential stages in a parallel stage in a declarative pipeline making use of the same agent can cause a StackOverflowError

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • Jenkins ver 2.121.2
      Windows Server 2016
      Pipeline Plugin: 2.5
      Pipeline Declarative Plugin: 1.3.1

      When running the below pipeline if I have 2 agents that are available and are using the RH7 label then everything completes as expected. However If I only have 1 available agent on the RH7 label either because the other agents are busy or offline then the job will fail with the below error message. Every machine on the RH7 label has a single executor.

      Additionally whilst the job is marked as failed, the agent that tried to run the job still shows it as running. I don't know if it would eventually time out but after a few minutes it still shows the job "running" in the executor window. Cancelling the job in the executor window returns the agent to a usable state.

      Pipeline:

      #!groovy
      
      pipeline {
          agent none
      
          stages {
              stage ("p") {
                  parallel {
                      stage ("p1") {
                          agent { label "RH7" }
      
                          stages {
                              stage ("p1s1") {
                                  steps {
                                      echo "Hello in p1s1"
                                  }
                              }
      
                              stage ("p1s2") {
                                  steps {
                                      echo "Hello in p1s2"
                                  }
                              }
                          }
                      }
      
                      stage ("p2") {
                          agent { label "RH7" }
      
                          stages {
                              stage ("p2s1") {
                                  steps {
                                      echo "Hello in p2s1"
                                  }
                              }
                          }
                      }
                  }
              }
          }
      }
      

      Error Message:

      Running in Durability level: MAX_SURVIVABILITY
      [Pipeline] stage
      [Pipeline] { (p)
      [Pipeline] parallel
      [Pipeline] [p1] { (Branch: p1)
      [Pipeline] [p2] { (Branch: p2)
      [Pipeline] [p1] stage
      [Pipeline] [p1] { (p1)
      [Pipeline] [p2] stage
      [Pipeline] [p2] { (p2)
      [Pipeline] [p1] node
      [p1] Running on Red Hat 7 - 2 in /jenkins/workspace/Problem@2
      [Pipeline] [p2] node
      [Pipeline] [p1] {
      [Pipeline] [p1] stage
      [Pipeline] [p1] { (p1s1)
      [Pipeline] [p1] echo
      [p1] Hello in p1s1
      [Pipeline] [p1] }
      [Pipeline] [p1] // stage
      [Pipeline] [p1] stage
      [Pipeline] [p1] { (p1s2)
      [Pipeline] End of Pipeline
      java.lang.StackOverflowError
      at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:111)
      at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
      at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
      at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
      at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
      at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
      TRUNCATED SEE ATTACHED LOG

      Also within the system log there is the following additional error:

      Aug 09, 2018 9:11:10 PM WARNING org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem

      Unexpected exception in CPS VM thread: CpsFlowExecutionOwner[Problem/39:Problem #39
      java.lang.IllegalStateException: JENKINS-50407: no loaded shell in CpsFlowExecutionOwner[Problem/39:Problem #39
      at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:52)
      at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
      at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
      at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
      at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
      at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
      at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
      at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
      at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)

      If there is any additional information you require please let me know.

          [JENKINS-52966] Two sequential stages in a parallel stage in a declarative pipeline making use of the same agent can cause a StackOverflowError

          Luke Ross created issue -
          Luke Ross made changes -
          Summary Original: Two sequential stages in a declarative pipeline making use of the same agent can cause a StackOverflowError New: Two sequential stages in a parallel stage in a declarative pipeline making use of the same agent can cause a StackOverflowError
          Luke Ross made changes -
          Description Original: When running the below pipeline if I have 2 agents that are available and are using the RH7 label then everything completes as expected. However If I only have 1 available agent on the RH7 label either because the other agents are busy or offline then the job will fail with the below error message.

          Additionally whilst the job is marked as failed, the agent that tried to run the job still shows it as running. I don't know if it would eventually time out but after a few minutes it still shows the job "running" in the executor window. Cancelling the job in the executor window returns the agent to a usable state.

          Pipeline:
          {code}#!groovy

          pipeline {
              agent none

              stages {
                  stage ("p") {
                      parallel {
                          stage ("p1") {
                              agent { label "RH7" }

                              stages {
                                  stage ("p1s1") {
                                      steps {
                                          echo "Hello in p1s1"
                                      }
                                  }

                                  stage ("p1s2") {
                                      steps {
                                          echo "Hello in p1s2"
                                      }
                                  }
                              }
                          }

                          stage ("p2") {
                              agent { label "RH7" }

                              stages {
                                  stage ("p2s1") {
                                      steps {
                                          echo "Hello in p2s1"
                                      }
                                  }
                              }
                          }
                      }
                  }
              }
          }{code}

          Error Message:
          {quote}Running in Durability level: MAX_SURVIVABILITY
          [Pipeline] stage
          [Pipeline] { (p)
          [Pipeline] parallel
          [Pipeline] [p1] { (Branch: p1)
          [Pipeline] [p2] { (Branch: p2)
          [Pipeline] [p1] stage
          [Pipeline] [p1] { (p1)
          [Pipeline] [p2] stage
          [Pipeline] [p2] { (p2)
          [Pipeline] [p1] node
          [p1] Running on Red Hat 7 - 2 in /jenkins/workspace/Problem@2
          [Pipeline] [p2] node
          [Pipeline] [p1] {
          [Pipeline] [p1] stage
          [Pipeline] [p1] { (p1s1)
          [Pipeline] [p1] echo
          [p1] Hello in p1s1
          [Pipeline] [p1] }
          [Pipeline] [p1] // stage
          [Pipeline] [p1] stage
          [Pipeline] [p1] { (p1s2)
          [Pipeline] End of Pipeline
          java.lang.StackOverflowError
          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:111)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
          TRUNCATED SEE ATTACHED LOG{quote}

          Also within the system log there is the following additional error:
          {quote}Aug 09, 2018 9:11:10 PM WARNING org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem

          Unexpected exception in CPS VM thread: CpsFlowExecution[Owner[Problem/39:Problem #39]]
          java.lang.IllegalStateException: JENKINS-50407: no loaded shell in CpsFlowExecution[Owner[Problem/39:Problem #39]]
          at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:52)
          at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
          at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
          at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
          at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
          at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
          at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
          at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
          at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
          at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at java.lang.Thread.run(Unknown Source)
          {quote}

          If there is any additional information you require please let me know.
          New: When running the below pipeline if I have 2 agents that are available and are using the RH7 label then everything completes as expected. However If I only have 1 available agent on the RH7 label either because the other agents are busy or offline then the job will fail with the below error message. Every machine on the RH7 label has a single executor.

          Additionally whilst the job is marked as failed, the agent that tried to run the job still shows it as running. I don't know if it would eventually time out but after a few minutes it still shows the job "running" in the executor window. Cancelling the job in the executor window returns the agent to a usable state.

          Pipeline:
          {code:java}
          #!groovy

          pipeline {
              agent none

              stages {
                  stage ("p") {
                      parallel {
                          stage ("p1") {
                              agent { label "RH7" }

                              stages {
                                  stage ("p1s1") {
                                      steps {
                                          echo "Hello in p1s1"
                                      }
                                  }

                                  stage ("p1s2") {
                                      steps {
                                          echo "Hello in p1s2"
                                      }
                                  }
                              }
                          }

                          stage ("p2") {
                              agent { label "RH7" }

                              stages {
                                  stage ("p2s1") {
                                      steps {
                                          echo "Hello in p2s1"
                                      }
                                  }
                              }
                          }
                      }
                  }
              }
          }{code}
          Error Message:
          {quote}Running in Durability level: MAX_SURVIVABILITY
           [Pipeline] stage
           [Pipeline] { (p)
           [Pipeline] parallel
           [Pipeline] [p1] { (Branch: p1)
           [Pipeline] [p2] { (Branch: p2)
           [Pipeline] [p1] stage
           [Pipeline] [p1] { (p1)
           [Pipeline] [p2] stage
           [Pipeline] [p2] { (p2)
           [Pipeline] [p1] node
           [p1] Running on Red Hat 7 - 2 in /jenkins/workspace/Problem@2
           [Pipeline] [p2] node
           [Pipeline] [p1] {
           [Pipeline] [p1] stage
           [Pipeline] [p1]
          Unknown macro: \{ (p1s1) [Pipeline] [p1] echo [p1] Hello in p1s1 [Pipeline] [p1] }
          [Pipeline] [p1] // stage
           [Pipeline] [p1] stage
           [Pipeline] [p1] { (p1s2)
           [Pipeline] End of Pipeline
           java.lang.StackOverflowError
           at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:111)
           at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
           at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
           at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
           at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
           at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
           at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
           at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
           TRUNCATED SEE ATTACHED LOG
          {quote}
          Also within the system log there is the following additional error:
          {quote}Aug 09, 2018 9:11:10 PM WARNING org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem

          Unexpected exception in CPS VM thread: CpsFlowExecution[Owner[Problem/39:Problem #39|#39]]
           java.lang.IllegalStateException: JENKINS-50407: no loaded shell in CpsFlowExecution[Owner[Problem/39:Problem #39|#39]]
           at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:52)
           at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
           at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
           at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
           at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
           at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
           at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
           at java.util.concurrent.FutureTask.run(Unknown Source)
           at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
           at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
           at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
           at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
           at java.util.concurrent.FutureTask.run(Unknown Source)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
           at java.lang.Thread.run(Unknown Source)
          {quote}
          If there is any additional information you require please let me know.
          Luke Ross made changes -
          Description Original: When running the below pipeline if I have 2 agents that are available and are using the RH7 label then everything completes as expected. However If I only have 1 available agent on the RH7 label either because the other agents are busy or offline then the job will fail with the below error message. Every machine on the RH7 label has a single executor.

          Additionally whilst the job is marked as failed, the agent that tried to run the job still shows it as running. I don't know if it would eventually time out but after a few minutes it still shows the job "running" in the executor window. Cancelling the job in the executor window returns the agent to a usable state.

          Pipeline:
          {code:java}
          #!groovy

          pipeline {
              agent none

              stages {
                  stage ("p") {
                      parallel {
                          stage ("p1") {
                              agent { label "RH7" }

                              stages {
                                  stage ("p1s1") {
                                      steps {
                                          echo "Hello in p1s1"
                                      }
                                  }

                                  stage ("p1s2") {
                                      steps {
                                          echo "Hello in p1s2"
                                      }
                                  }
                              }
                          }

                          stage ("p2") {
                              agent { label "RH7" }

                              stages {
                                  stage ("p2s1") {
                                      steps {
                                          echo "Hello in p2s1"
                                      }
                                  }
                              }
                          }
                      }
                  }
              }
          }{code}
          Error Message:
          {quote}Running in Durability level: MAX_SURVIVABILITY
           [Pipeline] stage
           [Pipeline] { (p)
           [Pipeline] parallel
           [Pipeline] [p1] { (Branch: p1)
           [Pipeline] [p2] { (Branch: p2)
           [Pipeline] [p1] stage
           [Pipeline] [p1] { (p1)
           [Pipeline] [p2] stage
           [Pipeline] [p2] { (p2)
           [Pipeline] [p1] node
           [p1] Running on Red Hat 7 - 2 in /jenkins/workspace/Problem@2
           [Pipeline] [p2] node
           [Pipeline] [p1] {
           [Pipeline] [p1] stage
           [Pipeline] [p1]
          Unknown macro: \{ (p1s1) [Pipeline] [p1] echo [p1] Hello in p1s1 [Pipeline] [p1] }
          [Pipeline] [p1] // stage
           [Pipeline] [p1] stage
           [Pipeline] [p1] { (p1s2)
           [Pipeline] End of Pipeline
           java.lang.StackOverflowError
           at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:111)
           at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
           at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
           at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
           at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
           at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
           at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
           at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
           TRUNCATED SEE ATTACHED LOG
          {quote}
          Also within the system log there is the following additional error:
          {quote}Aug 09, 2018 9:11:10 PM WARNING org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem

          Unexpected exception in CPS VM thread: CpsFlowExecution[Owner[Problem/39:Problem #39|#39]]
           java.lang.IllegalStateException: JENKINS-50407: no loaded shell in CpsFlowExecution[Owner[Problem/39:Problem #39|#39]]
           at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:52)
           at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
           at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
           at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
           at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
           at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
           at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
           at java.util.concurrent.FutureTask.run(Unknown Source)
           at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
           at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
           at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
           at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
           at java.util.concurrent.FutureTask.run(Unknown Source)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
           at java.lang.Thread.run(Unknown Source)
          {quote}
          If there is any additional information you require please let me know.
          New: When running the below pipeline if I have 2 agents that are available and are using the RH7 label then everything completes as expected. However If I only have 1 available agent on the RH7 label either because the other agents are busy or offline then the job will fail with the below error message. Every machine on the RH7 label has a single executor.

          Additionally whilst the job is marked as failed, the agent that tried to run the job still shows it as running. I don't know if it would eventually time out but after a few minutes it still shows the job "running" in the executor window. Cancelling the job in the executor window returns the agent to a usable state.

          Pipeline:
          {code:java}
          #!groovy

          pipeline {
              agent none

              stages {
                  stage ("p") {
                      parallel {
                          stage ("p1") {
                              agent { label "RH7" }

                              stages {
                                  stage ("p1s1") {
                                      steps {
                                          echo "Hello in p1s1"
                                      }
                                  }

                                  stage ("p1s2") {
                                      steps {
                                          echo "Hello in p1s2"
                                      }
                                  }
                              }
                          }

                          stage ("p2") {
                              agent { label "RH7" }

                              stages {
                                  stage ("p2s1") {
                                      steps {
                                          echo "Hello in p2s1"
                                      }
                                  }
                              }
                          }
                      }
                  }
              }
          }
          {code}

          Error Message:
          {quote}
          Running in Durability level: MAX_SURVIVABILITY
          [Pipeline] stage
          [Pipeline] \{ (p)
          [Pipeline] parallel
          [Pipeline] [p1] \{ (Branch: p1)
          [Pipeline] [p2] \{ (Branch: p2)
          [Pipeline] [p1] stage
          [Pipeline] [p1] \{ (p1)
          [Pipeline] [p2] stage
          [Pipeline] [p2] \{ (p2)
          [Pipeline] [p1] node
          [p1] Running on Red Hat 7 - 2 in /jenkins/workspace/Problem@2
          [Pipeline] [p2] node
          [Pipeline] [p1] \{
          [Pipeline] [p1] stage
          [Pipeline] [p1] \{ (p1s1)
          [Pipeline] [p1] echo
          [p1] Hello in p1s1
          [Pipeline] [p1] \}
          [Pipeline] [p1] // stage
          [Pipeline] [p1] stage
          [Pipeline] [p1] \{ (p1s2)
          [Pipeline] End of Pipeline
          java.lang.StackOverflowError
          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:111)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
          {quote}

          Also within the system log there is the following additional error:
          {quote}Aug 09, 2018 9:11:10 PM WARNING org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem

          Unexpected exception in CPS VM thread: CpsFlowExecution[Owner[Problem/39:Problem #39|#39]]
           java.lang.IllegalStateException: JENKINS-50407: no loaded shell in CpsFlowExecution[Owner[Problem/39:Problem #39|#39]]
           at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:52)
           at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
           at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
           at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
           at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
           at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
           at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
           at java.util.concurrent.FutureTask.run(Unknown Source)
           at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
           at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
           at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
           at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
           at java.util.concurrent.FutureTask.run(Unknown Source)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
           at java.lang.Thread.run(Unknown Source)
          {quote}
          If there is any additional information you require please let me know.
          Luke Ross made changes -
          Description Original: When running the below pipeline if I have 2 agents that are available and are using the RH7 label then everything completes as expected. However If I only have 1 available agent on the RH7 label either because the other agents are busy or offline then the job will fail with the below error message. Every machine on the RH7 label has a single executor.

          Additionally whilst the job is marked as failed, the agent that tried to run the job still shows it as running. I don't know if it would eventually time out but after a few minutes it still shows the job "running" in the executor window. Cancelling the job in the executor window returns the agent to a usable state.

          Pipeline:
          {code:java}
          #!groovy

          pipeline {
              agent none

              stages {
                  stage ("p") {
                      parallel {
                          stage ("p1") {
                              agent { label "RH7" }

                              stages {
                                  stage ("p1s1") {
                                      steps {
                                          echo "Hello in p1s1"
                                      }
                                  }

                                  stage ("p1s2") {
                                      steps {
                                          echo "Hello in p1s2"
                                      }
                                  }
                              }
                          }

                          stage ("p2") {
                              agent { label "RH7" }

                              stages {
                                  stage ("p2s1") {
                                      steps {
                                          echo "Hello in p2s1"
                                      }
                                  }
                              }
                          }
                      }
                  }
              }
          }
          {code}

          Error Message:
          {quote}
          Running in Durability level: MAX_SURVIVABILITY
          [Pipeline] stage
          [Pipeline] \{ (p)
          [Pipeline] parallel
          [Pipeline] [p1] \{ (Branch: p1)
          [Pipeline] [p2] \{ (Branch: p2)
          [Pipeline] [p1] stage
          [Pipeline] [p1] \{ (p1)
          [Pipeline] [p2] stage
          [Pipeline] [p2] \{ (p2)
          [Pipeline] [p1] node
          [p1] Running on Red Hat 7 - 2 in /jenkins/workspace/Problem@2
          [Pipeline] [p2] node
          [Pipeline] [p1] \{
          [Pipeline] [p1] stage
          [Pipeline] [p1] \{ (p1s1)
          [Pipeline] [p1] echo
          [p1] Hello in p1s1
          [Pipeline] [p1] \}
          [Pipeline] [p1] // stage
          [Pipeline] [p1] stage
          [Pipeline] [p1] \{ (p1s2)
          [Pipeline] End of Pipeline
          java.lang.StackOverflowError
          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:111)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
          {quote}

          Also within the system log there is the following additional error:
          {quote}Aug 09, 2018 9:11:10 PM WARNING org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem

          Unexpected exception in CPS VM thread: CpsFlowExecution[Owner[Problem/39:Problem #39|#39]]
           java.lang.IllegalStateException: JENKINS-50407: no loaded shell in CpsFlowExecution[Owner[Problem/39:Problem #39|#39]]
           at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:52)
           at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
           at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
           at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
           at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
           at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
           at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
           at java.util.concurrent.FutureTask.run(Unknown Source)
           at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
           at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
           at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
           at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
           at java.util.concurrent.FutureTask.run(Unknown Source)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
           at java.lang.Thread.run(Unknown Source)
          {quote}
          If there is any additional information you require please let me know.
          New: When running the below pipeline if I have 2 agents that are available and are using the RH7 label then everything completes as expected. However If I only have 1 available agent on the RH7 label either because the other agents are busy or offline then the job will fail with the below error message. Every machine on the RH7 label has a single executor.

          Additionally whilst the job is marked as failed, the agent that tried to run the job still shows it as running. I don't know if it would eventually time out but after a few minutes it still shows the job "running" in the executor window. Cancelling the job in the executor window returns the agent to a usable state.

          Pipeline:
          {code:java}
          #!groovy

          pipeline {
              agent none

              stages {
                  stage ("p") {
                      parallel {
                          stage ("p1") {
                              agent { label "RH7" }

                              stages {
                                  stage ("p1s1") {
                                      steps {
                                          echo "Hello in p1s1"
                                      }
                                  }

                                  stage ("p1s2") {
                                      steps {
                                          echo "Hello in p1s2"
                                      }
                                  }
                              }
                          }

                          stage ("p2") {
                              agent { label "RH7" }

                              stages {
                                  stage ("p2s1") {
                                      steps {
                                          echo "Hello in p2s1"
                                      }
                                  }
                              }
                          }
                      }
                  }
              }
          }
          {code}

          Error Message:
          {quote}
          Running in Durability level: MAX_SURVIVABILITY
          [Pipeline] stage
          [Pipeline] \{ (p)
          [Pipeline] parallel
          [Pipeline] [p1] \{ (Branch: p1)
          [Pipeline] [p2] \{ (Branch: p2)
          [Pipeline] [p1] stage
          [Pipeline] [p1] \{ (p1)
          [Pipeline] [p2] stage
          [Pipeline] [p2] \{ (p2)
          [Pipeline] [p1] node
          [p1] Running on Red Hat 7 - 2 in /jenkins/workspace/Problem@2
          [Pipeline] [p2] node
          [Pipeline] [p1] \{
          [Pipeline] [p1] stage
          [Pipeline] [p1] \{ (p1s1)
          [Pipeline] [p1] echo
          [p1] Hello in p1s1
          [Pipeline] [p1] \}
          [Pipeline] [p1] // stage
          [Pipeline] [p1] stage
          [Pipeline] [p1] \{ (p1s2)
          [Pipeline] End of Pipeline
          java.lang.StackOverflowError
          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:111)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
          TRUNCATED SEE ATTACHED LOG
          {quote}

          Also within the system log there is the following additional error:
          {quote}Aug 09, 2018 9:11:10 PM WARNING org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem

          Unexpected exception in CPS VM thread: CpsFlowExecution[Owner[Problem/39:Problem #39|#39]]
           java.lang.IllegalStateException: JENKINS-50407: no loaded shell in CpsFlowExecution[Owner[Problem/39:Problem #39|#39]]
           at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:52)
           at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
           at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
           at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
           at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
           at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
           at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
           at java.util.concurrent.FutureTask.run(Unknown Source)
           at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
           at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
           at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
           at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
           at java.util.concurrent.FutureTask.run(Unknown Source)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
           at java.lang.Thread.run(Unknown Source)
          {quote}
          If there is any additional information you require please let me know.
          Andrew Bayer made changes -
          Component/s New: pipeline-model-definition-plugin [ 21706 ]
          Component/s Original: pipeline [ 21692 ]

          Andrew Bayer added a comment -

          Huh - does this happen consistently? I can't get it to reproduce so far. The underlying issue is, obviously, something in the serialization that gets self-referential. My rough guess is that the second error is a side effect of the stack overflow, but I can't be sure.

          Andrew Bayer added a comment - Huh - does this happen consistently? I can't get it to reproduce so far. The underlying issue is, obviously, something in the serialization that gets self-referential. My rough guess is that the second error is a side effect of the stack overflow, but I can't be sure.

          Andrew Bayer added a comment -

          Oh, and does it always fail at the same point, i.e., right after [Pipeline] [p1] { (p1s2)?

          Andrew Bayer added a comment - Oh, and does it always fail at the same point, i.e., right after [Pipeline] [p1] { (p1s2) ?

          Luke Ross added a comment -

          Yeah it fails every time and fails at the same point every time.

          I just ran 10 more instances of the pipeline to confirm this.

          Luke Ross added a comment - Yeah it fails every time and fails at the same point every time. I just ran 10 more instances of the pipeline to confirm this.

          Louis Heche added a comment -

          I had the same problem, and we fixed it by allowing more stack to the each thread of the JVM. You can do that by modifying the Jenkins.xml file, add the parameter -Xss with the size of the stack you want to allow to each thread

          Louis Heche added a comment - I had the same problem, and we fixed it by allowing more stack to the each thread of the JVM. You can do that by modifying the Jenkins.xml file, add the parameter -Xss with the size of the stack you want to allow to each thread

            Unassigned Unassigned
            lukeross Luke Ross
            Votes:
            27 Vote for this issue
            Watchers:
            39 Start watching this issue

              Created:
              Updated: