Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37730

DurableTaskStep.Execution hanging after process is dead

XMLWordPrintable

      Found a case where a sh step ceased to produce more output in the middle of a command, for no apparent reason, and the build did not respond to normal abort. The virtual thread dump said

      Thread #80
	at DSL.sh(completed process (code -1) in /...@tmp/durable-... on ... (pid: ...))
	at ...

      But there is no active CPS VM thread, and nothing visibly happening on the agent, and all Timer threads are idle. So it seems that a call to check would have caused the step to fail—but perhaps none came?

      Possibly stop should do its own check for a non-null Controller.exitStatus and immediately fail in such a case (but we run the risk of delivering doubled-up events if check does run later); or synchronously call check (though this runs the risk of having two such calls run simultaneously—it is not thread safe); or somehow reschedule it (same problem).

      At a minimum, the virtual thread dump should indicate what the current recurrencePeriod is. And the calls to schedule could save their ScheduledFuture results in a transient field, so we can check cancelled and done flags. Such diagnostics might make it clearer next time what actually happened.

      Also a term claimed to be terminating the sh step, but the build still did not finish. Again nothing in the physical thread dumps, and virtual thread dump still claims to be inside sh. System log showed

      ... WARNING org.jenkinsci.plugins.workflow.cps.CpsStepContext onFailure
already completed CpsStepContext[186]:Owner[...]
java.lang.IllegalStateException: org.jenkinsci.plugins.workflow.steps.FlowInterruptedException
	at org.jenkinsci.plugins.workflow.cps.CpsStepContext.onFailure(CpsStepContext.java:325)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$5.onSuccess(WorkflowRun.java:300)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$5.onSuccess(WorkflowRun.java:296)
	at org.jenkinsci.plugins.workflow.support.concurrent.Futures$1.run(Futures.java:150)
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
	at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:170)
	at com.google.common.util.concurrent.SettableFuture.set(SettableFuture.java:53)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$5.onSuccess(CpsFlowExecution.java:702)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$5.onSuccess(CpsFlowExecution.java:689)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4$1.run(CpsFlowExecution.java:626)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:32)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.jenkinsci.plugins.workflow.steps.FlowInterruptedException
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.doTerm(WorkflowRun.java:295)
	at ...

      So the program state seems to be somehow inconsistent as well; perhaps sh did complete (it is not shown as in progress in flowGraphTable).

      Seems that the virtual thread dump needs some kind of fix TBD to better report the real state of a problematic program.

          [JENKINS-37730] DurableTaskStep.Execution hanging after process is dead

          Jesse Glick created issue -
          Jesse Glick made changes -
          Epic Link New: JENKINS-35399 [ 171192 ]
          Jesse Glick made changes -
          Assignee Original: Jesse Glick [ jglick ] New: Kohsuke Kawaguchi [ kohsuke ]
          Andrew Bayer made changes -
          Component/s New: pipeline [ 21692 ]
          Andrew Bayer made changes -
          Component/s Original: workflow-plugin [ 18820 ]

          Edward Easton added a comment - - edited

          Hi there,
          Just letting you know this is hitting me in the wild, thanks for raising this! I get the exact same traceback above in the server logs.

          A little background - I've built a moderately complex CPSWorkflowLib for building in-house Python projects with a DSL. The DSL specifies named test stages with simple closures to specify the stage body, and a number of workers to spread the stage over.
          This has been working well until I updated to the latest pipeline plugin versions a few days ago; now whenever one of the test stages raises a non-zero exit code in a `sh` step it will hang the build.

          I'm trying to come up with a minimal test case that reproduces the problem, it's tricky as there's a lot of setup code in the DSL library that might be affecting things. I'll post it when I get something workable.

          Versions: Jenkins 2.7.4, workflow-cps-global-lib 2.3
          I might also add this isn't 'minor' ! I can no longer use Pipeline until I find a workaround for this :/

          Edward Easton added a comment - - edited Hi there, Just letting you know this is hitting me in the wild, thanks for raising this! I get the exact same traceback above in the server logs. A little background - I've built a moderately complex CPSWorkflowLib for building in-house Python projects with a DSL. The DSL specifies named test stages with simple closures to specify the stage body, and a number of workers to spread the stage over. This has been working well until I updated to the latest pipeline plugin versions a few days ago; now whenever one of the test stages raises a non-zero exit code in a `sh` step it will hang the build. I'm trying to come up with a minimal test case that reproduces the problem, it's tricky as there's a lot of setup code in the DSL library that might be affecting things. I'll post it when I get something workable. Versions: Jenkins 2.7.4, workflow-cps-global-lib 2.3 I might also add this isn't 'minor' ! I can no longer use Pipeline until I find a workaround for this :/

          Edward Easton added a comment -

          Hi, I traced one of the hangs to the problem mentioned here: JENKINS-38566

          Edward Easton added a comment - Hi, I traced one of the hangs to the problem mentioned here: JENKINS-38566
          Ryan Campbell made changes -
          Labels Original: robustness New: pipeline-hangs robustness
          Ryan Campbell made changes -
          Priority Original: Minor [ 4 ] New: Critical [ 2 ]

          Ryan Campbell added a comment -

          Marking this as critical since it results in unkillable job on the agent, making the agent unusable. Is there a workaround?

          Ryan Campbell added a comment - Marking this as critical since it results in unkillable job on the agent, making the agent unusable. Is there a workaround?

            jglick Jesse Glick
            jglick Jesse Glick
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: