Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33761

Ability to disable Pipeline durability and "resume" build.

    • Blue Ocean 1.4 - beta 2, Pipeline - December

      Having some state being generated at the each node during execution, resuming builds after jenkins restarts or nodes reboots are just not feasible sometimes and can result in infinite hangs in some cases.  Also, providing durability results in extensive writes to disk that can bring performance crashing down. 

      It would be great to be able to specify that jobs don't resume upon interruptions, but rather just fail. This would increase the robustness of the system ideally, since upon nodes restarting, they quickly pick up jobs that tries to resume and hangs exhausting all available executors quickly.

      Implementation notes:

      • Requires a new OptionalJobProperty on the job, optionally a new BranchProperty in workflow-multibranch-plugin that echoes that same property
      • Needs some way to signal to storage (workflow-support) and execution (workflow-cps) that the pipeline is running with resume OFF to hint that they can use faster nondurable execution.

          [JENKINS-33761] Ability to disable Pipeline durability and "resume" build.

          Mike Kozell added a comment - - edited

          I would also like to see the ability to restart a Jenkins master without it restarting or resuming any pipeline builds. Before restarting Jenkins to change a startup parameter, I verified my Jenkins master server was idle with no jobs running on master or slave executors. After the restart I saw the following errors in the log file and an attempt was made to resume builds 38, 107, and 108 all which are weeks old. It appears one of these builds was originally hung on "java.lang.InterruptedException" and the other two were hung on "org.jenkinsci.plugins.workflow.steps.FlowInterruptedException". The information used in these builds is obsolete and I prefer a Jenkins master restart to not resume any builds. These builds did not resume properly and I had to force stop them anyways.

          Oct 31, 2017 10:17:51 PM org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem
          WARNING: Unexpected exception in CPS VM thread: CpsFlowExecution[Owner[JOBNAME/108:JOBNAME #108]]
          java.util.EmptyStackException
                  at java.util.Stack.peek(Stack.java:102)
                  at java.util.Stack.pop(Stack.java:84)
                  at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onProgramEnd(CpsFlowExecution.java:1026)
                  at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:350)
                  at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82)
                  at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:242)
                  at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:230)
                  at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                  at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
                  at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
                  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                  at java.lang.Thread.run(Thread.java:748)
          
          Oct 31, 2017 10:17:55 PM org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem
          WARNING: Unexpected exception in CPS VM thread: CpsFlowExecution[Owner[JOBNAME/107:JOBNAME #107]]
          java.util.EmptyStackException
                  at java.util.Stack.peek(Stack.java:102)
                  at java.util.Stack.pop(Stack.java:84)
                  at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onProgramEnd(CpsFlowExecution.java:1026)
                  at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:350)
                  at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82)
                  at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:242)
                  at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:230)
                  at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                  at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
                  at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
                  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                  at java.lang.Thread.run(Thread.java:748)
          
          Oct 31, 2017 10:18:02 PM org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem
          WARNING: Unexpected exception in CPS VM thread: CpsFlowExecution[Owner[JOBNAME/38:JOBNAME #38]]
          java.util.EmptyStackException
                  at java.util.Stack.peek(Stack.java:102)
                  at java.util.Stack.pop(Stack.java:84)
                  at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onProgramEnd(CpsFlowExecution.java:1026)
                  at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:350)
                  at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82)
                  at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:242)
                  at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:230)
                  at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                  at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
                  at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
                  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                  at java.lang.Thread.run(Thread.java:748)
          
          

          Mike Kozell added a comment - - edited I would also like to see the ability to restart a Jenkins master without it restarting or resuming any pipeline builds. Before restarting Jenkins to change a startup parameter, I verified my Jenkins master server was idle with no jobs running on master or slave executors. After the restart I saw the following errors in the log file and an attempt was made to resume builds 38, 107, and 108 all which are weeks old. It appears one of these builds was originally hung on "java.lang.InterruptedException" and the other two were hung on "org.jenkinsci.plugins.workflow.steps.FlowInterruptedException". The information used in these builds is obsolete and I prefer a Jenkins master restart to not resume any builds. These builds did not resume properly and I had to force stop them anyways. Oct 31, 2017 10:17:51 PM org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem WARNING: Unexpected exception in CPS VM thread: CpsFlowExecution[Owner[JOBNAME/108:JOBNAME #108]] java.util.EmptyStackException at java.util.Stack.peek(Stack.java:102) at java.util.Stack.pop(Stack.java:84) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onProgramEnd(CpsFlowExecution.java:1026) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:350) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:242) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:230) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748) Oct 31, 2017 10:17:55 PM org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem WARNING: Unexpected exception in CPS VM thread: CpsFlowExecution[Owner[JOBNAME/107:JOBNAME #107]] java.util.EmptyStackException at java.util.Stack.peek(Stack.java:102) at java.util.Stack.pop(Stack.java:84) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onProgramEnd(CpsFlowExecution.java:1026) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:350) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:242) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:230) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748) Oct 31, 2017 10:18:02 PM org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem WARNING: Unexpected exception in CPS VM thread: CpsFlowExecution[Owner[JOBNAME/38:JOBNAME #38]] java.util.EmptyStackException at java.util.Stack.peek(Stack.java:102) at java.util.Stack.pop(Stack.java:84) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onProgramEnd(CpsFlowExecution.java:1026) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:350) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:242) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:230) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748)

          Sam Van Oort added a comment -

          mkozell Have you tried the betas for the pipeline plugins that are currently in the experimental update center?  I'm fairly sure I fixed an error of this category when hardening the work in workflow-cps – this is also the same beta that provides the ability to prevent individual flows from resuming.  With a little work in the script console it should be possible to write a quick script to invoke that on all currently running builds. 

          Sam Van Oort added a comment - mkozell  Have you tried the betas for the pipeline plugins that are currently in the experimental update center?  I'm fairly sure I fixed an error of this category when hardening the work in workflow-cps – this is also the same beta that provides the ability to prevent individual flows from resuming.  With a little work in the script console it should be possible to write a quick script to invoke that on all currently running builds. 

          Code changed in jenkins
          User: Sam Van Oort
          Path:
          pom.xml
          src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowJob.java
          src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java
          src/main/java/org/jenkinsci/plugins/workflow/job/properties/DisableResumeJobProperty.java
          src/main/java/org/jenkinsci/plugins/workflow/job/properties/DurabilityHintJobProperty.java
          src/main/resources/org/jenkinsci/plugins/workflow/job/properties/DisableResumeJobProperty/config-details.jelly
          src/main/resources/org/jenkinsci/plugins/workflow/job/properties/DurabilityHintJobProperty/config-details.jelly
          src/main/resources/org/jenkinsci/plugins/workflow/job/properties/DurabilityHintJobProperty/help.html
          src/test/java/org/jenkinsci/plugins/workflow/job/MemoryCleanupTest.java
          src/test/java/org/jenkinsci/plugins/workflow/job/WorkflowRunRestartTest.java
          src/test/java/org/jenkinsci/plugins/workflow/job/WorkflowRunTest.java
          src/test/java/org/jenkinsci/plugins/workflow/job/properties/DurabilityHintJobPropertyTest.java
          http://jenkins-ci.org/commit/workflow-job-plugin/5d3b91a68514d74422cc4ec5bc67d99418d7962c
          Log:
          Merge pull request #75 from svanoort/disable-pipeline-resume-JENKINS-33761

          Provide job property for durability hints & add ability to disable pipeline resume JENKINS-33761

          Compare: https://github.com/jenkinsci/workflow-job-plugin/compare/2dfc94ac80bc...5d3b91a68514

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Sam Van Oort Path: pom.xml src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowJob.java src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java src/main/java/org/jenkinsci/plugins/workflow/job/properties/DisableResumeJobProperty.java src/main/java/org/jenkinsci/plugins/workflow/job/properties/DurabilityHintJobProperty.java src/main/resources/org/jenkinsci/plugins/workflow/job/properties/DisableResumeJobProperty/config-details.jelly src/main/resources/org/jenkinsci/plugins/workflow/job/properties/DurabilityHintJobProperty/config-details.jelly src/main/resources/org/jenkinsci/plugins/workflow/job/properties/DurabilityHintJobProperty/help.html src/test/java/org/jenkinsci/plugins/workflow/job/MemoryCleanupTest.java src/test/java/org/jenkinsci/plugins/workflow/job/WorkflowRunRestartTest.java src/test/java/org/jenkinsci/plugins/workflow/job/WorkflowRunTest.java src/test/java/org/jenkinsci/plugins/workflow/job/properties/DurabilityHintJobPropertyTest.java http://jenkins-ci.org/commit/workflow-job-plugin/5d3b91a68514d74422cc4ec5bc67d99418d7962c Log: Merge pull request #75 from svanoort/disable-pipeline-resume- JENKINS-33761 Provide job property for durability hints & add ability to disable pipeline resume JENKINS-33761 Compare: https://github.com/jenkinsci/workflow-job-plugin/compare/2dfc94ac80bc...5d3b91a68514

          Sam Van Oort added a comment -

          Released with... uh, well take a look at the Jenkins Pipeline Handbook entry on scaling pipeline for versions.

          Sam Van Oort added a comment - Released with... uh, well take a look at the Jenkins Pipeline Handbook entry on scaling pipeline for versions.

          Greg Smith added a comment -

          For those watching, found direct link Sam mentioned:

          https://jenkins.io/doc/book/pipeline/scaling-pipeline/

          Greg Smith added a comment - For those watching, found direct link Sam mentioned: https://jenkins.io/doc/book/pipeline/scaling-pipeline/

          Mike Kozell added a comment - - edited

          svanoort

          After upgrading Jenkins with the following, I was not able to reproduce the issue after a build timeout, cancelling a build, and restarting Jenkins in the middle of a build.

          Jenkins 2.89.4
          Pipeline 2.5
          Pipeline API 2.26
          Pipeline Nodes and Processes 2.19
          Pipeline Step API 2.14
          Scripts Security 1.41
          durabilityHint=PERFORMANCE_OPTIMIZED
          org.jenkinsci.plugins.workflow.job.properties.DisableResumeJobProperty
          Groovy Sandbox = disabled
          Java = 1.8.0_162

          Although my jobs correctly didn't resume after Jenkins restart, I did see the message below in the build logs.

          Resuming build at Sat Feb 24 06:38:10 UTC 2018 after Jenkins restart
           [Pipeline] End of Pipeline
           java.io.IOException: Cannot resume build – was not cleanly saved when Jenkins shut down.

          Mike Kozell added a comment - - edited svanoort After upgrading Jenkins with the following, I was not able to reproduce the issue after a build timeout, cancelling a build, and restarting Jenkins in the middle of a build. Jenkins 2.89.4 Pipeline 2.5 Pipeline API 2.26 Pipeline Nodes and Processes 2.19 Pipeline Step API 2.14 Scripts Security 1.41 durabilityHint=PERFORMANCE_OPTIMIZED org.jenkinsci.plugins.workflow.job.properties.DisableResumeJobProperty Groovy Sandbox = disabled Java = 1.8.0_162 Although my jobs correctly didn't resume after Jenkins restart, I did see the message below in the build logs. Resuming build at Sat Feb 24 06:38:10 UTC 2018 after Jenkins restart [Pipeline] End of Pipeline java.io.IOException: Cannot resume build – was not cleanly saved when Jenkins shut down.

          Roy Arnon added a comment -

          Hello,

          I am not sure this is related to this issue, but in our pipeline build job we recently added the disableResume step and it does not seem to work correctly:

          Jenkins 2.89.3
          Pipeline 2.5
          Pipeline API 2.27
          Pipeline Nodes and Processes 2.20
          Pipeline Step API 2.16
          Scripts Security 1.44
          durabilityHint=PERFORMANCE_OPTIMIZED
          org.jenkinsci.plugins.workflow.job.properties.DisableResumeJobProperty
          Groovy Sandbox = disabled

           

          Creating placeholder flownodes because failed loading originals.
          Resuming build at Thu Aug 30 12:42:45 UTC 2018 after Jenkins restart
          [Bitbucket] Notifying pull request build result
          [Bitbucket] Build result notified
          [lockable-resources] released lock on [UNIT_TEST_RESOURCE_3]
          java.io.IOException: Tried to load head FlowNodes for execution Owner[Products.Pipeline/PR-5615/7:Products.Pipeline/PR-5615 #7] but FlowNode was not found in storage for head id:FlowNodeId 1:586
          	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
          	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
          	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:875)
          	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:745)
          	at hudson.model.RunMap.retrieve(RunMap.java:225)
          	at hudson.model.RunMap.retrieve(RunMap.java:57)
          	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:500)
          	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:482)
          	at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:380)
          	at hudson.model.RunMap.getById(RunMap.java:205)
          	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:1098)
          	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:1109)
          	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
          	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
          	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
          	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
          	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
          	at jenkins.model.Jenkins.<init>(Jenkins.java:974)
          	at hudson.model.Hudson.<init>(Hudson.java:86)
          	at hudson.model.Hudson.<init>(Hudson.java:82)
          	at hudson.WebAppMain$3.run(WebAppMain.java:233)
          Finished: SUCCESS

          This is an issue for us as the build was marked as SUCCESS in bitbucket, which allowed a user to merge a failing test into our release branch.

          The job was definitely running with resume disabled, as this was printed at start of job:

          Resume disabled by user, switching to high-performance, low-durability mode.

          Any ideas? 

          Roy Arnon added a comment - Hello, I am not sure this is related to this issue, but in our pipeline build job we recently added the disableResume step and it does not seem to work correctly: Jenkins 2.89.3 Pipeline 2.5 Pipeline API 2.27 Pipeline Nodes and Processes 2.20 Pipeline Step API 2.16 Scripts Security 1.44 durabilityHint=PERFORMANCE_OPTIMIZED org.jenkinsci.plugins.workflow.job.properties.DisableResumeJobProperty Groovy Sandbox = disabled   Creating placeholder flownodes because failed loading originals. Resuming build at Thu Aug 30 12:42:45 UTC 2018 after Jenkins restart [Bitbucket] Notifying pull request build result [Bitbucket] Build result notified [lockable-resources] released lock on [UNIT_TEST_RESOURCE_3] java.io.IOException: Tried to load head FlowNodes for execution Owner[Products.Pipeline/PR-5615/7:Products.Pipeline/PR-5615 #7] but FlowNode was not found in storage for head id:FlowNodeId 1:586 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715) at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:875) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:745) at hudson.model.RunMap.retrieve(RunMap.java:225) at hudson.model.RunMap.retrieve(RunMap.java:57) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:500) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:482) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:380) at hudson.model.RunMap.getById(RunMap.java:205) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:1098) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:1109) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178) at jenkins.model.Jenkins.<init>(Jenkins.java:974) at hudson.model.Hudson.<init>(Hudson.java:86) at hudson.model.Hudson.<init>(Hudson.java:82) at hudson.WebAppMain$3.run(WebAppMain.java:233) Finished: SUCCESS This is an issue for us as the build was marked as SUCCESS in bitbucket, which allowed a user to merge a failing test into our release branch. The job was definitely running with resume disabled, as this was printed at start of job: Resume disabled by user, switching to high-performance, low-durability mode. Any ideas? 

          We have seen the same thing. Resume definitely disabled and still causing hangs.

          Russell Gallop added a comment - We have seen the same thing. Resume definitely disabled and still causing hangs.

          Thread dump from a node this is happening on attached.

           

          Jenkins 2.107.3

          Pipeline 2.5

          Pipeline API 2.27

          Pipeline Nodes and Processes 2.19

          Pipeline Step API 2.15

          JENKINS-33671_thread_dump.txt

           

          Oddly, having killed the job from "Build Executor Status" the node is freed up but the job seems to still think it is running:

          [Pipeline] {

          Creating placeholder flownodes because failed loading originals.

          Resuming build at Thu Sep 20 11:47:46 BST 2018 after Jenkins restart

          [Pipeline] End of Pipeline

          Finished: FAILURE

          <spinning indicator>

           

          The next thing that this would be doing would be retry { checkout git ... }

           

           

           

          Russell Gallop added a comment - Thread dump from a node this is happening on attached.   Jenkins 2.107.3 Pipeline 2.5 Pipeline API 2.27 Pipeline Nodes and Processes 2.19 Pipeline Step API 2.15 JENKINS-33671_thread_dump.txt   Oddly, having killed the job from "Build Executor Status" the node is freed up but the job seems to still think it is running: [Pipeline] { Creating placeholder flownodes because failed loading originals. Resuming build at Thu Sep 20 11:47:46 BST 2018 after Jenkins restart [Pipeline] End of Pipeline Finished: FAILURE <spinning indicator>   The next thing that this would be doing would be retry { checkout git ... }      

          Nick Jones added a comment - - edited

          I've just experienced the "Creating placeholder flownodes because failed loading originals." error with this stack trace on a Jenkins system running workflow-job 2.25 and workflow-cps 2.64:

          java.io.IOException: Tried to load head FlowNodes for execution Owner[Redacted/dev/3:Redacted/dev #3] but FlowNode was not found in storage for head id:FlowNodeId 1:59
          	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
          	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
          	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:875)
          	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:745)
          	at hudson.model.RunMap.retrieve(RunMap.java:225)
          	at hudson.model.RunMap.retrieve(RunMap.java:57)
          	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
          	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
          	at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
          	at hudson.model.RunMap.getById(RunMap.java:205)
          	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:1112)
          	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:1123)
          	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
          	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
          	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
          	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
          	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
          	at jenkins.model.Jenkins.<init>(Jenkins.java:989)
          	at hudson.model.Hudson.<init>(Hudson.java:85)
          	at hudson.model.Hudson.<init>(Hudson.java:81)
          	at hudson.WebAppMain$3.run(WebAppMain.java:233)
          Finished: FAILURE
          

          Restarting the job manually appears to have resolved it, but is there additional information I can provide to troubleshoot what might have caused this? Or is this a different issue than what's discussed here?

          Edit: I should add that workflow-cps was upgraded from 2.63 to 2.64 between the last successful job and the one that failed with the stack trace above. Workflow-job was not changed.

          Nick Jones added a comment - - edited I've just experienced the "Creating placeholder flownodes because failed loading originals." error with this stack trace on a Jenkins system running workflow-job 2.25 and workflow-cps 2.64: java.io.IOException: Tried to load head FlowNodes for execution Owner[Redacted/dev/3:Redacted/dev #3] but FlowNode was not found in storage for head id:FlowNodeId 1:59 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715) at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:875) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:745) at hudson.model.RunMap.retrieve(RunMap.java:225) at hudson.model.RunMap.retrieve(RunMap.java:57) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381) at hudson.model.RunMap.getById(RunMap.java:205) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:1112) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:1123) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178) at jenkins.model.Jenkins.<init>(Jenkins.java:989) at hudson.model.Hudson.<init>(Hudson.java:85) at hudson.model.Hudson.<init>(Hudson.java:81) at hudson.WebAppMain$3.run(WebAppMain.java:233) Finished: FAILURE Restarting the job manually appears to have resolved it, but is there additional information I can provide to troubleshoot what might have caused this? Or is this a different issue than what's discussed here? Edit: I should add that workflow-cps was upgraded from 2.63 to 2.64 between the last successful job and the one that failed with the stack trace above. Workflow-job was not changed.

            svanoort Sam Van Oort
            jtilander Jim Tilander
            Votes:
            47 Vote for this issue
            Watchers:
            55 Start watching this issue

              Created:
              Updated:
              Resolved: