-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
Jenkins ver. 2.138.2 Pipeline: Groovy 2.61
IMPORTANT: NOTE FROM A MAINTAINER:
STOP! YOUR STACK TRACE ALONE IS NOT GOING TO HELP SOLVE THIS!
(sorry to all caps but we're not going to make progress on this issue with commenters adding insufficient information)
Note from maintainer: We'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ($JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/) of the build that failed and upload it here along with the Jenkins system logs, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any messages in the Jenkins system logs that seem relevant, etc. In addition, please check service or other system level logs for Jenkins to see if there are any issues with Jenkins taking too long to shut down or anything like that. Thanks!
The main thing we are currently looking for is whether these messages are present in the Jenkins logs right before Jenkins shut down for the build which has the error:
- About to try to checkpoint the program for buildCpsFlowExecutionOwner[YourJobName/BuildNumber:YourJobName #BuildNumber]]
- Trying to save program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash
- Finished saving program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash
If these messages are not present, it means that Jenkins was unable to save the Pipeline, so the error is expected. If that is the case, fixing the issue probably requires changes to Jenkins packaging to configure longer service timeouts on shutdown, or totally changing how PERFORMANCE_OPTIMIZED works. If the messages are present, then something else is happening.
Exception:
Creating placeholder flownodes because failed loading originals. java.io.IOException: Tried to load head FlowNodes for execution Owner[Platform Service FBI Test/1605:Platform Service FBI Test #1605] but FlowNode was not found in storage for head id:FlowNodeId 1:17 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715) at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:659) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:525) at hudson.model.RunMap.retrieve(RunMap.java:225) at hudson.model.RunMap.retrieve(RunMap.java:57) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379) at hudson.model.RunMap.getById(RunMap.java:205) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:896) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:907) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178) at jenkins.model.Jenkins.<init>(Jenkins.java:975) at hudson.model.Hudson.<init>(Hudson.java:85) at hudson.model.Hudson.<init>(Hudson.java:81) at hudson.WebAppMain$3.run(WebAppMain.java:233) Finished: FAILURE
- is duplicated by
-
-
- Closed
-
-
-
- Closed
-
- links to
[JENKINS-55287] Pipeline: Failure to load flow node: FlowNode was not found in storage for head
Same issue here - a couple of builds of a multibranch pipeline that used to be completely stable and had not been altered in any way are now failing a few times a week with the following stacktrace:
Creating placeholder flownodes because failed loading originals. java.io.IOException: Tried to load head FlowNodes for execution Ownersonar-full-analysis-code-multibranch-pipeline/master/1602:sonar-full-analysis-code-multibranch-pipeline/master #1602 but FlowNode was not found in storage for head id:FlowNodeId 1:45 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715) at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:875) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:745) at hudson.model.RunMap.retrieve(RunMap.java:225) at hudson.model.RunMap.retrieve(RunMap.java:57) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:344) at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274) at jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:245) at org.jenkinsci.plugins.workflow.job.WorkflowJob.getLastBuild(WorkflowJob.java:242) at org.jenkinsci.plugins.workflow.job.WorkflowJob.getLastBuild(WorkflowJob.java:105) at hudson.model.Job.isLogUpdated(Job.java:310) at org.jenkinsci.plugins.workflow.job.WorkflowJob.getCauseOfBlockage(WorkflowJob.java:318) at hudson.model.Queue.getCauseOfBlockageForTask(Queue.java:1227) at hudson.model.Queue.getCauseOfBlockageForItem(Queue.java:1190) at hudson.model.Queue.maintain(Queue.java:1521) at com.cloudbees.opscenter.client.cloud.CloudImpl$3.run(CloudImpl.java:391) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Jenkins version 2.138.2.2
Yaroslava Romanets
added a comment - - edited Same issue here - a couple of builds of a multibranch pipeline that used to be completely stable and had not been altered in any way are now failing a few times a week with the following stacktrace:
Creating placeholder flownodes because failed loading originals.
java.io.IOException: Tried to load head FlowNodes for execution Ownersonar-full-analysis-code-multibranch-pipeline/master/1602:sonar-full-analysis-code-multibranch-pipeline/master #1602 but FlowNode was not found in storage for head id:FlowNodeId 1:45
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:875)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:745)
at hudson.model.RunMap.retrieve(RunMap.java:225)
at hudson.model.RunMap.retrieve(RunMap.java:57)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379)
at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:344)
at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
at jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:245)
at org.jenkinsci.plugins.workflow.job.WorkflowJob.getLastBuild(WorkflowJob.java:242)
at org.jenkinsci.plugins.workflow.job.WorkflowJob.getLastBuild(WorkflowJob.java:105)
at hudson.model.Job.isLogUpdated(Job.java:310)
at org.jenkinsci.plugins.workflow.job.WorkflowJob.getCauseOfBlockage(WorkflowJob.java:318)
at hudson.model.Queue.getCauseOfBlockageForTask(Queue.java:1227)
at hudson.model.Queue.getCauseOfBlockageForItem(Queue.java:1190)
at hudson.model.Queue.maintain(Queue.java:1521)
at com.cloudbees.opscenter.client.cloud.CloudImpl$3.run(CloudImpl.java:391)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang. Thread .run( Thread .java:748)
Jenkins version 2.138.2.2
Nils Schmidt
added a comment - It seems we are having a similar issue. Maybe the stacktrace helps. In our case the problem occurs roughly 1 out of 20 builds. And it always happens after we were waiting for a particular resource to be freed.
22:47:50 Trying to acquire lock on [someJar.zip] 22:47:50 Found 0 available resource(s). Waiting for correct amount: 1. 22:47:50 [someJar.zip] is locked, waiting... Creating placeholder flownodes because failed loading originals. java.io.IOException: Tried to load head FlowNodes for execution Owner[SomeJob/bugfix #254] but FlowNode was not found in storage for head id:FlowNodeId 1:133 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715) at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526) at hudson.model.RunMap.retrieve(RunMap.java:225) at hudson.model.RunMap.retrieve(RunMap.java:57) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381) at hudson.model.RunMap.getById(RunMap.java:205) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getFlowExecution(CpsStepContext.java:474) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getThreadGroupSynchronously(CpsStepContext.java:243) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getThreadSynchronously(CpsStepContext.java:237) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.doGet(CpsStepContext.java:298) at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:67) at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlockNames(LockableResourcesManager.java:396) at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlockNames(LockableResourcesManager.java:404) at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlock(LockableResourcesManager.java:347) at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlock(LockableResourcesManager.java:335) at org.jenkins.plugins.lockableresources.queue.LockRunListener.onCompleted(LockRunListener.java:95) at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:209) at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:586) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$900(WorkflowRun.java:133) at org.jenkinsci.plugins.workflow.job.WorkflowRun$FailOnLoadListener.lambda$onNewHead$0(WorkflowRun.java:983) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Finished: FAILURE
Nils Schmidt
added a comment - It seems we are having a similar issue. Maybe the stacktrace helps. In our case the problem occurs roughly 1 out of 20 builds. And it always happens after we were waiting for a particular resource to be freed.
22:47:50 Trying to acquire lock on [someJar.zip]
22:47:50 Found 0 available resource(s). Waiting for correct amount: 1.
22:47:50 [someJar.zip] is locked, waiting...
Creating placeholder flownodes because failed loading originals.
java.io.IOException: Tried to load head FlowNodes for execution Owner[SomeJob/bugfix #254] but FlowNode was not found in storage for head id:FlowNodeId 1:133
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526)
at hudson.model.RunMap.retrieve(RunMap.java:225)
at hudson.model.RunMap.retrieve(RunMap.java:57)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
at hudson.model.RunMap.getById(RunMap.java:205)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912)
at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getFlowExecution(CpsStepContext.java:474)
at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getThreadGroupSynchronously(CpsStepContext.java:243)
at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getThreadSynchronously(CpsStepContext.java:237)
at org.jenkinsci.plugins.workflow.cps.CpsStepContext.doGet(CpsStepContext.java:298)
at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:67)
at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlockNames(LockableResourcesManager.java:396)
at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlockNames(LockableResourcesManager.java:404)
at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlock(LockableResourcesManager.java:347)
at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlock(LockableResourcesManager.java:335)
at org.jenkins.plugins.lockableresources.queue.LockRunListener.onCompleted(LockRunListener.java:95)
at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:209)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:586)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$900(WorkflowRun.java:133)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$FailOnLoadListener.lambda$onNewHead$0(WorkflowRun.java:983)
at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang. Thread .run(Unknown Source)
Finished: FAILURE
I encountered this:
Creating placeholder flownodes because failed loading originals.
java.io.IOException: Tried to load head FlowNodes for execution Owner[DEV/Porx-07/82:DEV/Porx-07 #82] but FlowNode was not found in storage for head id:FlowNodeId 1:121
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526)
at hudson.model.RunMap.retrieve(RunMap.java:225)
at hudson.model.RunMap.retrieve(RunMap.java:57)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
at hudson.model.RunMap.getById(RunMap.java:205)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
at jenkins.model.Jenkins.<init>(Jenkins.java:989)
at hudson.model.Hudson.<init>(Hudson.java:85)
at hudson.model.Hudson.<init>(Hudson.java:81)
at hudson.WebAppMain$3.run(WebAppMain.java:233)
Finished: FAILURE
Craig Rodrigues
added a comment - - edited I encountered this:
Creating placeholder flownodes because failed loading originals.
java.io.IOException: Tried to load head FlowNodes for execution Owner[DEV/Porx-07/82:DEV/Porx-07 #82] but FlowNode was not found in storage for head id:FlowNodeId 1:121
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526)
at hudson.model.RunMap.retrieve(RunMap.java:225)
at hudson.model.RunMap.retrieve(RunMap.java:57)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
at hudson.model.RunMap.getById(RunMap.java:205)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
at jenkins.model.Jenkins.<init>(Jenkins.java:989)
at hudson.model.Hudson.<init>(Hudson.java:85)
at hudson.model.Hudson.<init>(Hudson.java:81)
at hudson.WebAppMain$3.run(WebAppMain.java:233)
Finished: FAILURE
Jason MCDev
added a comment - We also hit the exact same issue on a PR build today as the one posted by rodrigc
Jason MCDev
added a comment - We also hit the exact same issue on a PR build today as the one posted by rodrigc Craig Rodrigues
added a comment - @Jason MCDev I talked to @abayer on the #jenkins IRC channel. He said that he thinks that this problem occurs if we rebuild a pipeline from a specific stage.
During the build of a pipeline, an XML file is generated into the workspace which has flownode information.
@abayer suggested a workaround of deleting the build from the build history to remove the flownode XML. Then the pipeline should be run from the beginning (not a rebuild).
@abayer's suggestion doesn't match what I saw, since I did not rebuild from a stage, yet I saw the problem.
Craig Rodrigues
added a comment - @Jason MCDev I talked to @abayer on the #jenkins IRC channel. He said that he thinks that this problem occurs if we rebuild a pipeline from a specific stage.
During the build of a pipeline, an XML file is generated into the workspace which has flownode information.
@abayer suggested a workaround of deleting the build from the build history to remove the flownode XML. Then the pipeline should be run from the beginning (not a rebuild).
@abayer's suggestion doesn't match what I saw, since I did not rebuild from a stage, yet I saw the problem. Jason MCDev
added a comment - rodrigc We are always running full clean builds, so I concur that the suggestion does not fit our use case. It has really been causing us headaches for sure - random failures in release pipelines.
Jason MCDev
added a comment - rodrigc We are always running full clean builds, so I concur that the suggestion does not fit our use case. It has really been causing us headaches for sure - random failures in release pipelines. Jason MCDev
added a comment - Any movement here or additional information that I can provide? We are seeing this error multiple times a day.
Jason MCDev
added a comment - Any movement here or additional information that I can provide? We are seeing this error multiple times a day. 
Michael Sanders
added a comment - Hi there,
I'm also curious for whether there has been any movement with fixing this bug? We are encountering it with nightly scheduled builds, yet later in the day, the same build will succeed.
Jenkins 2.176.1 running on Windows, communicating with other Windows slave agents.
java.io.IOException: Tried to load head FlowNodes for execution Owner[BardsTale4/110:BardsTale4 #110] but FlowNode was not found in storage for head id:FlowNodeId 1:128 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716) at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526) at hudson.model.RunMap.retrieve(RunMap.java:225) at hudson.model.RunMap.retrieve(RunMap.java:57) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381) at hudson.model.RunMap.getById(RunMap.java:205) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178) at jenkins.model.Jenkins.<init>(Jenkins.java:995) at hudson.model.Hudson.<init>(Hudson.java:85) at hudson.model.Hudson.<init>(Hudson.java:81) at hudson.WebAppMain$3.run(WebAppMain.java:233) Finished: FAILURE
Michael Sanders
added a comment - Hi there,
I'm also curious for whether there has been any movement with fixing this bug? We are encountering it with nightly scheduled builds, yet later in the day, the same build will succeed.
Jenkins 2.176.1 running on Windows, communicating with other Windows slave agents.
java.io.IOException: Tried to load head FlowNodes for execution Owner[BardsTale4/110:BardsTale4 #110] but FlowNode was not found in storage for head id:FlowNodeId 1:128
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526)
at hudson.model.RunMap.retrieve(RunMap.java:225)
at hudson.model.RunMap.retrieve(RunMap.java:57)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
at hudson.model.RunMap.getById(RunMap.java:205)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
at jenkins.model.Jenkins.<init>(Jenkins.java:995)
at hudson.model.Hudson.<init>(Hudson.java:85)
at hudson.model.Hudson.<init>(Hudson.java:81)
at hudson.WebAppMain$3.run(WebAppMain.java:233)
Finished: FAILURE
I have also seen this occasionally on ci.jenkins.io, but have no idea about root cause. Does anyone have a minimal Jenkinsfile that can consistently reproduce the issue? If no one has that, then the next best thing would be a build folder for a build that failed with such an error. The relevant information to look for would be whether the flow node mentioned in the exception really was missing or if it is present on disk but could not be found or loaded for some other reason.
Marvin Grit
added a comment - After multiple stable nightly runs we suddenly got this error in our pipeline as well. Running on Jenkins 2.163.
Creating placeholder flownodes because failed loading originals. java.io.IOException: Tried to load head FlowNodes for execution Owner[Pipeline/93:Pipeline #93] but FlowNode was not found in storage for head id:FlowNodeId 1:201 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715) at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526) at hudson.model.RunMap.retrieve(RunMap.java:225) at hudson.model.RunMap.retrieve(RunMap.java:57) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381) at hudson.model.RunMap.getById(RunMap.java:205) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178) at jenkins.model.Jenkins.<init>(Jenkins.java:1000) at hudson.model.Hudson.<init>(Hudson.java:85) at hudson.model.Hudson.<init>(Hudson.java:81) at hudson.WebAppMain$3.run(WebAppMain.java:233)
In the log of; /log/all we also see this:
Jenkins Log aug 28, 2019 10:29:58 PM INFO jenkins.InitReactorRunner$1 onAttainedPrepared all plugins aug 28, 2019 10:29:58 PM INFO jenkins.InitReactorRunner$1 onAttainedStarted all plugins aug 28, 2019 10:30:07 PM INFO jenkins.InitReactorRunner$1 onAttainedAugmented all extensions aug 28, 2019 10:30:07 PM INFO jenkins.model.Jenkins setBuildsAndWorkspacesDirUsing non default builds directories: C:/Jenkins/${ITEM_FULLNAME}/builds. aug 28, 2019 10:30:07 PM INFO jenkins.model.Jenkins setBuildsAndWorkspacesDirUsing non default workspaces directories: C:/Jenkins/${ITEM_FULLNAME}/workspace. aug 28, 2019 10:30:08 PM INFO hudson.plugins.project_inheritance.projects.actions.service.CachingInheritanceListService Service CachingInheritanceListService instantiated aug 28, 2019 10:30:08 PM INFO hudson.plugins.project_inheritance.projects.actions.InheritanceListActionFactory Extension InheritanceListActionFactory instantiated aug 28, 2019 10:30:17 PM INFO jenkins.InitReactorRunner$1 onAttainedLoaded all jobs aug 28, 2019 10:30:17 PM INFO hudson.model.AsyncPeriodicWork$1 runStarted Download metadata aug 28, 2019 10:30:17 PM INFO hudson.util.Retrier startAttempt #1 to do the action check updates server aug 28, 2019 10:30:18 PM INFO org.springframework.context.support.AbstractApplicationContext prepareRefreshRefreshing org.springframework.web.context.support.StaticWebApplicationContext@1a8fabb: display name [Root WebApplicationContext]; startup date [Wed Aug 28 22:30:18 CEST 2019]; root of context hierarchy aug 28, 2019 10:30:18 PM INFO org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactoryBean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@1a8fabb]: org.springframework.beans.factory.support.DefaultListableBeanFactory@c17335 aug 28, 2019 10:30:18 PM INFO org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletonsPre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@c17335: defining beans [authenticationManager,activeDirectory]; root of factory hierarchy aug 28, 2019 10:30:18 PM INFO hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider Active Directory domain is DC=ksserver,DC=local aug 28, 2019 10:30:19 PM INFO org.springframework.context.support.AbstractApplicationContext prepareRefreshRefreshing org.springframework.web.context.support.StaticWebApplicationContext@1141fc2: display name [Root WebApplicationContext]; startup date [Wed Aug 28 22:30:19 CEST 2019]; root of context hierarchy aug 28, 2019 10:30:19 PM INFO org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactoryBean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@1141fc2]: org.springframework.beans.factory.support.DefaultListableBeanFactory@13ffd5c aug 28, 2019 10:30:19 PM INFO org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletonsPre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@13ffd5c: defining beans [filter,legacy]; root of factory hierarchy aug 28, 2019 10:30:19 PM INFO jenkins.InitReactorRunner$1 onAttainedCompleted initialization aug 28, 2019 10:30:26 PM INFO hudson.model.UpdateSite updateDataObtained the latest update center data file for UpdateSource default aug 28, 2019 10:30:31 PM WARNING jenkins.branch.WorkspaceLocatorImpl getWorkspaceRootJENKINS-2111 path sanitization ineffective when using legacy Workspace Root Directory ‘C:/Jenkins/${ITEM_FULLNAME}/workspace’; switch to ‘${JENKINS_HOME}/workspace/${ITEM_FULL_NAME}’ as in JENKINS-8446 / JENKINS-21942 aug 28, 2019 10:30:31 PM WARNING org.jenkinsci.plugins.workflow.cps.CpsFlowExecution onLoadError initializing storage and loading nodes, will try to create placeholders for: CpsFlowExecution[Owner[Pipeline/93:Pipeline #93]] java.io.IOException: Tried to load head FlowNodes for execution Owner[Pipeline/93:Pipeline #93] but FlowNode was not found in storage for head id:FlowNodeId 1:201 at <same stacktrace as above> aug 28, 2019 10:30:31 PM INFO org.jenkinsci.plugins.workflow.cps.CpsFlowExecution createPlaceholderNodesCreating placeholder flownodes for execution: CpsFlowExecution[Owner[Pipeline/93:Pipeline #93]] aug 28, 2019 10:30:31 PM INFO hudson.model.DownloadService$Downloadable loadObtained the updated data file for hudson.tasks.Maven.MavenInstaller aug 28, 2019 10:30:31 PM WARNING org.jenkinsci.plugins.workflow.job.WorkflowRun getExecutionFound incomplete build with completed execution - display name: Pipeline #93
Marvin Grit
added a comment - After multiple stable nightly runs we suddenly got this error in our pipeline as well. Running on Jenkins 2.163.
Creating placeholder flownodes because failed loading originals.
java.io.IOException: Tried to load head FlowNodes for execution Owner[Pipeline/93:Pipeline #93] but FlowNode was not found in storage for head id:FlowNodeId 1:201
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526)
at hudson.model.RunMap.retrieve(RunMap.java:225)
at hudson.model.RunMap.retrieve(RunMap.java:57)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
at hudson.model.RunMap.getById(RunMap.java:205)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
at jenkins.model.Jenkins.<init>(Jenkins.java:1000)
at hudson.model.Hudson.<init>(Hudson.java:85)
at hudson.model.Hudson.<init>(Hudson.java:81)
at hudson.WebAppMain$3.run(WebAppMain.java:233)
In the log of; /log/all we also see this:
Jenkins Log
aug 28, 2019 10:29:58 PM INFO jenkins.InitReactorRunner$1 onAttainedPrepared all plugins
aug 28, 2019 10:29:58 PM INFO jenkins.InitReactorRunner$1 onAttainedStarted all plugins
aug 28, 2019 10:30:07 PM INFO jenkins.InitReactorRunner$1 onAttainedAugmented all extensions
aug 28, 2019 10:30:07 PM INFO jenkins.model.Jenkins setBuildsAndWorkspacesDirUsing non default builds directories: C:/Jenkins/${ITEM_FULLNAME}/builds.
aug 28, 2019 10:30:07 PM INFO jenkins.model.Jenkins setBuildsAndWorkspacesDirUsing non default workspaces directories: C:/Jenkins/${ITEM_FULLNAME}/workspace.
aug 28, 2019 10:30:08 PM INFO hudson.plugins.project_inheritance.projects.actions.service.CachingInheritanceListService Service CachingInheritanceListService instantiated
aug 28, 2019 10:30:08 PM INFO hudson.plugins.project_inheritance.projects.actions.InheritanceListActionFactory Extension InheritanceListActionFactory instantiated
aug 28, 2019 10:30:17 PM INFO jenkins.InitReactorRunner$1 onAttainedLoaded all jobs
aug 28, 2019 10:30:17 PM INFO hudson.model.AsyncPeriodicWork$1 runStarted Download metadata
aug 28, 2019 10:30:17 PM INFO hudson.util.Retrier startAttempt #1 to do the action check updates server
aug 28, 2019 10:30:18 PM INFO org.springframework.context.support.AbstractApplicationContext prepareRefreshRefreshing org.springframework.web.context.support.StaticWebApplicationContext@1a8fabb: display name [Root WebApplicationContext]; startup date [Wed Aug 28 22:30:18 CEST 2019]; root of context hierarchy
aug 28, 2019 10:30:18 PM INFO org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactoryBean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@1a8fabb]: org.springframework.beans.factory.support.DefaultListableBeanFactory@c17335
aug 28, 2019 10:30:18 PM INFO org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletonsPre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@c17335: defining beans [authenticationManager,activeDirectory]; root of factory hierarchy
aug 28, 2019 10:30:18 PM INFO hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider Active Directory domain is DC=ksserver,DC=local
aug 28, 2019 10:30:19 PM INFO org.springframework.context.support.AbstractApplicationContext prepareRefreshRefreshing org.springframework.web.context.support.StaticWebApplicationContext@1141fc2: display name [Root WebApplicationContext]; startup date [Wed Aug 28 22:30:19 CEST 2019]; root of context hierarchy
aug 28, 2019 10:30:19 PM INFO org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactoryBean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@1141fc2]: org.springframework.beans.factory.support.DefaultListableBeanFactory@13ffd5c
aug 28, 2019 10:30:19 PM INFO org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletonsPre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@13ffd5c: defining beans [filter,legacy]; root of factory hierarchy
aug 28, 2019 10:30:19 PM INFO jenkins.InitReactorRunner$1 onAttainedCompleted initialization
aug 28, 2019 10:30:26 PM INFO hudson.model.UpdateSite updateDataObtained the latest update center data file for UpdateSource default
aug 28, 2019 10:30:31 PM WARNING jenkins.branch.WorkspaceLocatorImpl getWorkspaceRootJENKINS-2111 path sanitization ineffective when using legacy Workspace Root Directory ‘C:/Jenkins/${ITEM_FULLNAME}/workspace’; switch to ‘${JENKINS_HOME}/workspace/${ITEM_FULL_NAME}’ as in JENKINS-8446 / JENKINS-21942
aug 28, 2019 10:30:31 PM WARNING org.jenkinsci.plugins.workflow.cps.CpsFlowExecution onLoadError initializing storage and loading nodes, will try to create placeholders for : CpsFlowExecution[Owner[Pipeline/93:Pipeline #93]]
java.io.IOException: Tried to load head FlowNodes for execution Owner[Pipeline/93:Pipeline #93] but FlowNode was not found in storage for head id:FlowNodeId 1:201
at <same stacktrace as above>
aug 28, 2019 10:30:31 PM INFO org.jenkinsci.plugins.workflow.cps.CpsFlowExecution createPlaceholderNodesCreating placeholder flownodes for execution: CpsFlowExecution[Owner[Pipeline/93:Pipeline #93]]
aug 28, 2019 10:30:31 PM INFO hudson.model.DownloadService$Downloadable loadObtained the updated data file for hudson.tasks.Maven.MavenInstaller
aug 28, 2019 10:30:31 PM WARNING org.jenkinsci.plugins.workflow.job.WorkflowRun getExecutionFound incomplete build with completed execution - display name: Pipeline #93
We've faced this issue today. Our 300+ parallel pipeline builds have failed simultaneously due to the error in ticket's description. But I've found the following in Jenkins server's logs in the same time:
[207043.805s][warning][os,thread] Attempt to protect stack guard pages failed (0x00007f6d188c9000-0x00007f6d188cd000). [207043.805s][warning][os,thread] Attempt to deallocate stack guard pages failed. OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f6d187c8000, 16384, 0) failed; error='Not enough space' (errno=12) [207043.810s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached. [207043.811s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached. # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 16384 bytes for committing reserved memory. # An error report file with more information is saved as: # /tmp/hs_err_pid6.log
So I guess something is wrong with our settings (out of memory on server with 180GB of RAM? Our Jenkins instance's load isn't that much, we've run the same load on VM with 60GB of RAM, but with 20 GB of swap)
We've spot the same today, all our nightly pipelines break with this error:
04:02:41 04:02:40,990 INFO JettyHandler:43 - Got request: /status.xml. Sender is 10.1.53.1:40440 Creating placeholder flownodes because failed loading originals. java.io.IOException: Tried to load head FlowNodes for execution Owner[RT-Jobs/RT-Nightly/9:RT-Jobs/RT-Nightly #9] but FlowNode was not found in storage for head id:FlowNodeId 1:526 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716) at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:662) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526) at hudson.model.RunMap.retrieve(RunMap.java:225) at hudson.model.RunMap.retrieve(RunMap.java:57) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381) at hudson.model.RunMap.getById(RunMap.java:205) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:899) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:910) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178) at jenkins.model.Jenkins.<init>(Jenkins.java:995) at hudson.model.Hudson.<init>(Hudson.java:85) at hudson.model.Hudson.<init>(Hudson.java:81) at hudson.WebAppMain$3.run(WebAppMain.java:233) Finished: UNSTABLE
Quite annoying as it follows Murphy's Law: Occurs in the exact worst moment... :-/
I think this is happening due to our pipeline durability settings override - we've set it to MAX_PERFORMANCE. Therefore all pipelines die when Jenkins master dies
Mihai Stoichitescu
added a comment - We are also getting this from time to time after updating Jenkins from 2.121.3 to 2.190.1
I've attached a list of our plugins used.
Mihai Stoichitescu
added a comment - We are also getting this from time to time after updating Jenkins from 2.121.3 to 2.190.1
I've attached a list of our plugins used. For everyone experiencing this issue: we'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ($JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/) of the build that failed and upload it here, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any exceptions or warnings in the Jenkins system logs that seem relevant, etc.
My best hypothesis is that there is a concurrency issue that causes the flow graph to not be saved in in some cases where it should be saved in all durability modes, but this is only noticed in performance-optimized mode because the Pipeline is not saved regularly like in other modes. One thing that we could do to investigate this hypothesis would be to instrument all uses of BulkChange in workflow-job and workflow-cps with optional logging to understand if bulk changes being aborted when a save is requested could be causing the problem.
One thing that we could do to investigate this hypothesis would be to instrument all uses of BulkChange in workflow-job and workflow-cps with optional logging to understand if bulk changes being aborted when a save is requested could be causing the problem.
Looking into this some more, I think flow nodes and program.dat are saved through more direct paths that appear to bypass BulkChange, so that might not help. Maybe the implementations of FlowNodeStorage need some synchronization? That seems like a decent place to start.
I filed https://github.com/jenkinsci/workflow-support-plugin/pull/104 based on my above hypothesis. If anyone is running into this issue consistently, I can create a build of the plugin based on that PR to see if it helps. I have not been able to reproduce the issue on demand, so I don't have a good way to validate whether that PR actually helps or not.
Here is the SNAPSHOT build of workflow-support 3.5 with my PR in case anyone would like to try it out. If you do try out that build, or even if you don't try it, if you still run into this issue, please upload a build folder of the build that had the issue as described in my note in the issue description here.
Craig Rodrigues
added a comment - dnusbaum If you have tested this a bit, can you merge this and deploy a new workflow-support-plugin?
I keep hitting this at intermittent times, but I can never fully reproduce it.
Unfortunately, this keeps hitting me right in the middle of when I am doing important stuff, and I don't have time to debug this. 
Craig Rodrigues
added a comment - dnusbaum If you have tested this a bit, can you merge this and deploy a new workflow-support-plugin?
I keep hitting this at intermittent times, but I can never fully reproduce it.
Unfortunately, this keeps hitting me right in the middle of when I am doing important stuff, and I don't have time to debug this. 
Henrik Koren
added a comment - 48064.tar.gz
I have the same issues
I have uploaded the entire build from my Master
Creating placeholder flownodes because failed loading originals. java.io.IOException: Tried to load head FlowNodes for execution Owner[Utils/CreateNodePipeline/48064:Utils/CreateNodePipeline #48064] but FlowNode was not found in storage for head id:FlowNodeId 1:20 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716) at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526) at hudson.model.RunMap.retrieve(RunMap.java:225) at hudson.model.RunMap.retrieve(RunMap.java:57) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381) at hudson.model.RunMap.getById(RunMap.java:205) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178) at jenkins.model.Jenkins.<init>(Jenkins.java:999) at hudson.model.Hudson.<init>(Hudson.java:85) at hudson.model.Hudson.<init>(Hudson.java:81) at hudson.WebAppMain$3.run(WebAppMain.java:233) Finished: FAILURE
Henrik Koren
added a comment - 48064.tar.gz I have the same issues
I have uploaded the entire build from my Master
Creating placeholder flownodes because failed loading originals.
java.io.IOException: Tried to load head FlowNodes for execution Owner[Utils/CreateNodePipeline/48064:Utils/CreateNodePipeline #48064] but FlowNode was not found in storage for head id:FlowNodeId 1:20
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526)
at hudson.model.RunMap.retrieve(RunMap.java:225)
at hudson.model.RunMap.retrieve(RunMap.java:57)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
at hudson.model.RunMap.getById(RunMap.java:205)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
at jenkins.model.Jenkins.<init>(Jenkins.java:999)
at hudson.model.Hudson.<init>(Hudson.java:85)
at hudson.model.Hudson.<init>(Hudson.java:81)
at hudson.WebAppMain$3.run(WebAppMain.java:233)
Finished: FAILURE
I got feedback from some users who were able to test the PR that it did not fix the issue. I am waiting for a build folder from those users to confirm that they are seeing the same issue. I think the next step would be to make a new hypothesis as to the issue and submit a PR. If anyone has even a semi-consistent way to reproduce the issue, it would make it much more likely that we could fix it.
henrikk Thanks!
rodrigc No, see above.
Senthilkumar Palanisamy
added a comment - Hi,
We are facing this issue frequently is there any workaround or any fix.
Our Jenkins version 2.204.1
Senthilkumar Palanisamy
added a comment - Hi,
We are facing this issue frequently is there any workaround or any fix.
Our Jenkins version 2.204.1 I'm facing this issue on https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fldap-plugin/detail/PR-40/4/pipeline/ for this PR: https://github.com/jenkinsci/ldap-plugin/pull/40
I guess olamy can give you more information dnusbaum. The log shows:
[2020-03-24T12:26:50.886Z] java.io.IOException: Tried to load head FlowNodes for execution Owner[Plugins/ldap-plugin/PR-40/4:Plugins/ldap-plugin/PR-40 #4] but FlowNode was not found in storage for head id:FlowNodeId 1:76 [2020-03-24T12:26:50.886Z] at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679) [2020-03-24T12:26:50.886Z] at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716) [2020-03-24T12:26:50.886Z] at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:672) [2020-03-24T12:26:50.886Z] at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:533) [2020-03-24T12:26:50.886Z] at hudson.model.RunMap.retrieve(RunMap.java:225) [2020-03-24T12:26:50.886Z] at hudson.model.RunMap.retrieve(RunMap.java:57) [2020-03-24T12:26:50.886Z] at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501) [2020-03-24T12:26:50.886Z] at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483) [2020-03-24T12:26:50.886Z] at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381) [2020-03-24T12:26:50.886Z] at hudson.model.RunMap.getById(RunMap.java:205) [2020-03-24T12:26:50.886Z] at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:921) [2020-03-24T12:26:50.886Z] at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:932) [2020-03-24T12:26:50.886Z] at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getExecution(CpsStepContext.java:217) [2020-03-24T12:26:50.886Z] at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getThreadGroupSynchronously(CpsStepContext.java:242) [2020-03-24T12:26:50.886Z] at org.jenkinsci.plugins.workflow.cps.CpsStepContext.access$000(CpsStepContext.java:97) [2020-03-24T12:26:50.886Z] at org.jenkinsci.plugins.workflow.cps.CpsStepContext$1.call(CpsStepContext.java:263) [2020-03-24T12:26:50.886Z] at org.jenkinsci.plugins.workflow.cps.CpsStepContext$1.call(CpsStepContext.java:261) [2020-03-24T12:26:50.886Z] at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) [2020-03-24T12:26:50.886Z] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [2020-03-24T12:26:50.886Z] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [2020-03-24T12:26:50.886Z] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [2020-03-24T12:26:50.886Z] at java.base/java.lang.Thread.run(Thread.java:834) [2020-03-24T12:26:50.886Z] Finished: FAILURE
IMPORTANT: NOTE FROM A MAINTAINER:
STOP! YOUR STACK TRACE ALONE IS NOT GOING TO HELP SOLVE THIS!
(sorry to all caps but we're not going to make progress on this issue with commenters adding insufficient information)
We'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ($JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/) of the build that failed and upload it here, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any exceptions or warnings in the Jenkins system logs that seem relevant, etc. Thanks!
mramonleon
**Sorry, I don't think we can get that information from ci.jenkins.io - the infra team is overloaded and we need the zip of the build folder immediately (or very soon) after the error.
bitwiseman I think a lot of pressure is being put on the end user to perform weird steps in order to help debug this problem. This is very hard to do. Since the problem
is intermittent, it is hard to catch.
Why don't you modify the pipeline plugin so at the point that the pipeline tries to throw an exception with message but FlowNode was not found in storage for head, the plugin zips up the entire job directory, and puts it in some place like $TEMPDIR ?
You can release a version of the plugin with this change.
That way the end user doesn't have to perform a lot of weird steps, but they can simply retrieve a zip file with all the stuff the is needed for the pipeline plugin maintainer to diagnose and root cause the problem?
Craig Rodrigues
added a comment - - edited bitwiseman I think a lot of pressure is being put on the end user to perform weird steps in order to help debug this problem. This is very hard to do. Since the problem
is intermittent, it is hard to catch.
Why don't you modify the pipeline plugin so at the point that the pipeline tries to throw an exception with message but FlowNode was not found in storage for head , the plugin zips up the entire job directory, and puts it in some place like $TEMPDIR ?
You can release a version of the plugin with this change.
That way the end user doesn't have to perform a lot of weird steps, but they can simply retrieve a zip file with all the stuff the is needed for the pipeline plugin maintainer to diagnose and root cause the problem? I have three zipped build folders (all from around the same time, March 12th) but did not yet have the time to redact sensitive data. Do you have some hints what and where to look for (regarding sensitive data)?
I was looking into this issue recently and wanted to check whether the code that saves Pipelines using the PERFORMANCE_OPTIMIZED durability level when Jenkins is shutting down (CpsFlowExecution.suspendAll) is running successfully in these cases to make sure that the problem isn't something like the service configuration for Jenkins not giving it enough time to shut down.
For users seeing the problem, please check the Jenkins system logs for the shutdown that occurred before you saw this error. Do you see messages similar to these, in particular for the build that failed with the exception in the description of this ticket?
- About to try to checkpoint the program for buildCpsFlowExecution[Owner[YourJobName/BuildNumber:YourJobName #BuildNumber]]
- Trying to save program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash
- Finished saving program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash
Those messages should come in groups of three, in that order, and are logged at INFO level, and there should not be any WARNING messages interspersed between them or surrounding them. There should be a set of those messages for each Pipeline running using the PERFORMANCE_OPTIMIZED durability level when Jenkins shuts down. If you have configured Jenkins to run as a service, I would also check the service logs to see if Jenkins has been shutting down successfully or if there are any warnings/errors when Jenkins shuts down.
famod It depends on what your builds are doing and what you consider sensitive. The main files to check are build.xml, which contains a textual form of the Pipeline, and log, which contains the build logs. You might want to check for internal hostnames or IP addresses, unmasked passwords or API keys in your build log, etc.
dnusbaum I am in the process of redacting sensitive content. I will have to remove some files and folders. Is there anything useful for you in workspace-files?
Can I send you the archive in a less public way somehow?
famod You can email it to me: dnusbaum at cloudbees.com. And no, I don't need to see workspace files. The main thing I want to see is the Jenkins system logs and or/service logs around the time of the last Jenkins restart to see if the build in question was able to be saved before the shutdown or not.
Mark Hollingsworth
added a comment - I'm not able to provide data due to company rules, but as an additional datapoint, we experienced this when the job in question lost connection to the node, not from a jenkins restart.
[2020-07-24T13:47:01.622Z] Cannot contact someNodeNameHere: java.lang.InterruptedException Creating placeholder flownodes because failed loading originals. java.io.IOException: Tried to load head FlowNodes for execution Owner[SomeFolder/SomeJob/SomeBuild:SomeFolder/SomeJob #SomeBuild] but FlowNode was not found in storage for head id:FlowNodeId 1:1714
Mark Hollingsworth
added a comment - I'm not able to provide data due to company rules, but as an additional datapoint, we experienced this when the job in question lost connection to the node, not from a jenkins restart.
[2020-07-24T13:47:01.622Z] Cannot contact someNodeNameHere: java.lang.InterruptedException
Creating placeholder flownodes because failed loading originals.
java.io.IOException: Tried to load head FlowNodes for execution Owner[SomeFolder/SomeJob/SomeBuild:SomeFolder/SomeJob #SomeBuild] but FlowNode was not found in storage for head id:FlowNodeId 1:1714
mhollingsworthcs Thanks for the feedback. As far as I can tell though there is no way for the "Creating placeholder flownodes because failed loading originals" message to be printed to the build log unless the Pipeline is resuming, which should only happen when Jenkins starts (although bugs in historical builds that have a broken state could also cause them to attempt to resume when they shouldn't). The full stack trace would help clarify, if "jenkins.model.Jenkins.<init>" is part of the stack trace for the exception, then Jenkins is starting up.
As a general update, my current understanding of this problem based on the data I have received is that in most cases it happens for Pipelines using the PERFORMANCE_OPTIMIZED durability level when Jenkins crashes. The PERFORMANCE_OPTIMIZED durability level makes no guarantees that Pipelines will be resumable if Jenkins crashes, so this behavior is expected in that case. There is supposed to be a more user-friendly error message explaining that the Pipeline cannot be resumed for these reasons rather than just the raw error of what exactly kept the Pipeline from resuming, but that is broken because of JENKINS-53358.
I have a draft PR up to try to improve the messaging around this case, so that something like the following would be printed in these cases instead:
Unable to resume because the Pipeline is using the PERFORMANCE_OPTIMIZED durability level but was not saved before Jenkins stopped. Did Jenkins crash? See https://www.jenkins.io/doc/book/pipeline/scaling-pipeline/ for details about Pipeline durability levels.
If anyone has seen this issue with Pipelines that are not using the PERFORMANCE_OPTIMIZED durability level, or can show Jenkins system logs and service logs that show this issue occurring with a PERFORMANCE_OPTIMIZED Pipeline even with a normal Jenkins restart, with log messages as described in this comment showing that the Pipeline was persisted before shutdown, that would be very interesting and we should create new tickets to track those things because they would be distinct issues. For some of the stack traces here, it looks like there is a problem where CpsStepContext.isReady is resulting in Pipelines being resumed, which is strange; I am not sure how to reproduce those issues and they probably need to be investigated separately.
Mark Hollingsworth
added a comment -
dnusbaum so we definitely see jenkins.model.Jenkins.<init> as part of the trace, but I'm almost 100% certain that Jenkins did not startup (I would have been paged very early in the morning if it had). What I can tell you is that we have been reproducing this issue by terminating machines.
[2020-07-24T13:47:01.622Z] Cannot contact someNode: java.lang.InterruptedException Creating placeholder flownodes because failed loading originals. java.io.IOException: Tried to load head FlowNodes for execution Owner[someFolder/someJob/buildNumber:someFolder/someJob #buildNumber] but FlowNode was not found in storage for head id:FlowNodeId 1:1714 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716) at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:680) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:539) at hudson.model.RunMap.retrieve(RunMap.java:225) at hudson.model.RunMap.retrieve(RunMap.java:57) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381) at hudson.model.RunMap.getById(RunMap.java:205) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:929) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:940) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178) at jenkins.model.Jenkins.<init>(Jenkins.java:1017) at hudson.model.Hudson.<init>(Hudson.java:85) at hudson.model.Hudson.<init>(Hudson.java:81) at hudson.WebAppMain$3.run(WebAppMain.java:262)
What I can tell you is that we have been able to reproduce this issue by terminating agent machines too early.
In this case our infrastructure killed the agent in job1 before the job finished completely and we repro the issue.
Job2 is the following job that did not get cut off early.
Job 1 16:23:36 [WS-CLEANUP] Deleting project workspace... 16:23:36 [WS-CLEANUP] Deferred wipeout is used... 16:23:36 [WS-CLEANUP] done [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // node [Pipeline] } ---------------------------- Job 2 16:16:42 [WS-CLEANUP] Deleting project workspace... 16:16:42 [WS-CLEANUP] Deferred wipeout is used... 16:16:42 [WS-CLEANUP] done [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // parallel [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Declarative: Post Actions) [Pipeline] node 16:16:42 Running on a node [Pipeline] { [Pipeline] echo 16:16:42 Reporting build status: UNSTABLE [Pipeline] notifyBitbucket notifying stuff [Pipeline] step 16:16:44 Notifying Bitbucket 16:16:44 Notified Bitbucket [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timestamps [Pipeline] } [Pipeline] // withEnv [Pipeline] End of Pipeline
Hope that helps!
Mark Hollingsworth
added a comment -
dnusbaum so we definitely see jenkins.model.Jenkins.<init> as part of the trace, but I'm almost 100% certain that Jenkins did not startup (I would have been paged very early in the morning if it had). What I can tell you is that we have been reproducing this issue by terminating machines.
[2020-07-24T13:47:01.622Z] Cannot contact someNode: java.lang.InterruptedException
Creating placeholder flownodes because failed loading originals.
java.io.IOException: Tried to load head FlowNodes for execution Owner[someFolder/someJob/buildNumber:someFolder/someJob #buildNumber] but FlowNode was not found in storage for head id:FlowNodeId 1:1714
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:680)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:539)
at hudson.model.RunMap.retrieve(RunMap.java:225)
at hudson.model.RunMap.retrieve(RunMap.java:57)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
at hudson.model.RunMap.getById(RunMap.java:205)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:929)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:940)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
at jenkins.model.Jenkins.<init>(Jenkins.java:1017)
at hudson.model.Hudson.<init>(Hudson.java:85)
at hudson.model.Hudson.<init>(Hudson.java:81)
at hudson.WebAppMain$3.run(WebAppMain.java:262)
What I can tell you is that we have been able to reproduce this issue by terminating agent machines too early.
In this case our infrastructure killed the agent in job1 before the job finished completely and we repro the issue.
Job2 is the following job that did not get cut off early.
Job 1
16:23:36 [WS-CLEANUP] Deleting project workspace...
16:23:36 [WS-CLEANUP] Deferred wipeout is used...
16:23:36 [WS-CLEANUP] done
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] }
----------------------------
Job 2
16:16:42 [WS-CLEANUP] Deleting project workspace...
16:16:42 [WS-CLEANUP] Deferred wipeout is used...
16:16:42 [WS-CLEANUP] done
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] node
16:16:42 Running on a node
[Pipeline] {
[Pipeline] echo
16:16:42 Reporting build status: UNSTABLE
[Pipeline] notifyBitbucket
notifying stuff
[Pipeline] step
16:16:44 Notifying Bitbucket
16:16:44 Notified Bitbucket
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] End of Pipeline
Hope that helps!
Simon Sudler
added a comment - I found the reason for the FlowNode errors on my system:
Sep 6 00:29:20 ship kernel: [16183195.286559] Out of memory: Kill process 27501 (java) score 122 or sacrifice child Sep 6 00:29:20 ship kernel: [16183195.292176] Killed process 27501 (java) total-vm:21995844kB, anon-rss:5537216kB, file-rss:0kB, shmem-rss:0kB Sep 6 00:29:20 ship kernel: [16183196.039063] oom_reaper: reaped process 27501 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
And it is very consistent. I checked the last 20 occurrences, every time the flow node error occurs, the OOM-killer does his work. Why the java process on the build-node requires around 20GB of memory is unclear, because the build itself does not have this huge memory requirements (maybe some other issue).
Since the OOM kills the java process, there is no proper feedback from the FlowNode... maybe some sanity check if the build client process should do the trick and produce some helpful error message.
I my case, more memory helped...
Simon Sudler
added a comment - I found the reason for the FlowNode errors on my system:
Sep 6 00:29:20 ship kernel: [16183195.286559] Out of memory: Kill process 27501 (java) score 122 or sacrifice child
Sep 6 00:29:20 ship kernel: [16183195.292176] Killed process 27501 (java) total-vm:21995844kB, anon-rss:5537216kB, file-rss:0kB, shmem-rss:0kB
Sep 6 00:29:20 ship kernel: [16183196.039063] oom_reaper: reaped process 27501 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
And it is very consistent. I checked the last 20 occurrences, every time the flow node error occurs, the OOM-killer does his work. Why the java process on the build-node requires around 20GB of memory is unclear, because the build itself does not have this huge memory requirements (maybe some other issue).
Since the OOM kills the java process, there is no proper feedback from the FlowNode... maybe some sanity check if the build client process should do the trick and produce some helpful error message.
I my case, more memory helped...
I am able to consistently get this error. I have attached the jobs/yeticore/branches/master/builds/2/workflow-fallback/flowNodeStore.xml . My interpretation of this file could be wrong but it appears it it searching for flow node 49, but all the file has is flow nodes 51 and the 52. This the file in workflow-fallback, the flow in workflow seems to have a single node with id 2.
Totally blocked at this time, so willing to investigate further
Update - I am running jenkins inside docker container and in Docker - Preferences - Resources - Memory , I increased memory from 2GB to 16GB (I restarted docker) and I was able to get a successful run. Will keep monitoring.
nishant dani
added a comment - - edited I am able to consistently get this error. I have attached the jobs/yeticore/branches/master/builds/2/workflow-fallback/flowNodeStore.xml . My interpretation of this file could be wrong but it appears it it searching for flow node 49, but all the file has is flow nodes 51 and the 52. This the file in workflow-fallback, the flow in workflow seems to have a single node with id 2.
Totally blocked at this time, so willing to investigate further
Update - I am running jenkins inside docker container and in Docker - Preferences - Resources - Memory , I increased memory from 2GB to 16GB (I restarted docker) and I was able to get a successful run. Will keep monitoring.
flowNodeStore.xml On my side adding more memory to my docker container runtime worked as well.
It would be helpful if this exception would be catched and a meaningful message would be shown to the users (low memory).
Lowering the priority as the behavior is expected and the fix is to improve the log message.
Liam Baker
added a comment - Hi, I am also experiencing this issue.
2022-09-19 12:04:16 | [GitCheckoutListener] -> No new commits found 2022-09-19 12:04:16 | [Pipeline] Start of Pipeline 2022-09-19 12:04:16 | [Pipeline] withEnv 2022-09-19 12:04:16 | [Pipeline] { 2022-09-19 12:04:16 | [Pipeline] stage 2022-09-19 12:04:16 | [Pipeline] { (Track (linux_x64)) 2022-09-19 12:04:17 | Creating placeholder flownodes because failed loading originals. 2022-09-19 12:04:17 | [Pipeline] Start of Pipeline 2022-09-19 12:04:17 | [Pipeline] End of Pipeline 2022-09-19 12:04:17 | [GitHub Checks] GitHub check (name: [redacted], status: completed) has been published. 2022-09-19 12:04:18 | 2022-09-19 12:04:18 | GitHub has been notified of this commit’s build result 2022-09-19 12:04:18 | 2022-09-19 12:04:18 | Finished: FAILURE
- After printing the message "Finished: FAILURE", The job will actually continue and will run on a node when one with a suitable tag becomes available.
- The build job will be available at the build URL if manually typed, however no links to the job will show in the Jenkins UI
- Subsequent builds of this pipeline will not run until the 'broken' build is deleted using the script console.
- The failure has occurred on many different pipelines, both custom defined and github multi-branch.
- Occurrence rate is an average of two to four per month.
I am currently looking for a reason/triggering condition, as this error does not correlate with high load on the Controller (CPU/RAM/DISK), errors on the host system log (journalctl/dmesg), or errors/warnings in the Jenkins System log.
It is most common for jobs that are "replayed" or manually triggered (using the "Build Now"/"Build with parameters" button), and may be related to manually triggering multiple jobs within a short period of time (~10s).
Liam Baker
added a comment - Hi, I am also experiencing this issue.
2022-09-19 12:04:16 | [GitCheckoutListener] -> No new commits found
2022-09-19 12:04:16 | [Pipeline] Start of Pipeline
2022-09-19 12:04:16 | [Pipeline] withEnv
2022-09-19 12:04:16 | [Pipeline] {
2022-09-19 12:04:16 | [Pipeline] stage
2022-09-19 12:04:16 | [Pipeline] { (Track (linux_x64))
2022-09-19 12:04:17 | Creating placeholder flownodes because failed loading originals.
2022-09-19 12:04:17 | [Pipeline] Start of Pipeline
2022-09-19 12:04:17 | [Pipeline] End of Pipeline
2022-09-19 12:04:17 | [GitHub Checks] GitHub check (name: [redacted], status: completed) has been published.
2022-09-19 12:04:18 |
2022-09-19 12:04:18 | GitHub has been notified of this commit’s build result
2022-09-19 12:04:18 |
2022-09-19 12:04:18 | Finished: FAILURE
After printing the message "Finished: FAILURE", The job will actually continue and will run on a node when one with a suitable tag becomes available.
The build job will be available at the build URL if manually typed, however no links to the job will show in the Jenkins UI
Subsequent builds of this pipeline will not run until the 'broken' build is deleted using the script console.
The failure has occurred on many different pipelines, both custom defined and github multi-branch.
Occurrence rate is an average of two to four per month.
I am currently looking for a reason/triggering condition, as this error does not correlate with high load on the Controller (CPU/RAM/DISK), errors on the host system log (journalctl/dmesg), or errors/warnings in the Jenkins System log.
It is most common for jobs that are "replayed" or manually triggered (using the "Build Now"/"Build with parameters" button), and may be related to manually triggering multiple jobs within a short period of time (~10s).
I'm also affected by this issue occasionally when I Safe Restart Jenkins, but I don't have anything useful to add, except that I have always been using MAX_SURVIVABILITY as build durability setting and yet this happens. Since no one mentioned they were using MAX_SURVIVABILITY here, I thought maybe making this comment would be good.
Just got this exception on a CD pipeline that has run many times without any issues previously. Not sure where to look for the cause.
Jenkins version 2.150.3