Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55287

Pipeline: Failure to load flow node: FlowNode was not found in storage for head

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Blocker
    • Resolution: Unresolved
    • Component/s: workflow-cps-plugin
    • Labels:
      None
    • Environment:
      Jenkins ver. 2.138.2 Pipeline: Groovy 2.61
    • Similar Issues:

      Description

      IMPORTANT: NOTE FROM A MAINTAINER:

      STOP! YOUR STACK TRACE ALONE IS NOT GOING TO HELP SOLVE THIS!

      (sorry to all caps but we're not going to make progress on this issue with commenters adding insufficient information)

      Note from maintainer: We'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ($JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/) of the build that failed and upload it here along with the Jenkins system logs, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any messages in the Jenkins system logs that seem relevant, etc. In addition, please check service or other system level logs for Jenkins to see if there are any issues with Jenkins taking too long to shut down or anything like that. Thanks!

      The main thing we are currently looking for is whether these messages are present in the Jenkins logs right before Jenkins shut down for the build which has the error:

      • About to try to checkpoint the program for buildCpsFlowExecutionOwner[YourJobName/BuildNumber:YourJobName #BuildNumber]]
      • Trying to save program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash
      • Finished saving program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash

      If these messages are not present, it means that Jenkins was unable to save the Pipeline, so the error is expected. If that is the case, fixing the issue probably requires changes to Jenkins packaging to configure longer service timeouts on shutdown, or totally changing how PERFORMANCE_OPTIMIZED works. If the messages are present, then something else is happening.

      Exception:

      Creating placeholder flownodes because failed loading originals.
      java.io.IOException: Tried to load head FlowNodes for execution Owner[Platform Service FBI Test/1605:Platform Service FBI Test #1605] but FlowNode was not found in storage for head id:FlowNodeId 1:17
       at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
       at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
       at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:659)
       at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:525)
       at hudson.model.RunMap.retrieve(RunMap.java:225)
       at hudson.model.RunMap.retrieve(RunMap.java:57)
       at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499)
       at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481)
       at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379)
       at hudson.model.RunMap.getById(RunMap.java:205)
       at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:896)
       at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:907)
       at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
       at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
       at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
       at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
       at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
       at jenkins.model.Jenkins.<init>(Jenkins.java:975)
       at hudson.model.Hudson.<init>(Hudson.java:85)
       at hudson.model.Hudson.<init>(Hudson.java:81)
       at hudson.WebAppMain$3.run(WebAppMain.java:233)
      Finished: FAILURE
      

        Attachments

          Issue Links

            Activity

            haorui658 Rui Hao created issue -
            haorui658 Rui Hao made changes -
            Field Original Value New Value
            Priority Critical [ 2 ] Blocker [ 1 ]
            Summary Groovy pipeline plugin issue Groovy pipeline plugin issue cause to jenkins process exit
            Hide
            voronaam Aleksey Vorona added a comment - - edited

            Just got this exception on a CD pipeline that has run many times without any issues previously. Not sure where to look for the cause.

            Creating placeholder flownodes because failed loading originals.
            [BFA] Scanning build for known causes...
            [BFA] No failure causes found
            [BFA] Done. 0s
            java.io.IOException: Tried to load head FlowNodes for execution Owner[ugp/xmdatabaseservicerelease/109:ugp/xmdatabaseservicerelease #109] but FlowNode was not found in storage for head id:FlowNodeId 1:3590
            	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
            	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526)
            	at hudson.model.RunMap.retrieve(RunMap.java:225)
            	at hudson.model.RunMap.retrieve(RunMap.java:57)
            	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
            	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
            	at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
            	at hudson.model.RunMap.getById(RunMap.java:205)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912)
            	at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getFlowExecution(CpsStepContext.java:474)
            	at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getThreadGroupSynchronously(CpsStepContext.java:243)
            	at org.jenkinsci.plugins.workflow.cps.CpsStepContext.access$000(CpsStepContext.java:97)
            	at org.jenkinsci.plugins.workflow.cps.CpsStepContext$1.call(CpsStepContext.java:264)
            	at org.jenkinsci.plugins.workflow.cps.CpsStepContext$1.call(CpsStepContext.java:262)
            	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
            	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            	at java.lang.Thread.run(Thread.java:748)
            

            Jenkins version 2.150.3

            Show
            voronaam Aleksey Vorona added a comment - - edited Just got this exception on a CD pipeline that has run many times without any issues previously. Not sure where to look for the cause. Creating placeholder flownodes because failed loading originals. [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s java.io.IOException: Tried to load head FlowNodes for execution Owner[ugp/xmdatabaseservicerelease/109:ugp/xmdatabaseservicerelease #109] but FlowNode was not found in storage for head id:FlowNodeId 1:3590 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715) at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526) at hudson.model.RunMap.retrieve(RunMap.java:225) at hudson.model.RunMap.retrieve(RunMap.java:57) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381) at hudson.model.RunMap.getById(RunMap.java:205) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getFlowExecution(CpsStepContext.java:474) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getThreadGroupSynchronously(CpsStepContext.java:243) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.access$000(CpsStepContext.java:97) at org.jenkinsci.plugins.workflow.cps.CpsStepContext$1.call(CpsStepContext.java:264) at org.jenkinsci.plugins.workflow.cps.CpsStepContext$1.call(CpsStepContext.java:262) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748) Jenkins version 2.150.3
            jsoref Josh Soref made changes -
            Link This issue is duplicated by JENKINS-54676 [ JENKINS-54676 ]
            jsoref Josh Soref made changes -
            Link This issue is duplicated by JENKINS-55170 [ JENKINS-55170 ]
            Hide
            yaroslavar Yaroslava Romanets added a comment - - edited

            Same issue here - a couple of builds of a multibranch pipeline that used to be completely stable and had not been altered in any way are now failing a few times a week with the following stacktrace:

            Creating placeholder flownodes because failed loading originals.
             java.io.IOException: Tried to load head FlowNodes for execution Ownersonar-full-analysis-code-multibranch-pipeline/master/1602:sonar-full-analysis-code-multibranch-pipeline/master #1602 but FlowNode was not found in storage for head id:FlowNodeId 1:45
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:875)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:745)
             at hudson.model.RunMap.retrieve(RunMap.java:225)
             at hudson.model.RunMap.retrieve(RunMap.java:57)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:344)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
             at jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:245)
             at org.jenkinsci.plugins.workflow.job.WorkflowJob.getLastBuild(WorkflowJob.java:242)
             at org.jenkinsci.plugins.workflow.job.WorkflowJob.getLastBuild(WorkflowJob.java:105)
             at hudson.model.Job.isLogUpdated(Job.java:310)
             at org.jenkinsci.plugins.workflow.job.WorkflowJob.getCauseOfBlockage(WorkflowJob.java:318)
             at hudson.model.Queue.getCauseOfBlockageForTask(Queue.java:1227)
             at hudson.model.Queue.getCauseOfBlockageForItem(Queue.java:1190)
             at hudson.model.Queue.maintain(Queue.java:1521)
             at com.cloudbees.opscenter.client.cloud.CloudImpl$3.run(CloudImpl.java:391)
             at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
             at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
             at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
             at java.util.concurrent.FutureTask.run(FutureTask.java:266)
             at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
             at java.lang.Thread.run(Thread.java:748)

            Jenkins version 2.138.2.2

             

             

            Show
            yaroslavar Yaroslava Romanets added a comment - - edited Same issue here - a couple of builds of a multibranch pipeline that used to be completely stable and had not been altered in any way are now failing a few times a week with the following stacktrace: Creating placeholder flownodes because failed loading originals. java.io.IOException: Tried to load head FlowNodes for execution Ownersonar-full-analysis-code-multibranch-pipeline/master/1602:sonar-full-analysis-code-multibranch-pipeline/master #1602 but FlowNode was not found in storage for head id:FlowNodeId 1:45 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715) at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:875) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:745) at hudson.model.RunMap.retrieve(RunMap.java:225) at hudson.model.RunMap.retrieve(RunMap.java:57) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:344) at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274) at jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:245) at org.jenkinsci.plugins.workflow.job.WorkflowJob.getLastBuild(WorkflowJob.java:242) at org.jenkinsci.plugins.workflow.job.WorkflowJob.getLastBuild(WorkflowJob.java:105) at hudson.model.Job.isLogUpdated(Job.java:310) at org.jenkinsci.plugins.workflow.job.WorkflowJob.getCauseOfBlockage(WorkflowJob.java:318) at hudson.model.Queue.getCauseOfBlockageForTask(Queue.java:1227) at hudson.model.Queue.getCauseOfBlockageForItem(Queue.java:1190) at hudson.model.Queue.maintain(Queue.java:1521) at com.cloudbees.opscenter.client.cloud.CloudImpl$3.run(CloudImpl.java:391) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748) Jenkins version 2.138.2.2    
            dnusbaum Devin Nusbaum made changes -
            Component/s workflow-cps-plugin [ 21713 ]
            Component/s groovy-plugin [ 15549 ]
            dnusbaum Devin Nusbaum made changes -
            Summary Groovy pipeline plugin issue cause to jenkins process exit Pipeline Groovy plugin issue cause to jenkins process exit
            dnusbaum Devin Nusbaum made changes -
            Summary Pipeline Groovy plugin issue cause to jenkins process exit Pipeline: Failure to load flow node: FlowNode was not found in storage for head
            Hide
            nilsthewizzard Nils Schmidt added a comment -

            It seems we are having a similar issue. Maybe the stacktrace helps. In our case the problem occurs roughly 1 out of 20 builds. And it always happens after we were waiting for a particular resource to be freed. 

            22:47:50 Trying to acquire lock on [someJar.zip]
            22:47:50 Found 0 available resource(s). Waiting for correct amount: 1.
            22:47:50 [someJar.zip] is locked, waiting...
            Creating placeholder flownodes because failed loading originals.
            java.io.IOException: Tried to load head FlowNodes for execution Owner[SomeJob/bugfix #254] but FlowNode was not found in storage for head id:FlowNodeId 1:133
            	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
            	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526)
            	at hudson.model.RunMap.retrieve(RunMap.java:225)
            	at hudson.model.RunMap.retrieve(RunMap.java:57)
            	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
            	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
            	at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
            	at hudson.model.RunMap.getById(RunMap.java:205)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912)
            	at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getFlowExecution(CpsStepContext.java:474)
            	at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getThreadGroupSynchronously(CpsStepContext.java:243)
            	at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getThreadSynchronously(CpsStepContext.java:237)
            	at org.jenkinsci.plugins.workflow.cps.CpsStepContext.doGet(CpsStepContext.java:298)
            	at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:67)
            	at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlockNames(LockableResourcesManager.java:396)
            	at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlockNames(LockableResourcesManager.java:404)
            	at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlock(LockableResourcesManager.java:347)
            	at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlock(LockableResourcesManager.java:335)
            	at org.jenkins.plugins.lockableresources.queue.LockRunListener.onCompleted(LockRunListener.java:95)
            	at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:209)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:586)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$900(WorkflowRun.java:133)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun$FailOnLoadListener.lambda$onNewHead$0(WorkflowRun.java:983)
            	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
            	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
            	at java.util.concurrent.FutureTask.run(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            	at java.lang.Thread.run(Unknown Source)
            Finished: FAILURE
            
            Show
            nilsthewizzard Nils Schmidt added a comment - It seems we are having a similar issue. Maybe the stacktrace helps. In our case the problem occurs roughly 1 out of 20 builds. And it always happens after we were waiting for a particular resource to be freed.  22:47:50 Trying to acquire lock on [someJar.zip] 22:47:50 Found 0 available resource(s). Waiting for correct amount: 1. 22:47:50 [someJar.zip] is locked, waiting... Creating placeholder flownodes because failed loading originals. java.io.IOException: Tried to load head FlowNodes for execution Owner[SomeJob/bugfix #254] but FlowNode was not found in storage for head id:FlowNodeId 1:133 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715) at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526) at hudson.model.RunMap.retrieve(RunMap.java:225) at hudson.model.RunMap.retrieve(RunMap.java:57) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381) at hudson.model.RunMap.getById(RunMap.java:205) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getFlowExecution(CpsStepContext.java:474) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getThreadGroupSynchronously(CpsStepContext.java:243) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getThreadSynchronously(CpsStepContext.java:237) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.doGet(CpsStepContext.java:298) at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:67) at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlockNames(LockableResourcesManager.java:396) at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlockNames(LockableResourcesManager.java:404) at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlock(LockableResourcesManager.java:347) at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlock(LockableResourcesManager.java:335) at org.jenkins.plugins.lockableresources.queue.LockRunListener.onCompleted(LockRunListener.java:95) at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:209) at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:586) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$900(WorkflowRun.java:133) at org.jenkinsci.plugins.workflow.job.WorkflowRun$FailOnLoadListener.lambda$onNewHead$0(WorkflowRun.java:983) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang. Thread .run(Unknown Source) Finished: FAILURE
            abayer Andrew Bayer made changes -
            Assignee vjuranek [ vjuranek ]
            Hide
            rodrigc Craig Rodrigues added a comment - - edited

            I encountered this:

            Creating placeholder flownodes because failed loading originals.
            java.io.IOException: Tried to load head FlowNodes for execution Owner[DEV/Porx-07/82:DEV/Porx-07 #82] but FlowNode was not found in storage for head id:FlowNodeId 1:121
                at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679)
                at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716)
                at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664)
                at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526)
                at hudson.model.RunMap.retrieve(RunMap.java:225)
                at hudson.model.RunMap.retrieve(RunMap.java:57)
                at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
                at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
                at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
                at hudson.model.RunMap.getById(RunMap.java:205)
                at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901)
                at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912)
                at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
                at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
                at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
                at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
                at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
                at jenkins.model.Jenkins.<init>(Jenkins.java:989)
                at hudson.model.Hudson.<init>(Hudson.java:85)
                at hudson.model.Hudson.<init>(Hudson.java:81)
                at hudson.WebAppMain$3.run(WebAppMain.java:233)
            Finished: FAILURE
            
            Show
            rodrigc Craig Rodrigues added a comment - - edited I encountered this: Creating placeholder flownodes because failed loading originals. java.io.IOException: Tried to load head FlowNodes for execution Owner[DEV/Porx-07/82:DEV/Porx-07 #82] but FlowNode was not found in storage for head id:FlowNodeId 1:121 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716) at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526) at hudson.model.RunMap.retrieve(RunMap.java:225) at hudson.model.RunMap.retrieve(RunMap.java:57) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381) at hudson.model.RunMap.getById(RunMap.java:205) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178) at jenkins.model.Jenkins.<init>(Jenkins.java:989) at hudson.model.Hudson.<init>(Hudson.java:85) at hudson.model.Hudson.<init>(Hudson.java:81) at hudson.WebAppMain$3.run(WebAppMain.java:233) Finished: FAILURE
            Hide
            jsnmcdev Jason MCDev added a comment -

            We also hit the exact same issue on a PR build today as the one posted by Craig Rodrigues

            Show
            jsnmcdev Jason MCDev added a comment - We also hit the exact same issue on a PR build today as the one posted by Craig Rodrigues
            Hide
            rodrigc Craig Rodrigues added a comment -

            @Jason MCDev I talked to @abayer on the #jenkins IRC channel. He said that he thinks that this problem occurs if we rebuild a pipeline from a specific stage.

            During the build of a pipeline, an XML file is generated into the workspace which has flownode information.

            @abayer suggested a workaround of deleting the build from the build history to remove the flownode XML. Then the pipeline should be run from the beginning (not a rebuild).

            @abayer's suggestion doesn't match what I saw, since I did not rebuild from a stage, yet I saw the problem.

            Show
            rodrigc Craig Rodrigues added a comment - @Jason MCDev I talked to @abayer on the #jenkins IRC channel. He said that he thinks that this problem occurs if we rebuild a pipeline from a specific stage. During the build of a pipeline, an XML file is generated into the workspace which has flownode information. @abayer suggested a workaround of deleting the build from the build history to remove the flownode XML. Then the pipeline should be run from the beginning (not a rebuild). @abayer's suggestion doesn't match what I saw, since I did not rebuild from a stage, yet I saw the problem.
            Hide
            jsnmcdev Jason MCDev added a comment -

            Craig Rodrigues We are always running full clean builds, so I concur that the suggestion does not fit our use case.  It has really been causing us headaches for sure - random failures in release pipelines.

            Show
            jsnmcdev Jason MCDev added a comment - Craig Rodrigues We are always running full clean builds, so I concur that the suggestion does not fit our use case.  It has really been causing us headaches for sure - random failures in release pipelines.
            Hide
            jsnmcdev Jason MCDev added a comment -

            Any movement here or additional information that I can provide? We are seeing this error multiple times a day.

            Show
            jsnmcdev Jason MCDev added a comment - Any movement here or additional information that I can provide? We are seeing this error multiple times a day.
            Hide
            urinalcake Michael Sanders added a comment -

            Hi there,

            I'm also curious for whether there has been any movement with fixing this bug? We are encountering it with nightly scheduled builds, yet later in the day, the same build will succeed.

            Jenkins 2.176.1 running on Windows, communicating with other Windows slave agents.

            java.io.IOException: Tried to load head FlowNodes for execution Owner[BardsTale4/110:BardsTale4 #110] but FlowNode was not found in storage for head id:FlowNodeId 1:128
            	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679)
            	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526)
            	at hudson.model.RunMap.retrieve(RunMap.java:225)
            	at hudson.model.RunMap.retrieve(RunMap.java:57)
            	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
            	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
            	at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
            	at hudson.model.RunMap.getById(RunMap.java:205)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912)
            	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
            	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
            	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
            	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
            	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
            	at jenkins.model.Jenkins.<init>(Jenkins.java:995)
            	at hudson.model.Hudson.<init>(Hudson.java:85)
            	at hudson.model.Hudson.<init>(Hudson.java:81)
            	at hudson.WebAppMain$3.run(WebAppMain.java:233)
            Finished: FAILURE
            
            Show
            urinalcake Michael Sanders added a comment - Hi there, I'm also curious for whether there has been any movement with fixing this bug? We are encountering it with nightly scheduled builds, yet later in the day, the same build will succeed. Jenkins 2.176.1 running on Windows, communicating with other Windows slave agents. java.io.IOException: Tried to load head FlowNodes for execution Owner[BardsTale4/110:BardsTale4 #110] but FlowNode was not found in storage for head id:FlowNodeId 1:128 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716) at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526) at hudson.model.RunMap.retrieve(RunMap.java:225) at hudson.model.RunMap.retrieve(RunMap.java:57) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381) at hudson.model.RunMap.getById(RunMap.java:205) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178) at jenkins.model.Jenkins.<init>(Jenkins.java:995) at hudson.model.Hudson.<init>(Hudson.java:85) at hudson.model.Hudson.<init>(Hudson.java:81) at hudson.WebAppMain$3.run(WebAppMain.java:233) Finished: FAILURE
            Hide
            dnusbaum Devin Nusbaum added a comment -

            I have also seen this occasionally on ci.jenkins.io, but have no idea about root cause. Does anyone have a minimal Jenkinsfile that can consistently reproduce the issue? If no one has that, then the next best thing would be a build folder for a build that failed with such an error. The relevant information to look for would be whether the flow node mentioned in the exception really was missing or if it is present on disk but could not be found or loaded for some other reason.

            Show
            dnusbaum Devin Nusbaum added a comment - I have also seen this occasionally on ci.jenkins.io, but have no idea about root cause. Does anyone have a minimal Jenkinsfile that can consistently reproduce the issue? If no one has that, then the next best thing would be a build folder for a build that failed with such an error. The relevant information to look for would be whether the flow node mentioned in the exception really was missing or if it is present on disk but could not be found or loaded for some other reason.
            Hide
            parvusm Marvin Grit added a comment -

            After multiple stable nightly runs we suddenly got this error in our pipeline as well. Running on Jenkins 2.163.

            Creating placeholder flownodes because failed loading originals.
            java.io.IOException: Tried to load head FlowNodes for execution Owner[Pipeline/93:Pipeline #93] but FlowNode was not found in storage for head id:FlowNodeId 1:201
            	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
            	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526)
            	at hudson.model.RunMap.retrieve(RunMap.java:225)
            	at hudson.model.RunMap.retrieve(RunMap.java:57)
            	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
            	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
            	at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
            	at hudson.model.RunMap.getById(RunMap.java:205)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912)
            	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
            	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
            	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
            	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
            	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
            	at jenkins.model.Jenkins.<init>(Jenkins.java:1000)
            	at hudson.model.Hudson.<init>(Hudson.java:85)
            	at hudson.model.Hudson.<init>(Hudson.java:81)
            	at hudson.WebAppMain$3.run(WebAppMain.java:233)
            
            

            In the log of; /log/all  we also see this: 

            Jenkins Log
            aug 28, 2019 10:29:58 PM INFO jenkins.InitReactorRunner$1 onAttainedPrepared all plugins
            aug 28, 2019 10:29:58 PM INFO jenkins.InitReactorRunner$1 onAttainedStarted all plugins
            aug 28, 2019 10:30:07 PM INFO jenkins.InitReactorRunner$1 onAttainedAugmented all extensions
            aug 28, 2019 10:30:07 PM INFO jenkins.model.Jenkins setBuildsAndWorkspacesDirUsing non default builds directories: C:/Jenkins/${ITEM_FULLNAME}/builds.
            aug 28, 2019 10:30:07 PM INFO jenkins.model.Jenkins setBuildsAndWorkspacesDirUsing non default workspaces directories: C:/Jenkins/${ITEM_FULLNAME}/workspace.
            aug 28, 2019 10:30:08 PM INFO hudson.plugins.project_inheritance.projects.actions.service.CachingInheritanceListService Service CachingInheritanceListService instantiated
            aug 28, 2019 10:30:08 PM INFO hudson.plugins.project_inheritance.projects.actions.InheritanceListActionFactory Extension InheritanceListActionFactory instantiated
            aug 28, 2019 10:30:17 PM INFO jenkins.InitReactorRunner$1 onAttainedLoaded all jobs
            aug 28, 2019 10:30:17 PM INFO hudson.model.AsyncPeriodicWork$1 runStarted Download metadata
            aug 28, 2019 10:30:17 PM INFO hudson.util.Retrier startAttempt #1 to do the action check updates server
            aug 28, 2019 10:30:18 PM INFO org.springframework.context.support.AbstractApplicationContext prepareRefreshRefreshing org.springframework.web.context.support.StaticWebApplicationContext@1a8fabb: display name [Root WebApplicationContext]; startup date [Wed Aug 28 22:30:18 CEST 2019]; root of context hierarchy
            aug 28, 2019 10:30:18 PM INFO org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactoryBean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@1a8fabb]: org.springframework.beans.factory.support.DefaultListableBeanFactory@c17335
            aug 28, 2019 10:30:18 PM INFO org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletonsPre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@c17335: defining beans [authenticationManager,activeDirectory]; root of factory hierarchy
            aug 28, 2019 10:30:18 PM INFO hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider Active Directory domain is DC=ksserver,DC=local
            aug 28, 2019 10:30:19 PM INFO org.springframework.context.support.AbstractApplicationContext prepareRefreshRefreshing org.springframework.web.context.support.StaticWebApplicationContext@1141fc2: display name [Root WebApplicationContext]; startup date [Wed Aug 28 22:30:19 CEST 2019]; root of context hierarchy
            aug 28, 2019 10:30:19 PM INFO org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactoryBean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@1141fc2]: org.springframework.beans.factory.support.DefaultListableBeanFactory@13ffd5c
            aug 28, 2019 10:30:19 PM INFO org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletonsPre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@13ffd5c: defining beans [filter,legacy]; root of factory hierarchy
            aug 28, 2019 10:30:19 PM INFO jenkins.InitReactorRunner$1 onAttainedCompleted initialization
            aug 28, 2019 10:30:26 PM INFO hudson.model.UpdateSite updateDataObtained the latest update center data file for UpdateSource default
            aug 28, 2019 10:30:31 PM WARNING jenkins.branch.WorkspaceLocatorImpl getWorkspaceRootJENKINS-2111 path sanitization ineffective when using legacy Workspace Root Directory ‘C:/Jenkins/${ITEM_FULLNAME}/workspace’; switch to ‘${JENKINS_HOME}/workspace/${ITEM_FULL_NAME}’ as in JENKINS-8446 / JENKINS-21942
            
            aug 28, 2019 10:30:31 PM WARNING org.jenkinsci.plugins.workflow.cps.CpsFlowExecution onLoadError initializing storage and loading nodes, will try to create placeholders for: CpsFlowExecution[Owner[Pipeline/93:Pipeline #93]]
            java.io.IOException: Tried to load head FlowNodes for execution Owner[Pipeline/93:Pipeline #93] but FlowNode was not found in storage for head id:FlowNodeId 1:201
            	at <same stacktrace as above>
            
            aug 28, 2019 10:30:31 PM INFO org.jenkinsci.plugins.workflow.cps.CpsFlowExecution createPlaceholderNodesCreating placeholder flownodes for execution: CpsFlowExecution[Owner[Pipeline/93:Pipeline #93]]
            aug 28, 2019 10:30:31 PM INFO hudson.model.DownloadService$Downloadable loadObtained the updated data file for hudson.tasks.Maven.MavenInstaller
            aug 28, 2019 10:30:31 PM WARNING org.jenkinsci.plugins.workflow.job.WorkflowRun getExecutionFound incomplete build with completed execution - display name: Pipeline #93
            

             

            Show
            parvusm Marvin Grit added a comment - After multiple stable nightly runs we suddenly got this error in our pipeline as well. Running on Jenkins 2.163. Creating placeholder flownodes because failed loading originals. java.io.IOException: Tried to load head FlowNodes for execution Owner[Pipeline/93:Pipeline #93] but FlowNode was not found in storage for head id:FlowNodeId 1:201 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715) at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526) at hudson.model.RunMap.retrieve(RunMap.java:225) at hudson.model.RunMap.retrieve(RunMap.java:57) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381) at hudson.model.RunMap.getById(RunMap.java:205) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178) at jenkins.model.Jenkins.<init>(Jenkins.java:1000) at hudson.model.Hudson.<init>(Hudson.java:85) at hudson.model.Hudson.<init>(Hudson.java:81) at hudson.WebAppMain$3.run(WebAppMain.java:233) In the log of; /log/all  we also see this:  Jenkins Log aug 28, 2019 10:29:58 PM INFO jenkins.InitReactorRunner$1 onAttainedPrepared all plugins aug 28, 2019 10:29:58 PM INFO jenkins.InitReactorRunner$1 onAttainedStarted all plugins aug 28, 2019 10:30:07 PM INFO jenkins.InitReactorRunner$1 onAttainedAugmented all extensions aug 28, 2019 10:30:07 PM INFO jenkins.model.Jenkins setBuildsAndWorkspacesDirUsing non default builds directories: C:/Jenkins/${ITEM_FULLNAME}/builds. aug 28, 2019 10:30:07 PM INFO jenkins.model.Jenkins setBuildsAndWorkspacesDirUsing non default workspaces directories: C:/Jenkins/${ITEM_FULLNAME}/workspace. aug 28, 2019 10:30:08 PM INFO hudson.plugins.project_inheritance.projects.actions.service.CachingInheritanceListService Service CachingInheritanceListService instantiated aug 28, 2019 10:30:08 PM INFO hudson.plugins.project_inheritance.projects.actions.InheritanceListActionFactory Extension InheritanceListActionFactory instantiated aug 28, 2019 10:30:17 PM INFO jenkins.InitReactorRunner$1 onAttainedLoaded all jobs aug 28, 2019 10:30:17 PM INFO hudson.model.AsyncPeriodicWork$1 runStarted Download metadata aug 28, 2019 10:30:17 PM INFO hudson.util.Retrier startAttempt #1 to do the action check updates server aug 28, 2019 10:30:18 PM INFO org.springframework.context.support.AbstractApplicationContext prepareRefreshRefreshing org.springframework.web.context.support.StaticWebApplicationContext@1a8fabb: display name [Root WebApplicationContext]; startup date [Wed Aug 28 22:30:18 CEST 2019]; root of context hierarchy aug 28, 2019 10:30:18 PM INFO org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactoryBean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@1a8fabb]: org.springframework.beans.factory.support.DefaultListableBeanFactory@c17335 aug 28, 2019 10:30:18 PM INFO org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletonsPre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@c17335: defining beans [authenticationManager,activeDirectory]; root of factory hierarchy aug 28, 2019 10:30:18 PM INFO hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider Active Directory domain is DC=ksserver,DC=local aug 28, 2019 10:30:19 PM INFO org.springframework.context.support.AbstractApplicationContext prepareRefreshRefreshing org.springframework.web.context.support.StaticWebApplicationContext@1141fc2: display name [Root WebApplicationContext]; startup date [Wed Aug 28 22:30:19 CEST 2019]; root of context hierarchy aug 28, 2019 10:30:19 PM INFO org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactoryBean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@1141fc2]: org.springframework.beans.factory.support.DefaultListableBeanFactory@13ffd5c aug 28, 2019 10:30:19 PM INFO org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletonsPre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@13ffd5c: defining beans [filter,legacy]; root of factory hierarchy aug 28, 2019 10:30:19 PM INFO jenkins.InitReactorRunner$1 onAttainedCompleted initialization aug 28, 2019 10:30:26 PM INFO hudson.model.UpdateSite updateDataObtained the latest update center data file for UpdateSource default aug 28, 2019 10:30:31 PM WARNING jenkins.branch.WorkspaceLocatorImpl getWorkspaceRootJENKINS-2111 path sanitization ineffective when using legacy Workspace Root Directory ‘C:/Jenkins/${ITEM_FULLNAME}/workspace’; switch to ‘${JENKINS_HOME}/workspace/${ITEM_FULL_NAME}’ as in JENKINS-8446 / JENKINS-21942 aug 28, 2019 10:30:31 PM WARNING org.jenkinsci.plugins.workflow.cps.CpsFlowExecution onLoadError initializing storage and loading nodes, will try to create placeholders for : CpsFlowExecution[Owner[Pipeline/93:Pipeline #93]] java.io.IOException: Tried to load head FlowNodes for execution Owner[Pipeline/93:Pipeline #93] but FlowNode was not found in storage for head id:FlowNodeId 1:201 at <same stacktrace as above> aug 28, 2019 10:30:31 PM INFO org.jenkinsci.plugins.workflow.cps.CpsFlowExecution createPlaceholderNodesCreating placeholder flownodes for execution: CpsFlowExecution[Owner[Pipeline/93:Pipeline #93]] aug 28, 2019 10:30:31 PM INFO hudson.model.DownloadService$Downloadable loadObtained the updated data file for hudson.tasks.Maven.MavenInstaller aug 28, 2019 10:30:31 PM WARNING org.jenkinsci.plugins.workflow.job.WorkflowRun getExecutionFound incomplete build with completed execution - display name: Pipeline #93  
            Hide
            oxygenxo Andrey Babushkin added a comment -

            We've faced this issue today. Our 300+ parallel pipeline builds have failed simultaneously due to the error in ticket's description. But I've found the following in Jenkins server's logs in the same time:

            [207043.805s][warning][os,thread] Attempt to protect stack guard pages failed (0x00007f6d188c9000-0x00007f6d188cd000).
            [207043.805s][warning][os,thread] Attempt to deallocate stack guard pages failed.
            OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f6d187c8000, 16384, 0) failed; error='Not enough space' (errno=12)
            [207043.810s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
            [207043.811s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
            #
            # There is insufficient memory for the Java Runtime Environment to continue.
            # Native memory allocation (mmap) failed to map 16384 bytes for committing reserved memory.
            # An error report file with more information is saved as:
            # /tmp/hs_err_pid6.log
             

            So I guess something is wrong with our settings (out of memory on server with 180GB of RAM? Our Jenkins instance's load isn't that much, we've run the same load on VM with 60GB of RAM, but with 20 GB of swap)

            Show
            oxygenxo Andrey Babushkin added a comment - We've faced this issue today. Our 300+ parallel pipeline builds have failed simultaneously due to the error in ticket's description. But I've found the following in Jenkins server's logs in the same time: [207043.805s][warning][os,thread] Attempt to protect stack guard pages failed (0x00007f6d188c9000-0x00007f6d188cd000). [207043.805s][warning][os,thread] Attempt to deallocate stack guard pages failed. OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f6d187c8000, 16384, 0) failed; error= 'Not enough space' (errno=12) [207043.810s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached. [207043.811s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached. # # There is insufficient memory for the Java Runtime Environment to continue . # Native memory allocation (mmap) failed to map 16384 bytes for committing reserved memory. # An error report file with more information is saved as: # /tmp/hs_err_pid6.log So I guess something is wrong with our settings (out of memory on server with 180GB of RAM? Our Jenkins instance's load isn't that much, we've run the same load on VM with 60GB of RAM, but with 20 GB of swap)
            Hide
            swf Yves Schumann added a comment -

            We've spot the same today, all our nightly pipelines break with this error:

            04:02:41  04:02:40,990  INFO JettyHandler:43 - Got request: /status.xml. Sender is 10.1.53.1:40440
            Creating placeholder flownodes because failed loading originals.
            java.io.IOException: Tried to load head FlowNodes for execution Owner[RT-Jobs/RT-Nightly/9:RT-Jobs/RT-Nightly #9] but FlowNode was not found in storage for head id:FlowNodeId 1:526
            	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679)
            	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:662)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526)
            	at hudson.model.RunMap.retrieve(RunMap.java:225)
            	at hudson.model.RunMap.retrieve(RunMap.java:57)
            	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
            	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
            	at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
            	at hudson.model.RunMap.getById(RunMap.java:205)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:899)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:910)
            	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
            	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
            	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
            	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
            	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
            	at jenkins.model.Jenkins.<init>(Jenkins.java:995)
            	at hudson.model.Hudson.<init>(Hudson.java:85)
            	at hudson.model.Hudson.<init>(Hudson.java:81)
            	at hudson.WebAppMain$3.run(WebAppMain.java:233)
            Finished: UNSTABLE

            Quite annoying as it follows Murphy's Law: Occurs in the exact worst moment... :-/

            Show
            swf Yves Schumann added a comment - We've spot the same today, all our nightly pipelines break with this error: 04:02:41 04:02:40,990 INFO JettyHandler:43 - Got request: /status.xml. Sender is 10.1.53.1:40440 Creating placeholder flownodes because failed loading originals. java.io.IOException: Tried to load head FlowNodes for execution Owner[RT-Jobs/RT-Nightly/9:RT-Jobs/RT-Nightly #9] but FlowNode was not found in storage for head id:FlowNodeId 1:526 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716) at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:662) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526) at hudson.model.RunMap.retrieve(RunMap.java:225) at hudson.model.RunMap.retrieve(RunMap.java:57) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381) at hudson.model.RunMap.getById(RunMap.java:205) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:899) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:910) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178) at jenkins.model.Jenkins.<init>(Jenkins.java:995) at hudson.model.Hudson.<init>(Hudson.java:85) at hudson.model.Hudson.<init>(Hudson.java:81) at hudson.WebAppMain$3.run(WebAppMain.java:233) Finished: UNSTABLE Quite annoying as it follows Murphy's Law: Occurs in the exact worst moment... :-/
            Hide
            oxygenxo Andrey Babushkin added a comment -

            I think this is happening due to our pipeline durability settings override - we've set it to MAX_PERFORMANCE. Therefore all pipelines die when Jenkins master dies

            Show
            oxygenxo Andrey Babushkin added a comment - I think this is happening due to our pipeline durability settings override - we've set it to MAX_PERFORMANCE. Therefore all pipelines die when Jenkins master dies
            stoiky Mihai Stoichitescu made changes -
            Attachment plugins_versions_2.190.1.txt [ 49925 ]
            Hide
            stoiky Mihai Stoichitescu added a comment -

            We are also getting this from time to time after updating Jenkins from 2.121.3 to 2.190.1

            I've attached a list of our plugins used.

            Show
            stoiky Mihai Stoichitescu added a comment - We are also getting this from time to time after updating Jenkins from 2.121.3 to 2.190.1 I've attached a list of our plugins used.
            Hide
            dnusbaum Devin Nusbaum added a comment -

            For everyone experiencing this issue: we'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ($JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/) of the build that failed and upload it here, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any exceptions or warnings in the Jenkins system logs that seem relevant, etc.

            My best hypothesis is that there is a concurrency issue that causes the flow graph to not be saved in in some cases where it should be saved in all durability modes, but this is only noticed in performance-optimized mode because the Pipeline is not saved regularly like in other modes. One thing that we could do to investigate this hypothesis would be to instrument all uses of BulkChange in workflow-job and workflow-cps with optional logging to understand if bulk changes being aborted when a save is requested could be causing the problem.

            Show
            dnusbaum Devin Nusbaum added a comment - For everyone experiencing this issue: we'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ( $JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/ ) of the build that failed and upload it here, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any exceptions or warnings in the Jenkins system logs that seem relevant, etc. My best hypothesis is that there is a concurrency issue that causes the flow graph to not be saved in in some cases where it should be saved in all durability modes, but this is only noticed in performance-optimized mode because the Pipeline is not saved regularly like in other modes. One thing that we could do to investigate this hypothesis would be to instrument all uses of BulkChange in workflow-job and workflow-cps with optional logging to understand if bulk changes being aborted when a save is requested could be causing the problem.
            dnusbaum Devin Nusbaum made changes -
            Description Creating placeholder flownodes because failed loading originals.
            java.io.IOException: Tried to load head FlowNodes for execution Owner[Platform Service FBI Test/1605:Platform Service FBI Test #1605] but FlowNode was not found in storage for head id:FlowNodeId 1:17
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:659)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:525)
             at hudson.model.RunMap.retrieve(RunMap.java:225)
             at hudson.model.RunMap.retrieve(RunMap.java:57)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379)
             at hudson.model.RunMap.getById(RunMap.java:205)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:896)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:907)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
             at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
             at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
             at jenkins.model.Jenkins.<init>(Jenkins.java:975)
             at hudson.model.Hudson.<init>(Hudson.java:85)
             at hudson.model.Hudson.<init>(Hudson.java:81)
             at hudson.WebAppMain$3.run(WebAppMain.java:233)
            Finished: FAILURE
            {noformat}
            Creating placeholder flownodes because failed loading originals.
            java.io.IOException: Tried to load head FlowNodes for execution Owner[Platform Service FBI Test/1605:Platform Service FBI Test #1605] but FlowNode was not found in storage for head id:FlowNodeId 1:17
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:659)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:525)
             at hudson.model.RunMap.retrieve(RunMap.java:225)
             at hudson.model.RunMap.retrieve(RunMap.java:57)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379)
             at hudson.model.RunMap.getById(RunMap.java:205)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:896)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:907)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
             at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
             at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
             at jenkins.model.Jenkins.<init>(Jenkins.java:975)
             at hudson.model.Hudson.<init>(Hudson.java:85)
             at hudson.model.Hudson.<init>(Hudson.java:81)
             at hudson.WebAppMain$3.run(WebAppMain.java:233)
            Finished: FAILURE
            {noformat}
            dnusbaum Devin Nusbaum made changes -
            Description {noformat}
            Creating placeholder flownodes because failed loading originals.
            java.io.IOException: Tried to load head FlowNodes for execution Owner[Platform Service FBI Test/1605:Platform Service FBI Test #1605] but FlowNode was not found in storage for head id:FlowNodeId 1:17
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:659)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:525)
             at hudson.model.RunMap.retrieve(RunMap.java:225)
             at hudson.model.RunMap.retrieve(RunMap.java:57)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379)
             at hudson.model.RunMap.getById(RunMap.java:205)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:896)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:907)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
             at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
             at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
             at jenkins.model.Jenkins.<init>(Jenkins.java:975)
             at hudson.model.Hudson.<init>(Hudson.java:85)
             at hudson.model.Hudson.<init>(Hudson.java:81)
             at hudson.WebAppMain$3.run(WebAppMain.java:233)
            Finished: FAILURE
            {noformat}
            Note from maintainer: We'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ({{$JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/}}) of the build that failed and upload it here, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any exceptions or warnings in the Jenkins system logs that seem relevant, etc. Thanks!

            {noformat}
            Creating placeholder flownodes because failed loading originals.
            java.io.IOException: Tried to load head FlowNodes for execution Owner[Platform Service FBI Test/1605:Platform Service FBI Test #1605] but FlowNode was not found in storage for head id:FlowNodeId 1:17
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:659)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:525)
             at hudson.model.RunMap.retrieve(RunMap.java:225)
             at hudson.model.RunMap.retrieve(RunMap.java:57)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379)
             at hudson.model.RunMap.getById(RunMap.java:205)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:896)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:907)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
             at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
             at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
             at jenkins.model.Jenkins.<init>(Jenkins.java:975)
             at hudson.model.Hudson.<init>(Hudson.java:85)
             at hudson.model.Hudson.<init>(Hudson.java:81)
             at hudson.WebAppMain$3.run(WebAppMain.java:233)
            Finished: FAILURE
            {noformat}
            Hide
            dnusbaum Devin Nusbaum added a comment -

            One thing that we could do to investigate this hypothesis would be to instrument all uses of BulkChange in workflow-job and workflow-cps with optional logging to understand if bulk changes being aborted when a save is requested could be causing the problem.

            Looking into this some more, I think flow nodes and program.dat are saved through more direct paths that appear to bypass BulkChange, so that might not help. Maybe the implementations of FlowNodeStorage need some synchronization? That seems like a decent place to start.

            Show
            dnusbaum Devin Nusbaum added a comment - One thing that we could do to investigate this hypothesis would be to instrument all uses of BulkChange in workflow-job and workflow-cps with optional logging to understand if bulk changes being aborted when a save is requested could be causing the problem. Looking into this some more, I think flow nodes and program.dat are saved through more direct paths that appear to bypass BulkChange , so that might not help. Maybe the implementations of FlowNodeStorage need some synchronization? That seems like a decent place to start.
            Hide
            dnusbaum Devin Nusbaum added a comment -

            I filed https://github.com/jenkinsci/workflow-support-plugin/pull/104 based on my above hypothesis. If anyone is running into this issue consistently, I can create a build of the plugin based on that PR to see if it helps. I have not been able to reproduce the issue on demand, so I don't have a good way to validate whether that PR actually helps or not.

            Show
            dnusbaum Devin Nusbaum added a comment - I filed https://github.com/jenkinsci/workflow-support-plugin/pull/104 based on my above hypothesis. If anyone is running into this issue consistently, I can create a build of the plugin based on that PR to see if it helps. I have not been able to reproduce the issue on demand, so I don't have a good way to validate whether that PR actually helps or not.
            dnusbaum Devin Nusbaum made changes -
            Remote Link This issue links to "jenkinsci/workflow-support-plugin#104 (Web Link)" [ 24309 ]
            Hide
            dnusbaum Devin Nusbaum added a comment - - edited

            Here is the SNAPSHOT build of workflow-support 3.5 with my PR in case anyone would like to try it out. If you do try out that build, or even if you don't try it, if you still run into this issue, please upload a build folder of the build that had the issue as described in my note in the issue description here.

            Show
            dnusbaum Devin Nusbaum added a comment - - edited Here is the SNAPSHOT build of workflow-support 3.5 with my PR in case anyone would like to try it out. If you do try out that build, or even if you don't try it, if you still run into this issue, please upload a build folder of the build that had the issue as described in my note in the issue description here.
            Hide
            rodrigc Craig Rodrigues added a comment -

            Devin Nusbaum If you have tested this a bit, can you merge this and deploy a new workflow-support-plugin?
            I keep hitting this at intermittent times, but I can never fully reproduce it.
            Unfortunately, this keeps hitting me right in the middle of when I am doing important stuff, and I don't have time to debug this.

            Show
            rodrigc Craig Rodrigues added a comment - Devin Nusbaum If you have tested this a bit, can you merge this and deploy a new workflow-support-plugin? I keep hitting this at intermittent times, but I can never fully reproduce it. Unfortunately, this keeps hitting me right in the middle of when I am doing important stuff, and I don't have time to debug this.
            henrikk Henrik Koren made changes -
            Attachment 48064.tar.gz [ 50075 ]
            Hide
            henrikk Henrik Koren added a comment -

            48064.tar.gz I have the same issues 

            I have uploaded the entire build from my Master

            Creating placeholder flownodes because failed loading originals.
            java.io.IOException: Tried to load head FlowNodes for execution Owner[Utils/CreateNodePipeline/48064:Utils/CreateNodePipeline #48064] but FlowNode was not found in storage for head id:FlowNodeId 1:20
            	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679)
            	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526)
            	at hudson.model.RunMap.retrieve(RunMap.java:225)
            	at hudson.model.RunMap.retrieve(RunMap.java:57)
            	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
            	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
            	at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
            	at hudson.model.RunMap.getById(RunMap.java:205)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912)
            	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
            	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
            	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
            	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
            	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
            	at jenkins.model.Jenkins.<init>(Jenkins.java:999)
            	at hudson.model.Hudson.<init>(Hudson.java:85)
            	at hudson.model.Hudson.<init>(Hudson.java:81)
            	at hudson.WebAppMain$3.run(WebAppMain.java:233)
            Finished: FAILURE
            Show
            henrikk Henrik Koren added a comment - 48064.tar.gz  I have the same issues  I have uploaded the entire build from my Master Creating placeholder flownodes because failed loading originals. java.io.IOException: Tried to load head FlowNodes for execution Owner[Utils/CreateNodePipeline/48064:Utils/CreateNodePipeline #48064] but FlowNode was not found in storage for head id:FlowNodeId 1:20 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716) at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:664) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:526) at hudson.model.RunMap.retrieve(RunMap.java:225) at hudson.model.RunMap.retrieve(RunMap.java:57) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381) at hudson.model.RunMap.getById(RunMap.java:205) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:901) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:912) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178) at jenkins.model.Jenkins.<init>(Jenkins.java:999) at hudson.model.Hudson.<init>(Hudson.java:85) at hudson.model.Hudson.<init>(Hudson.java:81) at hudson.WebAppMain$3.run(WebAppMain.java:233) Finished: FAILURE
            Hide
            dnusbaum Devin Nusbaum added a comment -

            I got feedback from some users who were able to test the PR that it did not fix the issue. I am waiting for a build folder from those users to confirm that they are seeing the same issue. I think the next step would be to make a new hypothesis as to the issue and submit a PR. If anyone has even a semi-consistent way to reproduce the issue, it would make it much more likely that we could fix it.

            Henrik Koren Thanks!

            Craig Rodrigues No, see above.

            Show
            dnusbaum Devin Nusbaum added a comment - I got feedback from some users who were able to test the PR that it did not fix the issue. I am waiting for a build folder from those users to confirm that they are seeing the same issue. I think the next step would be to make a new hypothesis as to the issue and submit a PR. If anyone has even a semi-consistent way to reproduce the issue, it would make it much more likely that we could fix it. Henrik Koren Thanks! Craig Rodrigues No, see above.
            Hide
            famod Falko Modler added a comment -

            Any news? This is biting us on a regular basis.

            Show
            famod Falko Modler added a comment - Any news? This is biting us on a regular basis.
            Hide
            spalani Senthilkumar Palanisamy added a comment -

            Hi,

            We are facing this issue frequently is there any workaround or any fix.

            Our Jenkins version 2.204.1

            Show
            spalani Senthilkumar Palanisamy added a comment - Hi, We are facing this issue frequently is there any workaround or any fix. Our Jenkins version 2.204.1
            Hide
            mramonleon Ramon Leon added a comment -

            I'm facing this issue on https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fldap-plugin/detail/PR-40/4/pipeline/ for this PR: https://github.com/jenkinsci/ldap-plugin/pull/40

            I guess Olivier Lamy can give you more information Devin Nusbaum. The log shows:

            [2020-03-24T12:26:50.886Z] java.io.IOException: Tried to load head FlowNodes for execution Owner[Plugins/ldap-plugin/PR-40/4:Plugins/ldap-plugin/PR-40 #4] but FlowNode was not found in storage for head id:FlowNodeId 1:76
            [2020-03-24T12:26:50.886Z] 	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679)
            [2020-03-24T12:26:50.886Z] 	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716)
            [2020-03-24T12:26:50.886Z] 	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:672)
            [2020-03-24T12:26:50.886Z] 	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:533)
            [2020-03-24T12:26:50.886Z] 	at hudson.model.RunMap.retrieve(RunMap.java:225)
            [2020-03-24T12:26:50.886Z] 	at hudson.model.RunMap.retrieve(RunMap.java:57)
            [2020-03-24T12:26:50.886Z] 	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
            [2020-03-24T12:26:50.886Z] 	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
            [2020-03-24T12:26:50.886Z] 	at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
            [2020-03-24T12:26:50.886Z] 	at hudson.model.RunMap.getById(RunMap.java:205)
            [2020-03-24T12:26:50.886Z] 	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:921)
            [2020-03-24T12:26:50.886Z] 	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:932)
            [2020-03-24T12:26:50.886Z] 	at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getExecution(CpsStepContext.java:217)
            [2020-03-24T12:26:50.886Z] 	at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getThreadGroupSynchronously(CpsStepContext.java:242)
            [2020-03-24T12:26:50.886Z] 	at org.jenkinsci.plugins.workflow.cps.CpsStepContext.access$000(CpsStepContext.java:97)
            [2020-03-24T12:26:50.886Z] 	at org.jenkinsci.plugins.workflow.cps.CpsStepContext$1.call(CpsStepContext.java:263)
            [2020-03-24T12:26:50.886Z] 	at org.jenkinsci.plugins.workflow.cps.CpsStepContext$1.call(CpsStepContext.java:261)
            [2020-03-24T12:26:50.886Z] 	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
            [2020-03-24T12:26:50.886Z] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
            [2020-03-24T12:26:50.886Z] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
            [2020-03-24T12:26:50.886Z] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
            [2020-03-24T12:26:50.886Z] 	at java.base/java.lang.Thread.run(Thread.java:834)
            [2020-03-24T12:26:50.886Z] Finished: FAILURE
            
            Show
            mramonleon Ramon Leon added a comment - I'm facing this issue on https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fldap-plugin/detail/PR-40/4/pipeline/ for this PR: https://github.com/jenkinsci/ldap-plugin/pull/40 I guess Olivier Lamy can give you more information Devin Nusbaum . The log shows: [2020-03-24T12:26:50.886Z] java.io.IOException: Tried to load head FlowNodes for execution Owner[Plugins/ldap-plugin/PR-40/4:Plugins/ldap-plugin/PR-40 #4] but FlowNode was not found in storage for head id:FlowNodeId 1:76 [2020-03-24T12:26:50.886Z] at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679) [2020-03-24T12:26:50.886Z] at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716) [2020-03-24T12:26:50.886Z] at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:672) [2020-03-24T12:26:50.886Z] at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:533) [2020-03-24T12:26:50.886Z] at hudson.model.RunMap.retrieve(RunMap.java:225) [2020-03-24T12:26:50.886Z] at hudson.model.RunMap.retrieve(RunMap.java:57) [2020-03-24T12:26:50.886Z] at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501) [2020-03-24T12:26:50.886Z] at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483) [2020-03-24T12:26:50.886Z] at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381) [2020-03-24T12:26:50.886Z] at hudson.model.RunMap.getById(RunMap.java:205) [2020-03-24T12:26:50.886Z] at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:921) [2020-03-24T12:26:50.886Z] at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:932) [2020-03-24T12:26:50.886Z] at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getExecution(CpsStepContext.java:217) [2020-03-24T12:26:50.886Z] at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getThreadGroupSynchronously(CpsStepContext.java:242) [2020-03-24T12:26:50.886Z] at org.jenkinsci.plugins.workflow.cps.CpsStepContext.access$000(CpsStepContext.java:97) [2020-03-24T12:26:50.886Z] at org.jenkinsci.plugins.workflow.cps.CpsStepContext$1.call(CpsStepContext.java:263) [2020-03-24T12:26:50.886Z] at org.jenkinsci.plugins.workflow.cps.CpsStepContext$1.call(CpsStepContext.java:261) [2020-03-24T12:26:50.886Z] at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) [2020-03-24T12:26:50.886Z] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [2020-03-24T12:26:50.886Z] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [2020-03-24T12:26:50.886Z] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [2020-03-24T12:26:50.886Z] at java.base/java.lang. Thread .run( Thread .java:834) [2020-03-24T12:26:50.886Z] Finished: FAILURE
            bitwiseman Liam Newman made changes -
            Description Note from maintainer: We'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ({{$JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/}}) of the build that failed and upload it here, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any exceptions or warnings in the Jenkins system logs that seem relevant, etc. Thanks!

            {noformat}
            Creating placeholder flownodes because failed loading originals.
            java.io.IOException: Tried to load head FlowNodes for execution Owner[Platform Service FBI Test/1605:Platform Service FBI Test #1605] but FlowNode was not found in storage for head id:FlowNodeId 1:17
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:659)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:525)
             at hudson.model.RunMap.retrieve(RunMap.java:225)
             at hudson.model.RunMap.retrieve(RunMap.java:57)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379)
             at hudson.model.RunMap.getById(RunMap.java:205)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:896)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:907)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
             at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
             at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
             at jenkins.model.Jenkins.<init>(Jenkins.java:975)
             at hudson.model.Hudson.<init>(Hudson.java:85)
             at hudson.model.Hudson.<init>(Hudson.java:81)
             at hudson.WebAppMain$3.run(WebAppMain.java:233)
            Finished: FAILURE
            {noformat}
            *IMPORTANT: NOTE FROM A MAINTAINER:*

            +STOP!  YOUR STACK TRACE ALONE IS NOT GOING TO HELP US SOLVE THIS!+

            (sorry to all caps but we're not going to make progress on this issue with commenters adding insufficient information)

            *Note from maintainer: We'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ({{$JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/}}) of the build that failed and upload it here, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any exceptions or warnings in the Jenkins system logs that seem relevant, etc. Thanks!*

             
            {noformat}
            Creating placeholder flownodes because failed loading originals.
            java.io.IOException: Tried to load head FlowNodes for execution Owner[Platform Service FBI Test/1605:Platform Service FBI Test #1605] but FlowNode was not found in storage for head id:FlowNodeId 1:17
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:659)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:525)
             at hudson.model.RunMap.retrieve(RunMap.java:225)
             at hudson.model.RunMap.retrieve(RunMap.java:57)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379)
             at hudson.model.RunMap.getById(RunMap.java:205)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:896)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:907)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
             at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
             at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
             at jenkins.model.Jenkins.<init>(Jenkins.java:975)
             at hudson.model.Hudson.<init>(Hudson.java:85)
             at hudson.model.Hudson.<init>(Hudson.java:81)
             at hudson.WebAppMain$3.run(WebAppMain.java:233)
            Finished: FAILURE
            {noformat}
            bitwiseman Liam Newman made changes -
            Description *IMPORTANT: NOTE FROM A MAINTAINER:*

            +STOP!  YOUR STACK TRACE ALONE IS NOT GOING TO HELP US SOLVE THIS!+

            (sorry to all caps but we're not going to make progress on this issue with commenters adding insufficient information)

            *Note from maintainer: We'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ({{$JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/}}) of the build that failed and upload it here, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any exceptions or warnings in the Jenkins system logs that seem relevant, etc. Thanks!*

             
            {noformat}
            Creating placeholder flownodes because failed loading originals.
            java.io.IOException: Tried to load head FlowNodes for execution Owner[Platform Service FBI Test/1605:Platform Service FBI Test #1605] but FlowNode was not found in storage for head id:FlowNodeId 1:17
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:659)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:525)
             at hudson.model.RunMap.retrieve(RunMap.java:225)
             at hudson.model.RunMap.retrieve(RunMap.java:57)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379)
             at hudson.model.RunMap.getById(RunMap.java:205)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:896)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:907)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
             at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
             at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
             at jenkins.model.Jenkins.<init>(Jenkins.java:975)
             at hudson.model.Hudson.<init>(Hudson.java:85)
             at hudson.model.Hudson.<init>(Hudson.java:81)
             at hudson.WebAppMain$3.run(WebAppMain.java:233)
            Finished: FAILURE
            {noformat}
            *IMPORTANT: NOTE FROM A MAINTAINER:*

            +STOP!  YOUR STACK TRACE ALONE IS NOT GOING TO HELP SOLVE THIS!+

            (sorry to all caps but we're not going to make progress on this issue with commenters adding insufficient information)

            *Note from maintainer: We'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ({{$JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/}}) of the build that failed and upload it here, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any exceptions or warnings in the Jenkins system logs that seem relevant, etc. Thanks!*

             
            {noformat}
            Creating placeholder flownodes because failed loading originals.
            java.io.IOException: Tried to load head FlowNodes for execution Owner[Platform Service FBI Test/1605:Platform Service FBI Test #1605] but FlowNode was not found in storage for head id:FlowNodeId 1:17
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:659)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:525)
             at hudson.model.RunMap.retrieve(RunMap.java:225)
             at hudson.model.RunMap.retrieve(RunMap.java:57)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379)
             at hudson.model.RunMap.getById(RunMap.java:205)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:896)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:907)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
             at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
             at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
             at jenkins.model.Jenkins.<init>(Jenkins.java:975)
             at hudson.model.Hudson.<init>(Hudson.java:85)
             at hudson.model.Hudson.<init>(Hudson.java:81)
             at hudson.WebAppMain$3.run(WebAppMain.java:233)
            Finished: FAILURE
            {noformat}
            Hide
            bitwiseman Liam Newman added a comment -

            IMPORTANT: NOTE FROM A MAINTAINER:

            STOP!  YOUR STACK TRACE ALONE IS NOT GOING TO HELP SOLVE THIS!

            (sorry to all caps but we're not going to make progress on this issue with commenters adding insufficient information)

            We'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ($JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/) of the build that failed and upload it here, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any exceptions or warnings in the Jenkins system logs that seem relevant, etc. Thanks!

            Ramon Leon
            **Sorry, I don't think we can get that information from ci.jenkins.io - the infra team is overloaded and we need the zip of the build folder immediately (or very soon) after the error.

             

            Show
            bitwiseman Liam Newman added a comment - IMPORTANT: NOTE FROM A MAINTAINER: STOP!  YOUR STACK TRACE ALONE IS NOT GOING TO HELP SOLVE THIS! (sorry to all caps but we're not going to make progress on this issue with commenters adding insufficient information) We'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ( $JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/ ) of the build that failed and upload it here, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any exceptions or warnings in the Jenkins system logs that seem relevant, etc. Thanks! Ramon Leon **Sorry, I don't think we can get that information from ci.jenkins.io - the infra team is overloaded and we need the zip of the build folder immediately (or very soon) after the error.  
            Hide
            rodrigc Craig Rodrigues added a comment - - edited

            Liam Newman I think a lot of pressure is being put on the end user to perform weird steps in order to help debug this problem. This is very hard to do. Since the problem
            is intermittent, it is hard to catch.

            Why don't you modify the pipeline plugin so at the point that the pipeline tries to throw an exception with message but FlowNode was not found in storage for head, the plugin zips up the entire job directory, and puts it in some place like $TEMPDIR ?
            You can release a version of the plugin with this change.
            That way the end user doesn't have to perform a lot of weird steps, but they can simply retrieve a zip file with all the stuff the is needed for the pipeline plugin maintainer to diagnose and root cause the problem?

            Show
            rodrigc Craig Rodrigues added a comment - - edited Liam Newman I think a lot of pressure is being put on the end user to perform weird steps in order to help debug this problem. This is very hard to do. Since the problem is intermittent, it is hard to catch. Why don't you modify the pipeline plugin so at the point that the pipeline tries to throw an exception with message but FlowNode was not found in storage for head , the plugin zips up the entire job directory, and puts it in some place like $TEMPDIR ? You can release a version of the plugin with this change. That way the end user doesn't have to perform a lot of weird steps, but they can simply retrieve a zip file with all the stuff the is needed for the pipeline plugin maintainer to diagnose and root cause the problem?
            Hide
            portchavz Steven Chaves added a comment -

            I don't have rights to zip my Build Job folder.

            Show
            portchavz Steven Chaves added a comment - I don't have rights to zip my Build Job folder.
            dnusbaum Devin Nusbaum made changes -
            Description *IMPORTANT: NOTE FROM A MAINTAINER:*

            +STOP!  YOUR STACK TRACE ALONE IS NOT GOING TO HELP SOLVE THIS!+

            (sorry to all caps but we're not going to make progress on this issue with commenters adding insufficient information)

            *Note from maintainer: We'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ({{$JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/}}) of the build that failed and upload it here, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any exceptions or warnings in the Jenkins system logs that seem relevant, etc. Thanks!*

             
            {noformat}
            Creating placeholder flownodes because failed loading originals.
            java.io.IOException: Tried to load head FlowNodes for execution Owner[Platform Service FBI Test/1605:Platform Service FBI Test #1605] but FlowNode was not found in storage for head id:FlowNodeId 1:17
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:659)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:525)
             at hudson.model.RunMap.retrieve(RunMap.java:225)
             at hudson.model.RunMap.retrieve(RunMap.java:57)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379)
             at hudson.model.RunMap.getById(RunMap.java:205)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:896)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:907)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
             at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
             at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
             at jenkins.model.Jenkins.<init>(Jenkins.java:975)
             at hudson.model.Hudson.<init>(Hudson.java:85)
             at hudson.model.Hudson.<init>(Hudson.java:81)
             at hudson.WebAppMain$3.run(WebAppMain.java:233)
            Finished: FAILURE
            {noformat}
            *IMPORTANT: NOTE FROM A MAINTAINER:*

            +STOP!  YOUR STACK TRACE ALONE IS NOT GOING TO HELP SOLVE THIS!+

            (sorry to all caps but we're not going to make progress on this issue with commenters adding insufficient information)

            *Note from maintainer: We'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ({{$JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/}}) of the build that failed and upload it here along with the Jenkins system logs, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any messages in the Jenkins system logs that seem relevant, etc. Thanks!*

             
            {noformat}
            Creating placeholder flownodes because failed loading originals.
            java.io.IOException: Tried to load head FlowNodes for execution Owner[Platform Service FBI Test/1605:Platform Service FBI Test #1605] but FlowNode was not found in storage for head id:FlowNodeId 1:17
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:659)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:525)
             at hudson.model.RunMap.retrieve(RunMap.java:225)
             at hudson.model.RunMap.retrieve(RunMap.java:57)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379)
             at hudson.model.RunMap.getById(RunMap.java:205)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:896)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:907)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
             at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
             at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
             at jenkins.model.Jenkins.<init>(Jenkins.java:975)
             at hudson.model.Hudson.<init>(Hudson.java:85)
             at hudson.model.Hudson.<init>(Hudson.java:81)
             at hudson.WebAppMain$3.run(WebAppMain.java:233)
            Finished: FAILURE
            {noformat}
            Hide
            famod Falko Modler added a comment - - edited

            Devin Nusbaum & Liam Newman

            I have three zipped build folders (all from around the same time, March 12th) but did not yet have the time to redact sensitive data. Do you have some hints what and where to look for (regarding sensitive data)?

            Show
            famod Falko Modler added a comment - - edited Devin Nusbaum & Liam Newman I have three zipped build folders (all from around the same time, March 12th) but did not yet have the time to redact sensitive data. Do you have some hints what and where to look for (regarding sensitive data)?
            Hide
            dnusbaum Devin Nusbaum added a comment - - edited

            I was looking into this issue recently and wanted to check whether the code that saves Pipelines using the PERFORMANCE_OPTIMIZED durability level when Jenkins is shutting down (CpsFlowExecution.suspendAll) is running successfully in these cases to make sure that the problem isn't something like the service configuration for Jenkins not giving it enough time to shut down.

            For users seeing the problem, please check the Jenkins system logs for the shutdown that occurred before you saw this error. Do you see messages similar to these, in particular for the build that failed with the exception in the description of this ticket?

            • About to try to checkpoint the program for buildCpsFlowExecution[Owner[YourJobName/BuildNumber:YourJobName #BuildNumber]]
            • Trying to save program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash
            • Finished saving program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash

            Those messages should come in groups of three, in that order, and are logged at INFO level, and there should not be any WARNING messages interspersed between them or surrounding them. There should be a set of those messages for each Pipeline running using the PERFORMANCE_OPTIMIZED durability level when Jenkins shuts down. If you have configured Jenkins to run as a service, I would also check the service logs to see if Jenkins has been shutting down successfully or if there are any warnings/errors when Jenkins shuts down.

            Falko Modler It depends on what your builds are doing and what you consider sensitive. The main files to check are build.xml, which contains a textual form of the Pipeline, and log, which contains the build logs. You might want to check for internal hostnames or IP addresses, unmasked passwords or API keys in your build log, etc.

            Show
            dnusbaum Devin Nusbaum added a comment - - edited I was looking into this issue recently and wanted to check whether the code that saves Pipelines using the PERFORMANCE_OPTIMIZED durability level when Jenkins is shutting down ( CpsFlowExecution.suspendAll ) is running successfully in these cases to make sure that the problem isn't something like the service configuration for Jenkins not giving it enough time to shut down. For users seeing the problem, please check the Jenkins system logs for the shutdown that occurred before you saw this error. Do you see messages similar to these, in particular for the build that failed with the exception in the description of this ticket? About to try to checkpoint the program for buildCpsFlowExecution[Owner[YourJobName/BuildNumber:YourJobName #BuildNumber]] Trying to save program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash Finished saving program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash Those messages should come in groups of three, in that order, and are logged at INFO level, and there should not be any WARNING messages interspersed between them or surrounding them. There should be a set of those messages for each Pipeline running using the PERFORMANCE_OPTIMIZED durability level when Jenkins shuts down. If you have configured Jenkins to run as a service, I would also check the service logs to see if Jenkins has been shutting down successfully or if there are any warnings/errors when Jenkins shuts down. Falko Modler It depends on what your builds are doing and what you consider sensitive. The main files to check are build.xml , which contains a textual form of the Pipeline, and log , which contains the build logs. You might want to check for internal hostnames or IP addresses, unmasked passwords or API keys in your build log, etc.
            jequals5 Marky Jackson made changes -
            Issue Type Bug [ 1 ] Story [ 10002 ]
            jequals5 Marky Jackson made changes -
            Priority Blocker [ 1 ] Major [ 3 ]
            jequals5 Marky Jackson made changes -
            Assignee Marky Jackson [ jequals5 ]
            jequals5 Marky Jackson made changes -
            Summary Pipeline: Failure to load flow node: FlowNode was not found in storage for head Pipeline syntax Improvements
            jequals5 Marky Jackson made changes -
            Description *IMPORTANT: NOTE FROM A MAINTAINER:*

            +STOP!  YOUR STACK TRACE ALONE IS NOT GOING TO HELP SOLVE THIS!+

            (sorry to all caps but we're not going to make progress on this issue with commenters adding insufficient information)

            *Note from maintainer: We'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ({{$JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/}}) of the build that failed and upload it here along with the Jenkins system logs, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any messages in the Jenkins system logs that seem relevant, etc. Thanks!*

             
            {noformat}
            Creating placeholder flownodes because failed loading originals.
            java.io.IOException: Tried to load head FlowNodes for execution Owner[Platform Service FBI Test/1605:Platform Service FBI Test #1605] but FlowNode was not found in storage for head id:FlowNodeId 1:17
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:659)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:525)
             at hudson.model.RunMap.retrieve(RunMap.java:225)
             at hudson.model.RunMap.retrieve(RunMap.java:57)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379)
             at hudson.model.RunMap.getById(RunMap.java:205)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:896)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:907)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
             at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
             at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
             at jenkins.model.Jenkins.<init>(Jenkins.java:975)
             at hudson.model.Hudson.<init>(Hudson.java:85)
             at hudson.model.Hudson.<init>(Hudson.java:81)
             at hudson.WebAppMain$3.run(WebAppMain.java:233)
            Finished: FAILURE
            {noformat}
            This story addresses Pipeline syntax Improvements as detailed on the Jenkins roadmap for the pipeline-authoring SIG
            dnusbaum Devin Nusbaum made changes -
            Description This story addresses Pipeline syntax Improvements as detailed on the Jenkins roadmap for the pipeline-authoring SIG *IMPORTANT: NOTE FROM A MAINTAINER:*

            +STOP! YOUR STACK TRACE ALONE IS NOT GOING TO HELP SOLVE THIS!+

            (sorry to all caps but we're not going to make progress on this issue with commenters adding insufficient information)

            *Note from maintainer: We'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ({{$JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/}}) of the build that failed and upload it here along with the Jenkins system logs, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any messages in the Jenkins system logs that seem relevant, etc. Thanks!*

             
            {noformat}
            Creating placeholder flownodes because failed loading originals.
            java.io.IOException: Tried to load head FlowNodes for execution Owner[Platform Service FBI Test/1605:Platform Service FBI Test #1605] but FlowNode was not found in storage for head id:FlowNodeId 1:17
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:659)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:525)
             at hudson.model.RunMap.retrieve(RunMap.java:225)
             at hudson.model.RunMap.retrieve(RunMap.java:57)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379)
             at hudson.model.RunMap.getById(RunMap.java:205)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:896)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:907)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
             at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
             at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
             at jenkins.model.Jenkins.<init>(Jenkins.java:975)
             at hudson.model.Hudson.<init>(Hudson.java:85)
             at hudson.model.Hudson.<init>(Hudson.java:81)
             at hudson.WebAppMain$3.run(WebAppMain.java:233)
            Finished: FAILURE
            {noformat}
            dnusbaum Devin Nusbaum made changes -
            Description *IMPORTANT: NOTE FROM A MAINTAINER:*

            +STOP! YOUR STACK TRACE ALONE IS NOT GOING TO HELP SOLVE THIS!+

            (sorry to all caps but we're not going to make progress on this issue with commenters adding insufficient information)

            *Note from maintainer: We'd like to be able to fix this, but we really need more information to do so. Please, whenever you encounter the error in the description of the ticket, zip the build folder ({{$JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/}}) of the build that failed and upload it here along with the Jenkins system logs, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any messages in the Jenkins system logs that seem relevant, etc. Thanks!*

             
            {noformat}
            Creating placeholder flownodes because failed loading originals.
            java.io.IOException: Tried to load head FlowNodes for execution Owner[Platform Service FBI Test/1605:Platform Service FBI Test #1605] but FlowNode was not found in storage for head id:FlowNodeId 1:17
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:659)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:525)
             at hudson.model.RunMap.retrieve(RunMap.java:225)
             at hudson.model.RunMap.retrieve(RunMap.java:57)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379)
             at hudson.model.RunMap.getById(RunMap.java:205)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:896)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:907)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
             at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
             at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
             at jenkins.model.Jenkins.<init>(Jenkins.java:975)
             at hudson.model.Hudson.<init>(Hudson.java:85)
             at hudson.model.Hudson.<init>(Hudson.java:81)
             at hudson.WebAppMain$3.run(WebAppMain.java:233)
            Finished: FAILURE
            {noformat}
            *IMPORTANT: NOTE FROM A MAINTAINER:*

            +STOP! YOUR STACK TRACE ALONE IS NOT GOING TO HELP SOLVE THIS!+

            (sorry to all caps but we're not going to make progress on this issue with commenters adding insufficient information)

            *Note from maintainer: We'd like to be able to fix this, but we really need more information to do so.* Please, whenever you encounter the error in the description of the ticket, zip the build folder ({{$JENKINS_HOME/jobs/$PATH_TO_JOB/builds/$BUILD_NUMBER/}}) of the build that failed and upload it here along with the Jenkins system logs, redacting any sensitive content as necessary, and include any relevant information on frequency of the issue, steps to reproduce (did it happen after Jenkins was restarted normally, or did Jenkins crash), any messages in the Jenkins system logs that seem relevant, etc. In addition, please check service or other system level logs for Jenkins to see if there are any issues with Jenkins taking too long to shut down or anything like that. Thanks!

            The main thing we are currently looking for is whether these messages are present in the Jenkins logs right before Jenkins shut down for the build which has the error:
             * About to try to checkpoint the program for buildCpsFlowExecution[Owner[YourJobName/BuildNumber:YourJobName #BuildNumber]|#BuildNumber]]
             * Trying to save program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash
             * Finished saving program before shutdown org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$8@RandomHash

            If these messages are not present, it means that Jenkins was unable to save the Pipeline, so the error is expected. If that is the case, fixing the issue probably requires changes to Jenkins packaging to configure longer service timeouts on shutdown, or totally changing how PERFORMANCE_OPTIMIZED works. If the messages _are_ present, then something else is happening.

            Exception:
            {noformat}
            Creating placeholder flownodes because failed loading originals.
            java.io.IOException: Tried to load head FlowNodes for execution Owner[Platform Service FBI Test/1605:Platform Service FBI Test #1605] but FlowNode was not found in storage for head id:FlowNodeId 1:17
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:678)
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:715)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:659)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:525)
             at hudson.model.RunMap.retrieve(RunMap.java:225)
             at hudson.model.RunMap.retrieve(RunMap.java:57)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379)
             at hudson.model.RunMap.getById(RunMap.java:205)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:896)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:907)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
             at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
             at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
             at jenkins.model.Jenkins.<init>(Jenkins.java:975)
             at hudson.model.Hudson.<init>(Hudson.java:85)
             at hudson.model.Hudson.<init>(Hudson.java:81)
             at hudson.WebAppMain$3.run(WebAppMain.java:233)
            Finished: FAILURE
            {noformat}
            dnusbaum Devin Nusbaum made changes -
            Summary Pipeline syntax Improvements Pipeline: Failure to load flow node: FlowNode was not found in storage for head
            dnusbaum Devin Nusbaum made changes -
            Assignee Marky Jackson [ jequals5 ]
            dnusbaum Devin Nusbaum made changes -
            Issue Type Story [ 10002 ] Bug [ 1 ]
            Hide
            famod Falko Modler added a comment -

            Devin Nusbaum I am in the process of redacting sensitive content. I will have to remove some files and folders. Is there anything useful for you in workspace-files?

            Can I send you the archive in a less public way somehow?

            Show
            famod Falko Modler added a comment - Devin Nusbaum I am in the process of redacting sensitive content. I will have to remove some files and folders. Is there anything useful for you in workspace-files ? Can I send you the archive in a less public way somehow?
            oleg_nenashev Oleg Nenashev made changes -
            Priority Major [ 3 ] Blocker [ 1 ]
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Restored the priority set by the reporters

            Show
            oleg_nenashev Oleg Nenashev added a comment - Restored the priority set by the reporters
            Hide
            dnusbaum Devin Nusbaum added a comment - - edited

            Falko Modler You can email it to me: dnusbaum at cloudbees.com. And no, I don't need to see workspace files. The main thing I want to see is the Jenkins system logs and or/service logs around the time of the last Jenkins restart to see if the build in question was able to be saved before the shutdown or not.

            Show
            dnusbaum Devin Nusbaum added a comment - - edited Falko Modler You can email it to me: dnusbaum at cloudbees.com. And no, I don't need to see workspace files. The main thing I want to see is the Jenkins system logs and or/service logs around the time of the last Jenkins restart to see if the build in question was able to be saved before the shutdown or not.
            Hide
            mhollingsworthcs Mark Hollingsworth added a comment -

            I'm not able to provide data due to company rules, but as an additional datapoint, we experienced this when the job in question lost connection to the node, not from a jenkins restart.

             

            [2020-07-24T13:47:01.622Z] Cannot contact someNodeNameHere: java.lang.InterruptedException
            Creating placeholder flownodes because failed loading originals.
            java.io.IOException: Tried to load head FlowNodes for execution Owner[SomeFolder/SomeJob/SomeBuild:SomeFolder/SomeJob #SomeBuild] but FlowNode was not found in storage for head id:FlowNodeId 1:1714
            

             

             

            Show
            mhollingsworthcs Mark Hollingsworth added a comment - I'm not able to provide data due to company rules, but as an additional datapoint, we experienced this when the job in question lost connection to the node, not from a jenkins restart.   [2020-07-24T13:47:01.622Z] Cannot contact someNodeNameHere: java.lang.InterruptedException Creating placeholder flownodes because failed loading originals. java.io.IOException: Tried to load head FlowNodes for execution Owner[SomeFolder/SomeJob/SomeBuild:SomeFolder/SomeJob #SomeBuild] but FlowNode was not found in storage for head id:FlowNodeId 1:1714    
            Hide
            dnusbaum Devin Nusbaum added a comment - - edited

            Mark Hollingsworth Thanks for the feedback. As far as I can tell though there is no way for the "Creating placeholder flownodes because failed loading originals" message to be printed to the build log unless the Pipeline is resuming, which should only happen when Jenkins starts (although bugs in historical builds that have a broken state could also cause them to attempt to resume when they shouldn't). The full stack trace would help clarify, if "jenkins.model.Jenkins.<init>" is part of the stack trace for the exception, then Jenkins is starting up.

            As a general update, my current understanding of this problem based on the data I have received is that in most cases it happens for Pipelines using the PERFORMANCE_OPTIMIZED durability level when Jenkins crashes. The PERFORMANCE_OPTIMIZED durability level makes no guarantees that Pipelines will be resumable if Jenkins crashes, so this behavior is expected in that case. There is supposed to be a more user-friendly error message explaining that the Pipeline cannot be resumed for these reasons rather than just the raw error of what exactly kept the Pipeline from resuming, but that is broken because of JENKINS-53358.

            I have a draft PR up to try to improve the messaging around this case, so that something like the following would be printed in these cases instead:

            Unable to resume because the Pipeline is using the PERFORMANCE_OPTIMIZED durability level but was not saved before Jenkins stopped. Did Jenkins crash? See https://www.jenkins.io/doc/book/pipeline/scaling-pipeline/ for details about Pipeline durability levels.

            If anyone has seen this issue with Pipelines that are not using the PERFORMANCE_OPTIMIZED durability level, or can show Jenkins system logs and service logs that show this issue occurring with a PERFORMANCE_OPTIMIZED Pipeline even with a normal Jenkins restart, with log messages as described in this comment showing that the Pipeline was persisted before shutdown, that would be very interesting and we should create new tickets to track those things because they would be distinct issues. For some of the stack traces here, it looks like there is a problem where CpsStepContext.isReady is resulting in Pipelines being resumed, which is strange; I am not sure how to reproduce those issues and they probably need to be investigated separately.

            Show
            dnusbaum Devin Nusbaum added a comment - - edited Mark Hollingsworth Thanks for the feedback. As far as I can tell though there is no way for the "Creating placeholder flownodes because failed loading originals" message to be printed to the build log unless the Pipeline is resuming, which should only happen when Jenkins starts (although bugs in historical builds that have a broken state could also cause them to attempt to resume when they shouldn't). The full stack trace would help clarify, if "jenkins.model.Jenkins.<init>" is part of the stack trace for the exception, then Jenkins is starting up. As a general update, my current understanding of this problem based on the data I have received is that in most cases it happens for Pipelines using the PERFORMANCE_OPTIMIZED durability level when Jenkins crashes. The PERFORMANCE_OPTIMIZED durability level makes no guarantees that Pipelines will be resumable if Jenkins crashes, so this behavior is expected in that case. There is supposed to be a more user-friendly error message explaining that the Pipeline cannot be resumed for these reasons rather than just the raw error of what exactly kept the Pipeline from resuming, but that is broken because of JENKINS-53358 . I have a draft PR up to try to improve the messaging around this case, so that something like the following would be printed in these cases instead: Unable to resume because the Pipeline is using the PERFORMANCE_OPTIMIZED durability level but was not saved before Jenkins stopped. Did Jenkins crash? See https://www.jenkins.io/doc/book/pipeline/scaling-pipeline/ for details about Pipeline durability levels. If anyone has seen this issue with Pipelines that are not using the PERFORMANCE_OPTIMIZED durability level, or can show Jenkins system logs and service logs that show this issue occurring with a PERFORMANCE_OPTIMIZED Pipeline even with a normal Jenkins restart, with log messages as described in this comment showing that the Pipeline was persisted before shutdown, that would be very interesting and we should create new tickets to track those things because they would be distinct issues. For some of the stack traces here, it looks like there is a problem where CpsStepContext.isReady is resulting in Pipelines being resumed, which is strange; I am not sure how to reproduce those issues and they probably need to be investigated separately.
            dnusbaum Devin Nusbaum made changes -
            Remote Link This issue links to "jenkinsci/workflow-cps-plugin#363 (Web Link)" [ 25349 ]
            Hide
            mhollingsworthcs Mark Hollingsworth added a comment -

             

             

            Devin Nusbaum so we definitely see jenkins.model.Jenkins.<init> as part of the trace, but I'm almost 100% certain that Jenkins did not startup (I would have been paged very early in the morning if it had). What I can tell you is that we have been reproducing this issue by terminating machines. 

            [2020-07-24T13:47:01.622Z] Cannot contact someNode: java.lang.InterruptedException
            Creating placeholder flownodes because failed loading originals.
            java.io.IOException: Tried to load head FlowNodes for execution Owner[someFolder/someJob/buildNumber:someFolder/someJob #buildNumber] but FlowNode was not found in storage for head id:FlowNodeId 1:1714
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679)
             at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:680)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:539)
             at hudson.model.RunMap.retrieve(RunMap.java:225)
             at hudson.model.RunMap.retrieve(RunMap.java:57)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
             at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
             at hudson.model.RunMap.getById(RunMap.java:205)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:929)
             at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:940)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57)
             at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
             at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
             at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178)
             at jenkins.model.Jenkins.<init>(Jenkins.java:1017)
             at hudson.model.Hudson.<init>(Hudson.java:85)
             at hudson.model.Hudson.<init>(Hudson.java:81)
             at hudson.WebAppMain$3.run(WebAppMain.java:262)

            What I can tell you is that we have been able to reproduce this issue by terminating agent machines too early.

            In this case our infrastructure killed the agent in job1 before the job finished completely and we repro the issue.

            Job2 is the following job that did not get cut off early. 

             

            Job 1
            16:23:36 [WS-CLEANUP] Deleting project workspace...
            16:23:36 [WS-CLEANUP] Deferred wipeout is used...
            16:23:36 [WS-CLEANUP] done
            [Pipeline] }
            [Pipeline] // node
            [Pipeline] }
            [Pipeline] // stage
            [Pipeline] }
            [Pipeline] // node
            [Pipeline] }
            ----------------------------
            Job 2
            16:16:42 [WS-CLEANUP] Deleting project workspace...
            16:16:42 [WS-CLEANUP] Deferred wipeout is used...
            16:16:42 [WS-CLEANUP] done
            [Pipeline] }
            [Pipeline] // node
            [Pipeline] }
            [Pipeline] // stage
            [Pipeline] }
            [Pipeline] // node
            [Pipeline] }
            [Pipeline] // parallel
            [Pipeline] }
            [Pipeline] // script
            [Pipeline] }
            [Pipeline] // stage
            [Pipeline] stage
            [Pipeline] { (Declarative: Post Actions)
            [Pipeline] node
            16:16:42 Running on a node
            [Pipeline] {
            [Pipeline] echo
            16:16:42 Reporting build status: UNSTABLE
            [Pipeline] notifyBitbucket
             notifying stuff
            [Pipeline] step
            16:16:44 Notifying Bitbucket
            16:16:44 Notified Bitbucket 
            [Pipeline] }
            [Pipeline] // node
            [Pipeline] }
            [Pipeline] // stage
            [Pipeline] }
            [Pipeline] // timestamps
            [Pipeline] }
            [Pipeline] // withEnv
            [Pipeline] End of Pipeline
            

            Hope that helps!

             

             

             

            Show
            mhollingsworthcs Mark Hollingsworth added a comment -     Devin Nusbaum  so we definitely see jenkins.model.Jenkins.<init> as part of the trace, but I'm almost 100% certain that Jenkins did not startup (I would have been paged very early in the morning if it had). What I can tell you is that we have been reproducing this issue by terminating machines.  [2020-07-24T13:47:01.622Z] Cannot contact someNode: java.lang.InterruptedException Creating placeholder flownodes because failed loading originals. java.io.IOException: Tried to load head FlowNodes for execution Owner[someFolder/someJob/buildNumber:someFolder/someJob #buildNumber] but FlowNode was not found in storage for head id:FlowNodeId 1:1714 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.initializeStorage(CpsFlowExecution.java:679) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:716) at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:680) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:539) at hudson.model.RunMap.retrieve(RunMap.java:225) at hudson.model.RunMap.retrieve(RunMap.java:57) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381) at hudson.model.RunMap.getById(RunMap.java:205) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:929) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:940) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:65) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:57) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:178) at jenkins.model.Jenkins.<init>(Jenkins.java:1017) at hudson.model.Hudson.<init>(Hudson.java:85) at hudson.model.Hudson.<init>(Hudson.java:81) at hudson.WebAppMain$3.run(WebAppMain.java:262) What I can tell you is that we have been able to reproduce this issue by terminating agent machines too early. In this case our infrastructure killed the agent in job1 before the job finished completely and we repro the issue. Job2 is the following job that did not get cut off early.    Job 1 16:23:36 [WS-CLEANUP] Deleting project workspace... 16:23:36 [WS-CLEANUP] Deferred wipeout is used... 16:23:36 [WS-CLEANUP] done [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // node [Pipeline] } ---------------------------- Job 2 16:16:42 [WS-CLEANUP] Deleting project workspace... 16:16:42 [WS-CLEANUP] Deferred wipeout is used... 16:16:42 [WS-CLEANUP] done [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // parallel [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Declarative: Post Actions) [Pipeline] node 16:16:42 Running on a node [Pipeline] { [Pipeline] echo 16:16:42 Reporting build status: UNSTABLE [Pipeline] notifyBitbucket notifying stuff [Pipeline] step 16:16:44 Notifying Bitbucket 16:16:44 Notified Bitbucket [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timestamps [Pipeline] } [Pipeline] // withEnv [Pipeline] End of Pipeline Hope that helps!      
            Hide
            simon_sudler Simon Sudler added a comment -

            I found the reason for the FlowNode errors on my system:

            Sep  6 00:29:20 ship kernel: [16183195.286559] Out of memory: Kill process 27501 (java) score 122 or sacrifice child
            Sep  6 00:29:20 ship kernel: [16183195.292176] Killed process 27501 (java) total-vm:21995844kB, anon-rss:5537216kB, file-rss:0kB, shmem-rss:0kB
            Sep  6 00:29:20 ship kernel: [16183196.039063] oom_reaper: reaped process 27501 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
            

            And it is very consistent. I checked the last 20 occurrences, every time the flow node error occurs, the OOM-killer does his work. Why the java process on the build-node requires around 20GB of memory is unclear, because the build itself does not have this huge memory requirements (maybe some other issue).

            Since the OOM kills the java process, there is no proper feedback from the FlowNode... maybe some sanity check if the build client process should do the trick and produce some helpful error message.

            I my case, more memory helped...

            Show
            simon_sudler Simon Sudler added a comment - I found the reason for the FlowNode errors on my system: Sep 6 00:29:20 ship kernel: [16183195.286559] Out of memory: Kill process 27501 (java) score 122 or sacrifice child Sep 6 00:29:20 ship kernel: [16183195.292176] Killed process 27501 (java) total-vm:21995844kB, anon-rss:5537216kB, file-rss:0kB, shmem-rss:0kB Sep 6 00:29:20 ship kernel: [16183196.039063] oom_reaper: reaped process 27501 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB And it is very consistent. I checked the last 20 occurrences, every time the flow node error occurs, the OOM-killer does his work. Why the java process on the build-node requires around 20GB of memory is unclear, because the build itself does not have this huge memory requirements (maybe some other issue). Since the OOM kills the java process, there is no proper feedback from the FlowNode... maybe some sanity check if the build client process should do the trick and produce some helpful error message. I my case, more memory helped...
            nishant_dani nishant dani made changes -
            Attachment flowNodeStore.xml [ 52772 ]
            Hide
            nishant_dani nishant dani added a comment - - edited

            I am able to consistently get this error. I have attached the jobs/yeticore/branches/master/builds/2/workflow-fallback/flowNodeStore.xml . My interpretation of this file could be wrong but it appears it it searching for flow node 49, but all the file has is flow nodes 51 and the 52.  This the file in workflow-fallback, the flow in workflow seems to have a single node with id 2.

            Totally blocked at this time, so willing to investigate further 

            Update -  I am running jenkins inside docker container and in Docker - Preferences - Resources - Memory , I increased memory from 2GB to 16GB (I restarted docker) and I was able to get a successful run.  Will keep monitoring.

             

             

             

             flowNodeStore.xml

            Show
            nishant_dani nishant dani added a comment - - edited I am able to consistently get this error. I have attached the jobs/yeticore/branches/master/builds/2/workflow-fallback/flowNodeStore.xml . My interpretation of this file could be wrong but it appears it it searching for flow node 49, but all the file has is flow nodes 51 and the 52.  This the file in workflow-fallback, the flow in workflow seems to have a single node with id 2. Totally blocked at this time, so willing to investigate further  Update -  I am running jenkins inside docker container and in Docker - Preferences - Resources - Memory , I increased memory from 2GB to 16GB (I restarted docker) and I was able to get a successful run.  Will keep monitoring.         flowNodeStore.xml

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              haorui658 Rui Hao
              Votes:
              36 Vote for this issue
              Watchers:
              49 Start watching this issue

                Dates

                Created:
                Updated: