Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-25890

Deadlock between RunMap and Queue after restart; StepContext.isReady impl acquires lock

XMLWordPrintable

      Found one Java-level deadlock:
      =============================
      "Thread-5":
        waiting to lock monitor 0x00007f0984170b38 (object 0x0000000706fe3aa8, a hudson.model.RunMap),
        which is held by "Jenkins initialization thread"
      "Jenkins initialization thread":
        waiting to lock monitor 0x00007f0988015128 (object 0x00000007066b46c0, a hudson.model.Queue),
        which is held by "Thread-5"
      
      Java stack information for the threads listed above:
      ===================================================
      "Thread-5":
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:688)
      	- waiting to lock <0x0000000706fe3aa8> (a hudson.model.RunMap)
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:671)
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.getById(AbstractLazyLoadRunMap.java:543)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:523)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:533)
      	at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getFlowExecution(CpsStepContext.java:386)
      	at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getProgramPromise(CpsStepContext.java:230)
      	at org.jenkinsci.plugins.workflow.cps.CpsStepContext.isReady(CpsStepContext.java:236)
      	at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask.run(ExecutorStepExecution.java:262)
      	at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask.getDisplayName(ExecutorStepExecution.java:281)
      	at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask.getFullDisplayName(ExecutorStepExecution.java:290)
      	at hudson.model.LoadBalancer$1.assignGreedily(LoadBalancer.java:107)
      	at hudson.model.LoadBalancer$1.map(LoadBalancer.java:97)
      	at hudson.model.LoadBalancer$2.map(LoadBalancer.java:148)
      	at hudson.model.Queue.maintain(Queue.java:1053)
      	- locked <0x00000007066b46c0> (a hudson.model.Queue)
      	at hudson.model.Queue$1.call(Queue.java:316)
      	at hudson.model.Queue$1.call(Queue.java:313)
      	at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:94)
      	at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:84)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:104)
      	at java.lang.Thread.run(Thread.java:745)
      "Jenkins initialization thread":
      	at hudson.model.Queue.schedule2(Queue.java:639)
      	- waiting to lock <0x00000007066b46c0> (a hudson.model.Queue)
      	at org.jenkinsci.plugins.workflow.support.pickles.ExecutorPickle.rehydrate(ExecutorPickle.java:67)
      	at org.jenkinsci.plugins.workflow.support.pickles.serialization.PickleResolver.rehydrate(PickleResolver.java:68)
      	at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverReader.restorePickles(RiverReader.java:128)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.loadProgramAsync(CpsFlowExecution.java:401)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:379)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:300)
      	at hudson.model.RunMap.retrieve(RunMap.java:219)
      	at hudson.model.RunMap.retrieve(RunMap.java:56)
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:688)
      	- locked <0x0000000706fe3aa8> (a hudson.model.RunMap)
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:671)
      	at jenkins.model.lazy.AbstractLazyLoadRunMap.getById(AbstractLazyLoadRunMap.java:543)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:523)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:533)
      	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:59)
      	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$1.computeNext(FlowExecutionList.java:51)
      	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
      	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
      	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl.onLoaded(FlowExecutionList.java:165)
      	at jenkins.model.Jenkins.<init>(Jenkins.java:845)
      	at hudson.model.Hudson.<init>(Hudson.java:82)
      	at hudson.model.Hudson.<init>(Hudson.java:78)
      	at hudson.WebAppMain$3.run(WebAppMain.java:222)
      

      Ironically, StepContext.isReady is what is supposed to be breaking deadlocks, yet here it is acquiring a lock.

      Since getFlowExecution may block, I think getProgramPromise should be made to return a future which encompasses both getting the execution, and its programPromise.

            jglick Jesse Glick
            jglick Jesse Glick
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: