Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33219

Job.updateNextBuildNumber can cause deadlock

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      The attached thread dump shows two deadlocked threads:

      • "jenkins.util.Timer 7" daemon prio=5 BLOCKED
      • "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED)

      Jenkins 1.650

      This happens as follows:

      1. I use the Job DSL plugin to create a bunch of jobs. Many have SCM Poll trigger enabled, causing them to run instantly as soon as they are created (possibly also when updated, looking at the stacktrace).
      2. As part of job creation, Job.updateNextBuildNumber is called (Next Build Number plugin integrated with Job DSL)

      The two events above create deadlock.

      The first one locks Job since updateNextBuildNumber() is synchronized. Then it calls AbstractLazyLoadRunMap.getByNumber(). The synchronized block in this method is new as of 1.646(https://github.com/jenkinsci/jenkins/commit/d5167025a204750633c931ea8c1fff8d7561ab9c#diff-383116e240993025e5b727359e61db09)

      The second one calls AbstractLazyLoadRunMap.getByNumber() first, which causes a call to AbstractProject.save(), but Job is an AbstractProject... so, deadlock.

      Here are the relevant parts for convenience, bold elements are causing deadlock:

      "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED
      jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:356)
      jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
      jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
      jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
      hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
      hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
      hudson.model.Job.updateNextBuildNumber(Job.java:422)
      org.jvnet.hudson.plugins.nextbuildnumber.JobDslExtension.notifyItemUpdated(JobDslExtension.java:30)

      "jenkins.util.Timer 7" daemon prio=5 BLOCKED
      hudson.model.AbstractProject.save(AbstractProject.java:305)
      hudson.model.Job.addProperty(Job.java:523)
      hudson.model.AbstractProject.addProperty(AbstractProject.java:785)
      hudson.plugins.disk_usage.DiskUsageUtil.addProperty(DiskUsageUtil.java:58)
      hudson.plugins.disk_usage.BuildDiskUsageAction.<init>(BuildDiskUsageAction.java:38)
      hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:31)
      hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:21)
      hudson.model.Actionable.createFor(Actionable.java:107)
      hudson.model.Actionable.getAllActions(Actionable.java:98)
      hudson.model.Run.onLoad(Run.java:343)
      hudson.model.RunMap.retrieve(RunMap.java:224)
      hudson.model.RunMap.retrieve(RunMap.java:56)
      jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:479)
      jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:461)
      jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:367)
      jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
      jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
      jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
      hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
      hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
      hudson.views.AbstractBuildTrendFilter.matches(AbstractBuildTrendFilter.java:71)
      hudson.views.AbstractIncludeExcludeJobFilter.doFilter(AbstractIncludeExcludeJobFilter.java:68)
      hudson.views.AbstractIncludeExcludeJobFilter.filter(AbstractIncludeExcludeJobFilter.java:57)
      hudson.model.ListView.getItems(ListView.java:195)
      hudson.model.ListView.getItems(ListView.java:67)
      jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.isJobInView(ViewBasedJobInclusionStrategy.java:182)
      jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.contains(ViewBasedJobInclusionStrategy.java:149)
      jenkins.advancedqueue.PriorityConfiguration.getJobGroup(PriorityConfiguration.java:241)
      jenkins.advancedqueue.PriorityConfiguration.getPriorityInternal(PriorityConfiguration.java:225)
      jenkins.advancedqueue.PriorityConfiguration.getPriority(PriorityConfiguration.java:203)
      jenkins.advancedqueue.sorter.AdvancedQueueSorter.onNewItem(AdvancedQueueSorter.java:136)
      jenkins.advancedqueue.sorter.AdvancedQueueSorterQueueListener.onEnterWaiting(AdvancedQueueSorterQueueListener.java:46)
      hudson.model.Queue$WaitingItem.enter(Queue.java:2348)
      hudson.model.Queue.scheduleInternal(Queue.java:599)
      hudson.model.Queue.schedule2(Queue.java:555)
      jenkins.model.ParameterizedJobMixIn.scheduleBuild2(ParameterizedJobMixIn.java:138)
      jenkins.model.ParameterizedJobMixIn.scheduleBuild(ParameterizedJobMixIn.java:94)
      hudson.model.AbstractProject.scheduleBuild(AbstractProject.java:836)

      My Job DSL setup is rather complicated and I have not been able to extract a simple test for this - if I do, I'll attach it.

      Update: 1.645 also deadlocks. AbstractLazyLoadRunMap.load() is synchronized. I am confused about why this started happening once I upgraded from 1.645 but continues to happen once I downgraded back. Maybe the newer version of some other plugin is making this more likely (ie SVN plugin polls more agressively, etc)

        Attachments

          Issue Links

            Activity

            akom Alexander Komarov created issue -
            akom Alexander Komarov made changes -
            Field Original Value New Value
            Description The attached thread dump shows two deadlocked threads:

            "jenkins.util.Timer [#7]" daemon prio=5 BLOCKED
            "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED)

            Jenkins 1.650

            This happens as follows:

            # I use the *Job DSL plugin* to create a bunch of jobs. Many have SCM Poll trigger enabled, causing them to run instantly as soon as they are created (possibly also when updated, looking at the stacktrace).
            # As part of job creation, *Job.updateNextBuildNumber* is called (Next Build Number plugin integrated with Job DSL)

            The two events above create deadlock.

            The first one locks *AbstractLazyLoadRunMap.getByNumber()*. The synchronized block in this method is new as of *1.646 *(https://github.com/jenkinsci/jenkins/commit/d5167025a204750633c931ea8c1fff8d7561ab9c#diff-383116e240993025e5b727359e61db09)

            The second one locks Job, aka AbstractProject.

            Then the two attempt to acquire the other lock and deadlock. Here are the relevant parts for convenience:
            {quote}
            "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED
            jenkins.model.lazy.AbstractLazyLoadRunMap.*getByNumber*(AbstractLazyLoadRunMap.java:356)
            jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
            jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
            jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
            hudson.model.Job.updateNextBuildNumber(Job.java:422)
            org.jvnet.hudson.plugins.nextbuildnumber.JobDslExtension.notifyItemUpdated(JobDslExtension.java:30)
            {quote}

            {quote}
            "jenkins.util.Timer [#7]" daemon prio=5 BLOCKED
            hudson.model.*AbstractProject*.save(AbstractProject.java:305)
            hudson.model.Job.addProperty(Job.java:523)
            hudson.model.AbstractProject.addProperty(AbstractProject.java:785)
            hudson.plugins.disk_usage.DiskUsageUtil.addProperty(DiskUsageUtil.java:58)
            hudson.plugins.disk_usage.BuildDiskUsageAction.<init>(BuildDiskUsageAction.java:38)
            hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:31)
            hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:21)
            hudson.model.Actionable.createFor(Actionable.java:107)
            hudson.model.Actionable.getAllActions(Actionable.java:98)
            hudson.model.Run.onLoad(Run.java:343)
            hudson.model.RunMap.retrieve(RunMap.java:224)
            hudson.model.RunMap.retrieve(RunMap.java:56)
            jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:479)
            jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:461)
            jenkins.model.lazy.AbstractLazyLoadRunMap.*getByNumber*(AbstractLazyLoadRunMap.java:367)
            jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
            jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
            jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
            hudson.views.AbstractBuildTrendFilter.matches(AbstractBuildTrendFilter.java:71)
            hudson.views.AbstractIncludeExcludeJobFilter.doFilter(AbstractIncludeExcludeJobFilter.java:68)
            hudson.views.AbstractIncludeExcludeJobFilter.filter(AbstractIncludeExcludeJobFilter.java:57)
            hudson.model.ListView.getItems(ListView.java:195)
            hudson.model.ListView.getItems(ListView.java:67)
            jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.isJobInView(ViewBasedJobInclusionStrategy.java:182)
            jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.contains(ViewBasedJobInclusionStrategy.java:149)
            jenkins.advancedqueue.PriorityConfiguration.getJobGroup(PriorityConfiguration.java:241)
            jenkins.advancedqueue.PriorityConfiguration.getPriorityInternal(PriorityConfiguration.java:225)
            jenkins.advancedqueue.PriorityConfiguration.getPriority(PriorityConfiguration.java:203)
            jenkins.advancedqueue.sorter.AdvancedQueueSorter.onNewItem(AdvancedQueueSorter.java:136)
            jenkins.advancedqueue.sorter.AdvancedQueueSorterQueueListener.onEnterWaiting(AdvancedQueueSorterQueueListener.java:46)
            hudson.model.Queue$WaitingItem.enter(Queue.java:2348)
            hudson.model.Queue.scheduleInternal(Queue.java:599)
            hudson.model.Queue.schedule2(Queue.java:555)
            jenkins.model.ParameterizedJobMixIn.scheduleBuild2(ParameterizedJobMixIn.java:138)
            jenkins.model.ParameterizedJobMixIn.scheduleBuild(ParameterizedJobMixIn.java:94)
            hudson.model.AbstractProject.scheduleBuild(AbstractProject.java:836)
            {quote}

            My Job DSL setup is rather complicated and I have not been able to extract a simple test for this - if I do, I'll attach it.
            The attached thread dump shows two deadlocked threads:

            "jenkins.util.Timer [#7]" daemon prio=5 BLOCKED
            "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED)

            Jenkins 1.650

            This happens as follows:

            # I use the *Job DSL plugin* to create a bunch of jobs. Many have SCM Poll trigger enabled, causing them to run instantly as soon as they are created (possibly also when updated, looking at the stacktrace).
            # As part of job creation, *Job.updateNextBuildNumber* is called (Next Build Number plugin integrated with Job DSL)

            The two events above create deadlock.

            The first one locks *AbstractLazyLoadRunMap.getByNumber()*. The synchronized block in this method is new as of *1.646 *(https://github.com/jenkinsci/jenkins/commit/d5167025a204750633c931ea8c1fff8d7561ab9c#diff-383116e240993025e5b727359e61db09)

            The second one locks Job, aka AbstractProject.

            Then the two attempt to acquire the other lock and deadlock. Here are the relevant parts for convenience:
            {quote}
            "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED
            jenkins.model.lazy.AbstractLazyLoadRunMap.*getByNumber*(AbstractLazyLoadRunMap.java:356)
            jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
            jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
            jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
            hudson.model.*Job.updateNextBuildNumbe*r(Job.java:422)
            org.jvnet.hudson.plugins.nextbuildnumber.JobDslExtension.notifyItemUpdated(JobDslExtension.java:30)
            {quote}

            {quote}
            "jenkins.util.Timer [#7]" daemon prio=5 BLOCKED
            hudson.model.*AbstractProject*.save(AbstractProject.java:305)
            hudson.model.Job.addProperty(Job.java:523)
            hudson.model.AbstractProject.addProperty(AbstractProject.java:785)
            hudson.plugins.disk_usage.DiskUsageUtil.addProperty(DiskUsageUtil.java:58)
            hudson.plugins.disk_usage.BuildDiskUsageAction.<init>(BuildDiskUsageAction.java:38)
            hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:31)
            hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:21)
            hudson.model.Actionable.createFor(Actionable.java:107)
            hudson.model.Actionable.getAllActions(Actionable.java:98)
            hudson.model.Run.onLoad(Run.java:343)
            hudson.model.RunMap.retrieve(RunMap.java:224)
            hudson.model.RunMap.retrieve(RunMap.java:56)
            jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:479)
            jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:461)
            jenkins.model.lazy.AbstractLazyLoadRunMap.*getByNumber*(AbstractLazyLoadRunMap.java:367)
            jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
            jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
            jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
            hudson.views.AbstractBuildTrendFilter.matches(AbstractBuildTrendFilter.java:71)
            hudson.views.AbstractIncludeExcludeJobFilter.doFilter(AbstractIncludeExcludeJobFilter.java:68)
            hudson.views.AbstractIncludeExcludeJobFilter.filter(AbstractIncludeExcludeJobFilter.java:57)
            hudson.model.ListView.getItems(ListView.java:195)
            hudson.model.ListView.getItems(ListView.java:67)
            jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.isJobInView(ViewBasedJobInclusionStrategy.java:182)
            jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.contains(ViewBasedJobInclusionStrategy.java:149)
            jenkins.advancedqueue.PriorityConfiguration.getJobGroup(PriorityConfiguration.java:241)
            jenkins.advancedqueue.PriorityConfiguration.getPriorityInternal(PriorityConfiguration.java:225)
            jenkins.advancedqueue.PriorityConfiguration.getPriority(PriorityConfiguration.java:203)
            jenkins.advancedqueue.sorter.AdvancedQueueSorter.onNewItem(AdvancedQueueSorter.java:136)
            jenkins.advancedqueue.sorter.AdvancedQueueSorterQueueListener.onEnterWaiting(AdvancedQueueSorterQueueListener.java:46)
            hudson.model.Queue$WaitingItem.enter(Queue.java:2348)
            hudson.model.Queue.scheduleInternal(Queue.java:599)
            hudson.model.Queue.schedule2(Queue.java:555)
            jenkins.model.ParameterizedJobMixIn.scheduleBuild2(ParameterizedJobMixIn.java:138)
            jenkins.model.ParameterizedJobMixIn.scheduleBuild(ParameterizedJobMixIn.java:94)
            hudson.model.AbstractProject.scheduleBuild(AbstractProject.java:836)
            {quote}

            My Job DSL setup is rather complicated and I have not been able to extract a simple test for this - if I do, I'll attach it.
            akom Alexander Komarov made changes -
            Description The attached thread dump shows two deadlocked threads:

            "jenkins.util.Timer [#7]" daemon prio=5 BLOCKED
            "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED)

            Jenkins 1.650

            This happens as follows:

            # I use the *Job DSL plugin* to create a bunch of jobs. Many have SCM Poll trigger enabled, causing them to run instantly as soon as they are created (possibly also when updated, looking at the stacktrace).
            # As part of job creation, *Job.updateNextBuildNumber* is called (Next Build Number plugin integrated with Job DSL)

            The two events above create deadlock.

            The first one locks *AbstractLazyLoadRunMap.getByNumber()*. The synchronized block in this method is new as of *1.646 *(https://github.com/jenkinsci/jenkins/commit/d5167025a204750633c931ea8c1fff8d7561ab9c#diff-383116e240993025e5b727359e61db09)

            The second one locks Job, aka AbstractProject.

            Then the two attempt to acquire the other lock and deadlock. Here are the relevant parts for convenience:
            {quote}
            "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED
            jenkins.model.lazy.AbstractLazyLoadRunMap.*getByNumber*(AbstractLazyLoadRunMap.java:356)
            jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
            jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
            jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
            hudson.model.*Job.updateNextBuildNumbe*r(Job.java:422)
            org.jvnet.hudson.plugins.nextbuildnumber.JobDslExtension.notifyItemUpdated(JobDslExtension.java:30)
            {quote}

            {quote}
            "jenkins.util.Timer [#7]" daemon prio=5 BLOCKED
            hudson.model.*AbstractProject*.save(AbstractProject.java:305)
            hudson.model.Job.addProperty(Job.java:523)
            hudson.model.AbstractProject.addProperty(AbstractProject.java:785)
            hudson.plugins.disk_usage.DiskUsageUtil.addProperty(DiskUsageUtil.java:58)
            hudson.plugins.disk_usage.BuildDiskUsageAction.<init>(BuildDiskUsageAction.java:38)
            hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:31)
            hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:21)
            hudson.model.Actionable.createFor(Actionable.java:107)
            hudson.model.Actionable.getAllActions(Actionable.java:98)
            hudson.model.Run.onLoad(Run.java:343)
            hudson.model.RunMap.retrieve(RunMap.java:224)
            hudson.model.RunMap.retrieve(RunMap.java:56)
            jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:479)
            jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:461)
            jenkins.model.lazy.AbstractLazyLoadRunMap.*getByNumber*(AbstractLazyLoadRunMap.java:367)
            jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
            jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
            jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
            hudson.views.AbstractBuildTrendFilter.matches(AbstractBuildTrendFilter.java:71)
            hudson.views.AbstractIncludeExcludeJobFilter.doFilter(AbstractIncludeExcludeJobFilter.java:68)
            hudson.views.AbstractIncludeExcludeJobFilter.filter(AbstractIncludeExcludeJobFilter.java:57)
            hudson.model.ListView.getItems(ListView.java:195)
            hudson.model.ListView.getItems(ListView.java:67)
            jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.isJobInView(ViewBasedJobInclusionStrategy.java:182)
            jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.contains(ViewBasedJobInclusionStrategy.java:149)
            jenkins.advancedqueue.PriorityConfiguration.getJobGroup(PriorityConfiguration.java:241)
            jenkins.advancedqueue.PriorityConfiguration.getPriorityInternal(PriorityConfiguration.java:225)
            jenkins.advancedqueue.PriorityConfiguration.getPriority(PriorityConfiguration.java:203)
            jenkins.advancedqueue.sorter.AdvancedQueueSorter.onNewItem(AdvancedQueueSorter.java:136)
            jenkins.advancedqueue.sorter.AdvancedQueueSorterQueueListener.onEnterWaiting(AdvancedQueueSorterQueueListener.java:46)
            hudson.model.Queue$WaitingItem.enter(Queue.java:2348)
            hudson.model.Queue.scheduleInternal(Queue.java:599)
            hudson.model.Queue.schedule2(Queue.java:555)
            jenkins.model.ParameterizedJobMixIn.scheduleBuild2(ParameterizedJobMixIn.java:138)
            jenkins.model.ParameterizedJobMixIn.scheduleBuild(ParameterizedJobMixIn.java:94)
            hudson.model.AbstractProject.scheduleBuild(AbstractProject.java:836)
            {quote}

            My Job DSL setup is rather complicated and I have not been able to extract a simple test for this - if I do, I'll attach it.
            The attached thread dump shows two deadlocked threads:

            "jenkins.util.Timer [#7]" daemon prio=5 BLOCKED
            "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED)

            Jenkins 1.650

            This happens as follows:

            # I use the *Job DSL plugin* to create a bunch of jobs. Many have SCM Poll trigger enabled, causing them to run instantly as soon as they are created (possibly also when updated, looking at the stacktrace).
            # As part of job creation, *Job.updateNextBuildNumber* is called (Next Build Number plugin integrated with Job DSL)

            The two events above create deadlock.

            The first one locks *AbstractLazyLoadRunMap.getByNumber()*. The synchronized block in this method is new as of *1.646 *(https://github.com/jenkinsci/jenkins/commit/d5167025a204750633c931ea8c1fff8d7561ab9c#diff-383116e240993025e5b727359e61db09)

            The second one locks Job, aka AbstractProject.

            Then the two attempt to acquire the other lock and deadlock. Here are the relevant parts for convenience:
            {quote}
            "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED
            jenkins.model.lazy.AbstractLazyLoadRunMap.*getByNumber*(AbstractLazyLoadRunMap.java:356)
            jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
            jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
            jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
            hudson.model.*Job.updateNextBuildNumber*(Job.java:422)
            org.jvnet.hudson.plugins.nextbuildnumber.JobDslExtension.notifyItemUpdated(JobDslExtension.java:30)
            {quote}

            {quote}
            "jenkins.util.Timer [#7]" daemon prio=5 BLOCKED
            hudson.model.*AbstractProject*.save(AbstractProject.java:305)
            hudson.model.Job.addProperty(Job.java:523)
            hudson.model.AbstractProject.addProperty(AbstractProject.java:785)
            hudson.plugins.disk_usage.DiskUsageUtil.addProperty(DiskUsageUtil.java:58)
            hudson.plugins.disk_usage.BuildDiskUsageAction.<init>(BuildDiskUsageAction.java:38)
            hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:31)
            hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:21)
            hudson.model.Actionable.createFor(Actionable.java:107)
            hudson.model.Actionable.getAllActions(Actionable.java:98)
            hudson.model.Run.onLoad(Run.java:343)
            hudson.model.RunMap.retrieve(RunMap.java:224)
            hudson.model.RunMap.retrieve(RunMap.java:56)
            jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:479)
            jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:461)
            jenkins.model.lazy.AbstractLazyLoadRunMap.*getByNumber*(AbstractLazyLoadRunMap.java:367)
            jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
            jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
            jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
            hudson.views.AbstractBuildTrendFilter.matches(AbstractBuildTrendFilter.java:71)
            hudson.views.AbstractIncludeExcludeJobFilter.doFilter(AbstractIncludeExcludeJobFilter.java:68)
            hudson.views.AbstractIncludeExcludeJobFilter.filter(AbstractIncludeExcludeJobFilter.java:57)
            hudson.model.ListView.getItems(ListView.java:195)
            hudson.model.ListView.getItems(ListView.java:67)
            jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.isJobInView(ViewBasedJobInclusionStrategy.java:182)
            jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.contains(ViewBasedJobInclusionStrategy.java:149)
            jenkins.advancedqueue.PriorityConfiguration.getJobGroup(PriorityConfiguration.java:241)
            jenkins.advancedqueue.PriorityConfiguration.getPriorityInternal(PriorityConfiguration.java:225)
            jenkins.advancedqueue.PriorityConfiguration.getPriority(PriorityConfiguration.java:203)
            jenkins.advancedqueue.sorter.AdvancedQueueSorter.onNewItem(AdvancedQueueSorter.java:136)
            jenkins.advancedqueue.sorter.AdvancedQueueSorterQueueListener.onEnterWaiting(AdvancedQueueSorterQueueListener.java:46)
            hudson.model.Queue$WaitingItem.enter(Queue.java:2348)
            hudson.model.Queue.scheduleInternal(Queue.java:599)
            hudson.model.Queue.schedule2(Queue.java:555)
            jenkins.model.ParameterizedJobMixIn.scheduleBuild2(ParameterizedJobMixIn.java:138)
            jenkins.model.ParameterizedJobMixIn.scheduleBuild(ParameterizedJobMixIn.java:94)
            hudson.model.AbstractProject.scheduleBuild(AbstractProject.java:836)
            {quote}

            My Job DSL setup is rather complicated and I have not been able to extract a simple test for this - if I do, I'll attach it.
            akom Alexander Komarov made changes -
            Description The attached thread dump shows two deadlocked threads:

            "jenkins.util.Timer [#7]" daemon prio=5 BLOCKED
            "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED)

            Jenkins 1.650

            This happens as follows:

            # I use the *Job DSL plugin* to create a bunch of jobs. Many have SCM Poll trigger enabled, causing them to run instantly as soon as they are created (possibly also when updated, looking at the stacktrace).
            # As part of job creation, *Job.updateNextBuildNumber* is called (Next Build Number plugin integrated with Job DSL)

            The two events above create deadlock.

            The first one locks *AbstractLazyLoadRunMap.getByNumber()*. The synchronized block in this method is new as of *1.646 *(https://github.com/jenkinsci/jenkins/commit/d5167025a204750633c931ea8c1fff8d7561ab9c#diff-383116e240993025e5b727359e61db09)

            The second one locks Job, aka AbstractProject.

            Then the two attempt to acquire the other lock and deadlock. Here are the relevant parts for convenience:
            {quote}
            "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED
            jenkins.model.lazy.AbstractLazyLoadRunMap.*getByNumber*(AbstractLazyLoadRunMap.java:356)
            jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
            jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
            jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
            hudson.model.*Job.updateNextBuildNumber*(Job.java:422)
            org.jvnet.hudson.plugins.nextbuildnumber.JobDslExtension.notifyItemUpdated(JobDslExtension.java:30)
            {quote}

            {quote}
            "jenkins.util.Timer [#7]" daemon prio=5 BLOCKED
            hudson.model.*AbstractProject*.save(AbstractProject.java:305)
            hudson.model.Job.addProperty(Job.java:523)
            hudson.model.AbstractProject.addProperty(AbstractProject.java:785)
            hudson.plugins.disk_usage.DiskUsageUtil.addProperty(DiskUsageUtil.java:58)
            hudson.plugins.disk_usage.BuildDiskUsageAction.<init>(BuildDiskUsageAction.java:38)
            hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:31)
            hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:21)
            hudson.model.Actionable.createFor(Actionable.java:107)
            hudson.model.Actionable.getAllActions(Actionable.java:98)
            hudson.model.Run.onLoad(Run.java:343)
            hudson.model.RunMap.retrieve(RunMap.java:224)
            hudson.model.RunMap.retrieve(RunMap.java:56)
            jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:479)
            jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:461)
            jenkins.model.lazy.AbstractLazyLoadRunMap.*getByNumber*(AbstractLazyLoadRunMap.java:367)
            jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
            jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
            jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
            hudson.views.AbstractBuildTrendFilter.matches(AbstractBuildTrendFilter.java:71)
            hudson.views.AbstractIncludeExcludeJobFilter.doFilter(AbstractIncludeExcludeJobFilter.java:68)
            hudson.views.AbstractIncludeExcludeJobFilter.filter(AbstractIncludeExcludeJobFilter.java:57)
            hudson.model.ListView.getItems(ListView.java:195)
            hudson.model.ListView.getItems(ListView.java:67)
            jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.isJobInView(ViewBasedJobInclusionStrategy.java:182)
            jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.contains(ViewBasedJobInclusionStrategy.java:149)
            jenkins.advancedqueue.PriorityConfiguration.getJobGroup(PriorityConfiguration.java:241)
            jenkins.advancedqueue.PriorityConfiguration.getPriorityInternal(PriorityConfiguration.java:225)
            jenkins.advancedqueue.PriorityConfiguration.getPriority(PriorityConfiguration.java:203)
            jenkins.advancedqueue.sorter.AdvancedQueueSorter.onNewItem(AdvancedQueueSorter.java:136)
            jenkins.advancedqueue.sorter.AdvancedQueueSorterQueueListener.onEnterWaiting(AdvancedQueueSorterQueueListener.java:46)
            hudson.model.Queue$WaitingItem.enter(Queue.java:2348)
            hudson.model.Queue.scheduleInternal(Queue.java:599)
            hudson.model.Queue.schedule2(Queue.java:555)
            jenkins.model.ParameterizedJobMixIn.scheduleBuild2(ParameterizedJobMixIn.java:138)
            jenkins.model.ParameterizedJobMixIn.scheduleBuild(ParameterizedJobMixIn.java:94)
            hudson.model.AbstractProject.scheduleBuild(AbstractProject.java:836)
            {quote}

            My Job DSL setup is rather complicated and I have not been able to extract a simple test for this - if I do, I'll attach it.
            The attached thread dump shows two deadlocked threads:

            "jenkins.util.Timer [#7]" daemon prio=5 BLOCKED
            "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED)

            Jenkins 1.650

            This happens as follows:

            # I use the *Job DSL plugin* to create a bunch of jobs. Many have SCM Poll trigger enabled, causing them to run instantly as soon as they are created (possibly also when updated, looking at the stacktrace).
            # As part of job creation, *Job.updateNextBuildNumber* is called (Next Build Number plugin integrated with Job DSL)

            The two events above create deadlock.

            The first one locks *Job* since updateNextBuildNumber() is synchronized. Then it calls *AbstractLazyLoadRunMap.getByNumber()*. The synchronized block in this method is new as of *1.646 *(https://github.com/jenkinsci/jenkins/commit/d5167025a204750633c931ea8c1fff8d7561ab9c#diff-383116e240993025e5b727359e61db09)

            The second one calls *AbstractLazyLoadRunMap.getByNumber()* first, which causes a call to AbstractProject.save(), but Job is an AbstractProject... so, deadlock.

            Here are the relevant parts for convenience, bold elements are causing deadlock:
            {quote}
            "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED
            jenkins.model.lazy.*AbstractLazyLoadRunMap.getByNumber*(AbstractLazyLoadRunMap.java:356)
            jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
            jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
            jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
            hudson.model.*Job.updateNextBuildNumber*(Job.java:422)
            org.jvnet.hudson.plugins.nextbuildnumber.JobDslExtension.notifyItemUpdated(JobDslExtension.java:30)
            {quote}

            {quote}
            "jenkins.util.Timer [#7]" daemon prio=5 BLOCKED
            hudson.model.*AbstractProject.save*(AbstractProject.java:305)
            hudson.model.Job.addProperty(Job.java:523)
            hudson.model.AbstractProject.addProperty(AbstractProject.java:785)
            hudson.plugins.disk_usage.DiskUsageUtil.addProperty(DiskUsageUtil.java:58)
            hudson.plugins.disk_usage.BuildDiskUsageAction.<init>(BuildDiskUsageAction.java:38)
            hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:31)
            hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:21)
            hudson.model.Actionable.createFor(Actionable.java:107)
            hudson.model.Actionable.getAllActions(Actionable.java:98)
            hudson.model.Run.onLoad(Run.java:343)
            hudson.model.RunMap.retrieve(RunMap.java:224)
            hudson.model.RunMap.retrieve(RunMap.java:56)
            jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:479)
            jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:461)
            jenkins.model.lazy.*AbstractLazyLoadRunMap.getByNumber*(AbstractLazyLoadRunMap.java:367)
            jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
            jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
            jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
            hudson.views.AbstractBuildTrendFilter.matches(AbstractBuildTrendFilter.java:71)
            hudson.views.AbstractIncludeExcludeJobFilter.doFilter(AbstractIncludeExcludeJobFilter.java:68)
            hudson.views.AbstractIncludeExcludeJobFilter.filter(AbstractIncludeExcludeJobFilter.java:57)
            hudson.model.ListView.getItems(ListView.java:195)
            hudson.model.ListView.getItems(ListView.java:67)
            jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.isJobInView(ViewBasedJobInclusionStrategy.java:182)
            jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.contains(ViewBasedJobInclusionStrategy.java:149)
            jenkins.advancedqueue.PriorityConfiguration.getJobGroup(PriorityConfiguration.java:241)
            jenkins.advancedqueue.PriorityConfiguration.getPriorityInternal(PriorityConfiguration.java:225)
            jenkins.advancedqueue.PriorityConfiguration.getPriority(PriorityConfiguration.java:203)
            jenkins.advancedqueue.sorter.AdvancedQueueSorter.onNewItem(AdvancedQueueSorter.java:136)
            jenkins.advancedqueue.sorter.AdvancedQueueSorterQueueListener.onEnterWaiting(AdvancedQueueSorterQueueListener.java:46)
            hudson.model.Queue$WaitingItem.enter(Queue.java:2348)
            hudson.model.Queue.scheduleInternal(Queue.java:599)
            hudson.model.Queue.schedule2(Queue.java:555)
            jenkins.model.ParameterizedJobMixIn.scheduleBuild2(ParameterizedJobMixIn.java:138)
            jenkins.model.ParameterizedJobMixIn.scheduleBuild(ParameterizedJobMixIn.java:94)
            hudson.model.AbstractProject.scheduleBuild(AbstractProject.java:836)
            {quote}

            My Job DSL setup is rather complicated and I have not been able to extract a simple test for this - if I do, I'll attach it.
            akom Alexander Komarov made changes -
            Description The attached thread dump shows two deadlocked threads:

            "jenkins.util.Timer [#7]" daemon prio=5 BLOCKED
            "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED)

            Jenkins 1.650

            This happens as follows:

            # I use the *Job DSL plugin* to create a bunch of jobs. Many have SCM Poll trigger enabled, causing them to run instantly as soon as they are created (possibly also when updated, looking at the stacktrace).
            # As part of job creation, *Job.updateNextBuildNumber* is called (Next Build Number plugin integrated with Job DSL)

            The two events above create deadlock.

            The first one locks *Job* since updateNextBuildNumber() is synchronized. Then it calls *AbstractLazyLoadRunMap.getByNumber()*. The synchronized block in this method is new as of *1.646 *(https://github.com/jenkinsci/jenkins/commit/d5167025a204750633c931ea8c1fff8d7561ab9c#diff-383116e240993025e5b727359e61db09)

            The second one calls *AbstractLazyLoadRunMap.getByNumber()* first, which causes a call to AbstractProject.save(), but Job is an AbstractProject... so, deadlock.

            Here are the relevant parts for convenience, bold elements are causing deadlock:
            {quote}
            "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED
            jenkins.model.lazy.*AbstractLazyLoadRunMap.getByNumber*(AbstractLazyLoadRunMap.java:356)
            jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
            jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
            jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
            hudson.model.*Job.updateNextBuildNumber*(Job.java:422)
            org.jvnet.hudson.plugins.nextbuildnumber.JobDslExtension.notifyItemUpdated(JobDslExtension.java:30)
            {quote}

            {quote}
            "jenkins.util.Timer [#7]" daemon prio=5 BLOCKED
            hudson.model.*AbstractProject.save*(AbstractProject.java:305)
            hudson.model.Job.addProperty(Job.java:523)
            hudson.model.AbstractProject.addProperty(AbstractProject.java:785)
            hudson.plugins.disk_usage.DiskUsageUtil.addProperty(DiskUsageUtil.java:58)
            hudson.plugins.disk_usage.BuildDiskUsageAction.<init>(BuildDiskUsageAction.java:38)
            hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:31)
            hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:21)
            hudson.model.Actionable.createFor(Actionable.java:107)
            hudson.model.Actionable.getAllActions(Actionable.java:98)
            hudson.model.Run.onLoad(Run.java:343)
            hudson.model.RunMap.retrieve(RunMap.java:224)
            hudson.model.RunMap.retrieve(RunMap.java:56)
            jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:479)
            jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:461)
            jenkins.model.lazy.*AbstractLazyLoadRunMap.getByNumber*(AbstractLazyLoadRunMap.java:367)
            jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
            jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
            jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
            hudson.views.AbstractBuildTrendFilter.matches(AbstractBuildTrendFilter.java:71)
            hudson.views.AbstractIncludeExcludeJobFilter.doFilter(AbstractIncludeExcludeJobFilter.java:68)
            hudson.views.AbstractIncludeExcludeJobFilter.filter(AbstractIncludeExcludeJobFilter.java:57)
            hudson.model.ListView.getItems(ListView.java:195)
            hudson.model.ListView.getItems(ListView.java:67)
            jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.isJobInView(ViewBasedJobInclusionStrategy.java:182)
            jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.contains(ViewBasedJobInclusionStrategy.java:149)
            jenkins.advancedqueue.PriorityConfiguration.getJobGroup(PriorityConfiguration.java:241)
            jenkins.advancedqueue.PriorityConfiguration.getPriorityInternal(PriorityConfiguration.java:225)
            jenkins.advancedqueue.PriorityConfiguration.getPriority(PriorityConfiguration.java:203)
            jenkins.advancedqueue.sorter.AdvancedQueueSorter.onNewItem(AdvancedQueueSorter.java:136)
            jenkins.advancedqueue.sorter.AdvancedQueueSorterQueueListener.onEnterWaiting(AdvancedQueueSorterQueueListener.java:46)
            hudson.model.Queue$WaitingItem.enter(Queue.java:2348)
            hudson.model.Queue.scheduleInternal(Queue.java:599)
            hudson.model.Queue.schedule2(Queue.java:555)
            jenkins.model.ParameterizedJobMixIn.scheduleBuild2(ParameterizedJobMixIn.java:138)
            jenkins.model.ParameterizedJobMixIn.scheduleBuild(ParameterizedJobMixIn.java:94)
            hudson.model.AbstractProject.scheduleBuild(AbstractProject.java:836)
            {quote}

            My Job DSL setup is rather complicated and I have not been able to extract a simple test for this - if I do, I'll attach it.
            The attached thread dump shows two deadlocked threads:

            * "jenkins.util.Timer [#7]" daemon prio=5 BLOCKED
            * "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED)

            *Jenkins 1.650*

            This happens as follows:

            # I use the *Job DSL plugin* to create a bunch of jobs. Many have SCM Poll trigger enabled, causing them to run instantly as soon as they are created (possibly also when updated, looking at the stacktrace).
            # As part of job creation, *Job.updateNextBuildNumber* is called (Next Build Number plugin integrated with Job DSL)

            The two events above create deadlock.

            The first one locks *Job* since updateNextBuildNumber() is synchronized. Then it calls *AbstractLazyLoadRunMap.getByNumber()*. The synchronized block in this method is new as of *1.646*(https://github.com/jenkinsci/jenkins/commit/d5167025a204750633c931ea8c1fff8d7561ab9c#diff-383116e240993025e5b727359e61db09)

            The second one calls *AbstractLazyLoadRunMap.getByNumber()* first, which causes a call to *AbstractProject.save()*, but Job is an AbstractProject... so, deadlock.

            Here are the relevant parts for convenience, bold elements are causing deadlock:
            {quote}
            "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED
            jenkins.model.lazy.*AbstractLazyLoadRunMap.getByNumber*(AbstractLazyLoadRunMap.java:356)
            jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
            jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
            jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
            hudson.model.*Job.updateNextBuildNumber*(Job.java:422)
            org.jvnet.hudson.plugins.nextbuildnumber.JobDslExtension.notifyItemUpdated(JobDslExtension.java:30)
            {quote}

            {quote}
            "jenkins.util.Timer [#7]" daemon prio=5 BLOCKED
            hudson.model.*AbstractProject.save*(AbstractProject.java:305)
            hudson.model.Job.addProperty(Job.java:523)
            hudson.model.AbstractProject.addProperty(AbstractProject.java:785)
            hudson.plugins.disk_usage.DiskUsageUtil.addProperty(DiskUsageUtil.java:58)
            hudson.plugins.disk_usage.BuildDiskUsageAction.<init>(BuildDiskUsageAction.java:38)
            hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:31)
            hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:21)
            hudson.model.Actionable.createFor(Actionable.java:107)
            hudson.model.Actionable.getAllActions(Actionable.java:98)
            hudson.model.Run.onLoad(Run.java:343)
            hudson.model.RunMap.retrieve(RunMap.java:224)
            hudson.model.RunMap.retrieve(RunMap.java:56)
            jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:479)
            jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:461)
            jenkins.model.lazy.*AbstractLazyLoadRunMap.getByNumber*(AbstractLazyLoadRunMap.java:367)
            jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
            jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
            jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
            hudson.views.AbstractBuildTrendFilter.matches(AbstractBuildTrendFilter.java:71)
            hudson.views.AbstractIncludeExcludeJobFilter.doFilter(AbstractIncludeExcludeJobFilter.java:68)
            hudson.views.AbstractIncludeExcludeJobFilter.filter(AbstractIncludeExcludeJobFilter.java:57)
            hudson.model.ListView.getItems(ListView.java:195)
            hudson.model.ListView.getItems(ListView.java:67)
            jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.isJobInView(ViewBasedJobInclusionStrategy.java:182)
            jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.contains(ViewBasedJobInclusionStrategy.java:149)
            jenkins.advancedqueue.PriorityConfiguration.getJobGroup(PriorityConfiguration.java:241)
            jenkins.advancedqueue.PriorityConfiguration.getPriorityInternal(PriorityConfiguration.java:225)
            jenkins.advancedqueue.PriorityConfiguration.getPriority(PriorityConfiguration.java:203)
            jenkins.advancedqueue.sorter.AdvancedQueueSorter.onNewItem(AdvancedQueueSorter.java:136)
            jenkins.advancedqueue.sorter.AdvancedQueueSorterQueueListener.onEnterWaiting(AdvancedQueueSorterQueueListener.java:46)
            hudson.model.Queue$WaitingItem.enter(Queue.java:2348)
            hudson.model.Queue.scheduleInternal(Queue.java:599)
            hudson.model.Queue.schedule2(Queue.java:555)
            jenkins.model.ParameterizedJobMixIn.scheduleBuild2(ParameterizedJobMixIn.java:138)
            jenkins.model.ParameterizedJobMixIn.scheduleBuild(ParameterizedJobMixIn.java:94)
            hudson.model.AbstractProject.scheduleBuild(AbstractProject.java:836)
            {quote}

            My Job DSL setup is rather complicated and I have not been able to extract a simple test for this - if I do, I'll attach it.
            akom Alexander Komarov made changes -
            Description The attached thread dump shows two deadlocked threads:

            * "jenkins.util.Timer [#7]" daemon prio=5 BLOCKED
            * "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED)

            *Jenkins 1.650*

            This happens as follows:

            # I use the *Job DSL plugin* to create a bunch of jobs. Many have SCM Poll trigger enabled, causing them to run instantly as soon as they are created (possibly also when updated, looking at the stacktrace).
            # As part of job creation, *Job.updateNextBuildNumber* is called (Next Build Number plugin integrated with Job DSL)

            The two events above create deadlock.

            The first one locks *Job* since updateNextBuildNumber() is synchronized. Then it calls *AbstractLazyLoadRunMap.getByNumber()*. The synchronized block in this method is new as of *1.646*(https://github.com/jenkinsci/jenkins/commit/d5167025a204750633c931ea8c1fff8d7561ab9c#diff-383116e240993025e5b727359e61db09)

            The second one calls *AbstractLazyLoadRunMap.getByNumber()* first, which causes a call to *AbstractProject.save()*, but Job is an AbstractProject... so, deadlock.

            Here are the relevant parts for convenience, bold elements are causing deadlock:
            {quote}
            "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED
            jenkins.model.lazy.*AbstractLazyLoadRunMap.getByNumber*(AbstractLazyLoadRunMap.java:356)
            jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
            jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
            jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
            hudson.model.*Job.updateNextBuildNumber*(Job.java:422)
            org.jvnet.hudson.plugins.nextbuildnumber.JobDslExtension.notifyItemUpdated(JobDslExtension.java:30)
            {quote}

            {quote}
            "jenkins.util.Timer [#7]" daemon prio=5 BLOCKED
            hudson.model.*AbstractProject.save*(AbstractProject.java:305)
            hudson.model.Job.addProperty(Job.java:523)
            hudson.model.AbstractProject.addProperty(AbstractProject.java:785)
            hudson.plugins.disk_usage.DiskUsageUtil.addProperty(DiskUsageUtil.java:58)
            hudson.plugins.disk_usage.BuildDiskUsageAction.<init>(BuildDiskUsageAction.java:38)
            hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:31)
            hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:21)
            hudson.model.Actionable.createFor(Actionable.java:107)
            hudson.model.Actionable.getAllActions(Actionable.java:98)
            hudson.model.Run.onLoad(Run.java:343)
            hudson.model.RunMap.retrieve(RunMap.java:224)
            hudson.model.RunMap.retrieve(RunMap.java:56)
            jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:479)
            jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:461)
            jenkins.model.lazy.*AbstractLazyLoadRunMap.getByNumber*(AbstractLazyLoadRunMap.java:367)
            jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
            jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
            jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
            hudson.views.AbstractBuildTrendFilter.matches(AbstractBuildTrendFilter.java:71)
            hudson.views.AbstractIncludeExcludeJobFilter.doFilter(AbstractIncludeExcludeJobFilter.java:68)
            hudson.views.AbstractIncludeExcludeJobFilter.filter(AbstractIncludeExcludeJobFilter.java:57)
            hudson.model.ListView.getItems(ListView.java:195)
            hudson.model.ListView.getItems(ListView.java:67)
            jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.isJobInView(ViewBasedJobInclusionStrategy.java:182)
            jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.contains(ViewBasedJobInclusionStrategy.java:149)
            jenkins.advancedqueue.PriorityConfiguration.getJobGroup(PriorityConfiguration.java:241)
            jenkins.advancedqueue.PriorityConfiguration.getPriorityInternal(PriorityConfiguration.java:225)
            jenkins.advancedqueue.PriorityConfiguration.getPriority(PriorityConfiguration.java:203)
            jenkins.advancedqueue.sorter.AdvancedQueueSorter.onNewItem(AdvancedQueueSorter.java:136)
            jenkins.advancedqueue.sorter.AdvancedQueueSorterQueueListener.onEnterWaiting(AdvancedQueueSorterQueueListener.java:46)
            hudson.model.Queue$WaitingItem.enter(Queue.java:2348)
            hudson.model.Queue.scheduleInternal(Queue.java:599)
            hudson.model.Queue.schedule2(Queue.java:555)
            jenkins.model.ParameterizedJobMixIn.scheduleBuild2(ParameterizedJobMixIn.java:138)
            jenkins.model.ParameterizedJobMixIn.scheduleBuild(ParameterizedJobMixIn.java:94)
            hudson.model.AbstractProject.scheduleBuild(AbstractProject.java:836)
            {quote}

            My Job DSL setup is rather complicated and I have not been able to extract a simple test for this - if I do, I'll attach it.
            The attached thread dump shows two deadlocked threads:

            * "jenkins.util.Timer [#7]" daemon prio=5 BLOCKED
            * "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED)

            *Jenkins 1.650*

            This happens as follows:

            # I use the *Job DSL plugin* to create a bunch of jobs. Many have SCM Poll trigger enabled, causing them to run instantly as soon as they are created (possibly also when updated, looking at the stacktrace).
            # As part of job creation, *Job.updateNextBuildNumber* is called (Next Build Number plugin integrated with Job DSL)

            The two events above create deadlock.

            The first one locks *Job* since updateNextBuildNumber() is synchronized. Then it calls *AbstractLazyLoadRunMap.getByNumber()*. The synchronized block in this method is new as of *1.646*(https://github.com/jenkinsci/jenkins/commit/d5167025a204750633c931ea8c1fff8d7561ab9c#diff-383116e240993025e5b727359e61db09)

            The second one calls *AbstractLazyLoadRunMap.getByNumber()* first, which causes a call to *AbstractProject.save()*, but Job is an AbstractProject... so, deadlock.

            Here are the relevant parts for convenience, bold elements are causing deadlock:
            {quote}
            "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED
            jenkins.model.lazy.*AbstractLazyLoadRunMap.getByNumber*(AbstractLazyLoadRunMap.java:356)
            jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
            jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
            jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
            hudson.model.*Job.updateNextBuildNumber*(Job.java:422)
            org.jvnet.hudson.plugins.nextbuildnumber.JobDslExtension.notifyItemUpdated(JobDslExtension.java:30)
            {quote}

            {quote}
            "jenkins.util.Timer [#7]" daemon prio=5 BLOCKED
            hudson.model.*AbstractProject.save*(AbstractProject.java:305)
            hudson.model.Job.addProperty(Job.java:523)
            hudson.model.AbstractProject.addProperty(AbstractProject.java:785)
            hudson.plugins.disk_usage.DiskUsageUtil.addProperty(DiskUsageUtil.java:58)
            hudson.plugins.disk_usage.BuildDiskUsageAction.<init>(BuildDiskUsageAction.java:38)
            hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:31)
            hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:21)
            hudson.model.Actionable.createFor(Actionable.java:107)
            hudson.model.Actionable.getAllActions(Actionable.java:98)
            hudson.model.Run.onLoad(Run.java:343)
            hudson.model.RunMap.retrieve(RunMap.java:224)
            hudson.model.RunMap.retrieve(RunMap.java:56)
            jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:479)
            jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:461)
            jenkins.model.lazy.*AbstractLazyLoadRunMap.getByNumber*(AbstractLazyLoadRunMap.java:367)
            jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
            jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
            jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
            hudson.views.AbstractBuildTrendFilter.matches(AbstractBuildTrendFilter.java:71)
            hudson.views.AbstractIncludeExcludeJobFilter.doFilter(AbstractIncludeExcludeJobFilter.java:68)
            hudson.views.AbstractIncludeExcludeJobFilter.filter(AbstractIncludeExcludeJobFilter.java:57)
            hudson.model.ListView.getItems(ListView.java:195)
            hudson.model.ListView.getItems(ListView.java:67)
            jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.isJobInView(ViewBasedJobInclusionStrategy.java:182)
            jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.contains(ViewBasedJobInclusionStrategy.java:149)
            jenkins.advancedqueue.PriorityConfiguration.getJobGroup(PriorityConfiguration.java:241)
            jenkins.advancedqueue.PriorityConfiguration.getPriorityInternal(PriorityConfiguration.java:225)
            jenkins.advancedqueue.PriorityConfiguration.getPriority(PriorityConfiguration.java:203)
            jenkins.advancedqueue.sorter.AdvancedQueueSorter.onNewItem(AdvancedQueueSorter.java:136)
            jenkins.advancedqueue.sorter.AdvancedQueueSorterQueueListener.onEnterWaiting(AdvancedQueueSorterQueueListener.java:46)
            hudson.model.Queue$WaitingItem.enter(Queue.java:2348)
            hudson.model.Queue.scheduleInternal(Queue.java:599)
            hudson.model.Queue.schedule2(Queue.java:555)
            jenkins.model.ParameterizedJobMixIn.scheduleBuild2(ParameterizedJobMixIn.java:138)
            jenkins.model.ParameterizedJobMixIn.scheduleBuild(ParameterizedJobMixIn.java:94)
            hudson.model.AbstractProject.scheduleBuild(AbstractProject.java:836)
            {quote}

            My Job DSL setup is rather complicated and I have not been able to extract a simple test for this - if I do, I'll attach it.


            Update: 1.645 also deadlocks. AbstractLazyLoadRunMap.load() is synchronized. I am confused about why this started happening once I upgraded from 1.645 but continues to happen once I downgraded back.
            akom Alexander Komarov made changes -
            Description The attached thread dump shows two deadlocked threads:

            * "jenkins.util.Timer [#7]" daemon prio=5 BLOCKED
            * "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED)

            *Jenkins 1.650*

            This happens as follows:

            # I use the *Job DSL plugin* to create a bunch of jobs. Many have SCM Poll trigger enabled, causing them to run instantly as soon as they are created (possibly also when updated, looking at the stacktrace).
            # As part of job creation, *Job.updateNextBuildNumber* is called (Next Build Number plugin integrated with Job DSL)

            The two events above create deadlock.

            The first one locks *Job* since updateNextBuildNumber() is synchronized. Then it calls *AbstractLazyLoadRunMap.getByNumber()*. The synchronized block in this method is new as of *1.646*(https://github.com/jenkinsci/jenkins/commit/d5167025a204750633c931ea8c1fff8d7561ab9c#diff-383116e240993025e5b727359e61db09)

            The second one calls *AbstractLazyLoadRunMap.getByNumber()* first, which causes a call to *AbstractProject.save()*, but Job is an AbstractProject... so, deadlock.

            Here are the relevant parts for convenience, bold elements are causing deadlock:
            {quote}
            "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED
            jenkins.model.lazy.*AbstractLazyLoadRunMap.getByNumber*(AbstractLazyLoadRunMap.java:356)
            jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
            jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
            jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
            hudson.model.*Job.updateNextBuildNumber*(Job.java:422)
            org.jvnet.hudson.plugins.nextbuildnumber.JobDslExtension.notifyItemUpdated(JobDslExtension.java:30)
            {quote}

            {quote}
            "jenkins.util.Timer [#7]" daemon prio=5 BLOCKED
            hudson.model.*AbstractProject.save*(AbstractProject.java:305)
            hudson.model.Job.addProperty(Job.java:523)
            hudson.model.AbstractProject.addProperty(AbstractProject.java:785)
            hudson.plugins.disk_usage.DiskUsageUtil.addProperty(DiskUsageUtil.java:58)
            hudson.plugins.disk_usage.BuildDiskUsageAction.<init>(BuildDiskUsageAction.java:38)
            hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:31)
            hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:21)
            hudson.model.Actionable.createFor(Actionable.java:107)
            hudson.model.Actionable.getAllActions(Actionable.java:98)
            hudson.model.Run.onLoad(Run.java:343)
            hudson.model.RunMap.retrieve(RunMap.java:224)
            hudson.model.RunMap.retrieve(RunMap.java:56)
            jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:479)
            jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:461)
            jenkins.model.lazy.*AbstractLazyLoadRunMap.getByNumber*(AbstractLazyLoadRunMap.java:367)
            jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
            jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
            jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
            hudson.views.AbstractBuildTrendFilter.matches(AbstractBuildTrendFilter.java:71)
            hudson.views.AbstractIncludeExcludeJobFilter.doFilter(AbstractIncludeExcludeJobFilter.java:68)
            hudson.views.AbstractIncludeExcludeJobFilter.filter(AbstractIncludeExcludeJobFilter.java:57)
            hudson.model.ListView.getItems(ListView.java:195)
            hudson.model.ListView.getItems(ListView.java:67)
            jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.isJobInView(ViewBasedJobInclusionStrategy.java:182)
            jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.contains(ViewBasedJobInclusionStrategy.java:149)
            jenkins.advancedqueue.PriorityConfiguration.getJobGroup(PriorityConfiguration.java:241)
            jenkins.advancedqueue.PriorityConfiguration.getPriorityInternal(PriorityConfiguration.java:225)
            jenkins.advancedqueue.PriorityConfiguration.getPriority(PriorityConfiguration.java:203)
            jenkins.advancedqueue.sorter.AdvancedQueueSorter.onNewItem(AdvancedQueueSorter.java:136)
            jenkins.advancedqueue.sorter.AdvancedQueueSorterQueueListener.onEnterWaiting(AdvancedQueueSorterQueueListener.java:46)
            hudson.model.Queue$WaitingItem.enter(Queue.java:2348)
            hudson.model.Queue.scheduleInternal(Queue.java:599)
            hudson.model.Queue.schedule2(Queue.java:555)
            jenkins.model.ParameterizedJobMixIn.scheduleBuild2(ParameterizedJobMixIn.java:138)
            jenkins.model.ParameterizedJobMixIn.scheduleBuild(ParameterizedJobMixIn.java:94)
            hudson.model.AbstractProject.scheduleBuild(AbstractProject.java:836)
            {quote}

            My Job DSL setup is rather complicated and I have not been able to extract a simple test for this - if I do, I'll attach it.


            Update: 1.645 also deadlocks. AbstractLazyLoadRunMap.load() is synchronized. I am confused about why this started happening once I upgraded from 1.645 but continues to happen once I downgraded back.
            The attached thread dump shows two deadlocked threads:

            * "jenkins.util.Timer [#7]" daemon prio=5 BLOCKED
            * "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED)

            *Jenkins 1.650*

            This happens as follows:

            # I use the *Job DSL plugin* to create a bunch of jobs. Many have SCM Poll trigger enabled, causing them to run instantly as soon as they are created (possibly also when updated, looking at the stacktrace).
            # As part of job creation, *Job.updateNextBuildNumber* is called (Next Build Number plugin integrated with Job DSL)

            The two events above create deadlock.

            The first one locks *Job* since updateNextBuildNumber() is synchronized. Then it calls *AbstractLazyLoadRunMap.getByNumber()*. The synchronized block in this method is new as of *1.646*(https://github.com/jenkinsci/jenkins/commit/d5167025a204750633c931ea8c1fff8d7561ab9c#diff-383116e240993025e5b727359e61db09)

            The second one calls *AbstractLazyLoadRunMap.getByNumber()* first, which causes a call to *AbstractProject.save()*, but Job is an AbstractProject... so, deadlock.

            Here are the relevant parts for convenience, bold elements are causing deadlock:
            {quote}
            "Executor #1 for master : executing DSL Job Builder #390" daemon prio=5 BLOCKED
            jenkins.model.lazy.*AbstractLazyLoadRunMap.getByNumber*(AbstractLazyLoadRunMap.java:356)
            jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
            jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
            jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
            hudson.model.*Job.updateNextBuildNumber*(Job.java:422)
            org.jvnet.hudson.plugins.nextbuildnumber.JobDslExtension.notifyItemUpdated(JobDslExtension.java:30)
            {quote}

            {quote}
            "jenkins.util.Timer [#7]" daemon prio=5 BLOCKED
            hudson.model.*AbstractProject.save*(AbstractProject.java:305)
            hudson.model.Job.addProperty(Job.java:523)
            hudson.model.AbstractProject.addProperty(AbstractProject.java:785)
            hudson.plugins.disk_usage.DiskUsageUtil.addProperty(DiskUsageUtil.java:58)
            hudson.plugins.disk_usage.BuildDiskUsageAction.<init>(BuildDiskUsageAction.java:38)
            hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:31)
            hudson.plugins.disk_usage.DiskUsageBuildActionFactory.createFor(DiskUsageBuildActionFactory.java:21)
            hudson.model.Actionable.createFor(Actionable.java:107)
            hudson.model.Actionable.getAllActions(Actionable.java:98)
            hudson.model.Run.onLoad(Run.java:343)
            hudson.model.RunMap.retrieve(RunMap.java:224)
            hudson.model.RunMap.retrieve(RunMap.java:56)
            jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:479)
            jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:461)
            jenkins.model.lazy.*AbstractLazyLoadRunMap.getByNumber*(AbstractLazyLoadRunMap.java:367)
            jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
            jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:274)
            jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:238)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:993)
            hudson.model.AbstractProject.getLastBuild(AbstractProject.java:144)
            hudson.views.AbstractBuildTrendFilter.matches(AbstractBuildTrendFilter.java:71)
            hudson.views.AbstractIncludeExcludeJobFilter.doFilter(AbstractIncludeExcludeJobFilter.java:68)
            hudson.views.AbstractIncludeExcludeJobFilter.filter(AbstractIncludeExcludeJobFilter.java:57)
            hudson.model.ListView.getItems(ListView.java:195)
            hudson.model.ListView.getItems(ListView.java:67)
            jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.isJobInView(ViewBasedJobInclusionStrategy.java:182)
            jenkins.advancedqueue.jobinclusion.strategy.ViewBasedJobInclusionStrategy.contains(ViewBasedJobInclusionStrategy.java:149)
            jenkins.advancedqueue.PriorityConfiguration.getJobGroup(PriorityConfiguration.java:241)
            jenkins.advancedqueue.PriorityConfiguration.getPriorityInternal(PriorityConfiguration.java:225)
            jenkins.advancedqueue.PriorityConfiguration.getPriority(PriorityConfiguration.java:203)
            jenkins.advancedqueue.sorter.AdvancedQueueSorter.onNewItem(AdvancedQueueSorter.java:136)
            jenkins.advancedqueue.sorter.AdvancedQueueSorterQueueListener.onEnterWaiting(AdvancedQueueSorterQueueListener.java:46)
            hudson.model.Queue$WaitingItem.enter(Queue.java:2348)
            hudson.model.Queue.scheduleInternal(Queue.java:599)
            hudson.model.Queue.schedule2(Queue.java:555)
            jenkins.model.ParameterizedJobMixIn.scheduleBuild2(ParameterizedJobMixIn.java:138)
            jenkins.model.ParameterizedJobMixIn.scheduleBuild(ParameterizedJobMixIn.java:94)
            hudson.model.AbstractProject.scheduleBuild(AbstractProject.java:836)
            {quote}

            My Job DSL setup is rather complicated and I have not been able to extract a simple test for this - if I do, I'll attach it.


            Update: *1.645 also deadlocks*. AbstractLazyLoadRunMap.load() is synchronized. I am confused about why this started happening once I upgraded from 1.645 but continues to happen once I downgraded back. Maybe the newer version of some other plugin is making this more likely (ie SVN plugin polls more agressively, etc)
            olivergondza Oliver Gondža made changes -
            Link This issue is blocking JENKINS-22767 [ JENKINS-22767 ]
            jglick Jesse Glick made changes -
            Labels jenkins deadlock regression
            jglick Jesse Glick made changes -
            Priority Minor [ 4 ] Major [ 3 ]
            jglick Jesse Glick made changes -
            Component/s disk-usage-plugin [ 15537 ]
            Component/s core [ 15593 ]
            Assignee Lucie Votypkova [ lvotypkova ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 169102 ] JNJira + In-Review [ 183358 ]
            olivergondza Oliver Gondža made changes -
            Assignee Lucie Votypkova [ lvotypkova ]

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              akom Alexander Komarov
              Votes:
              2 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated: