Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-72678

Thread Deadlock during GitHub Organization Scan and WeatherColumn

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • Running Jenkins controllers on AWS ECS. Using EFS for Jenkins home storage.

      Jenkins Controller: 2.246.1, 2.426.3
      github-branch-source-plugin: 1752.vc201a_0235d80
      cloudbees-folder-plugin: 6.858.v898218f3609d

      We are seeing occasional deadlocked/blocked threads on the Github Organization Scans due to possible health checks/WeatherColumn.  

      From the catalina.log:

      Some health checks are reporting as unhealthy: [thread-deadlock : [Executor #-1 for Built-In Node : executing OrganizationScan[CP-Models] locked on hudson.model.RunMap@50446244 (owned by Handling GET /job/CP-Models/ from 192.168.1.143 : Jetty (winstone)-345663 View/index.jelly WeatherColumn/column.jelly):
           at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:526)
           at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:502)
           at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:435)
           at jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:254)
           at org.jenkinsci.plugins.workflow.job.WorkflowJob.getLastBuild(WorkflowJob.java:242)
           at org.jenkinsci.plugins.workflow.job.WorkflowJob.getLastBuild(WorkflowJob.java:105)
           at hudson.model.Job.getBuildHealthReports(Job.java:1200)
           at hudson.model.Job.getBuildHealth(Job.java:1193)
           at com.cloudbees.hudson.plugins.folder.health.FolderHealthMetric.getHealthReport(FolderHealthMetric.java:48)
           at com.cloudbees.hudson.plugins.folder.health.WorstChildHealthMetric$ReporterImpl.observe(WorstChildHealthMetric.java:86)
           at com.cloudbees.hudson.plugins.folder.AbstractFolder.getBuildHealthReports(AbstractFolder.java:924)
           at java.base@11.0.21/java.lang.invoke.DirectMethodHandle$Holder.invokeVirtual(DirectMethodHandle$Holder)
           at java.base@11.0.21/java.lang.invoke.LambdaForm$MH/0x0000000800277840.invoke(LambdaForm$MH)
           at java.base@11.0.21/java.lang.invoke.LambdaForm$MH/0x00000008002a7440.invoke_MT(LambdaForm$MH)
           at org.kohsuke.stapler.export.MethodProperty.getValue(MethodProperty.java:76)
           at org.kohsuke.stapler.export.ExportInterceptor$1.getValue(ExportInterceptor.java:46)
           at io.jenkins.plugins.generic.event.json.ExportedBeanProcessor$IgnoreURLExportInterceptor.getValue(ExportedBeanProcessor.java:71)
           at org.kohsuke.stapler.export.Property.writeTo(Property.java:136)
           at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:222)
           at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:218)
           at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:218)
           at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:218)
           at org.kohsuke.stapler.export.Model.writeTo(Model.java:193)
           at org.kohsuke.stapler.export.Model.writeTo(Model.java:213)
           at org.kohsuke.stapler.export.Model.writeTo(Model.java:181)
           at io.jenkins.plugins.generic.event.json.ExportedBeanProcessor.processBean(ExportedBeanProcessor.java:43)
           at net.sf.json.JSONObject._fromBean(JSONObject.java:619)
           at net.sf.json.JSONObject.fromObject(JSONObject.java:169)
           at net.sf.json.AbstractJSON._processValue(AbstractJSON.java:250)
      ...

      See the attached full catalina.log stack trace.   I also attached the thread dump showing the BLOCKED threads.   

      This looked like the Weather health metric/folder performance issue as documented here by Cloudbees.  But we have disabled all health metrics on all folders including Github Organization folders and we still have deadlocks/blocked threads.

      How to disable the weather column to resolve instance slowness? (cloudbees.com)

      When this occurs, we have to restart the Jenkins controller.   This seems to have started after we upgraded to 2.246.1 back in December as we have had rock solid controllers on 2.387.3 which was the previous version.   We are currently running 2.246.3 and see the issue.

          [JENKINS-72678] Thread Deadlock during GitHub Organization Scan and WeatherColumn

          We have 80+ Jenkins controllers and we are seeing 1-2 controllers deadlock this issue per week.  This started when we upgraded to 2.246.1 + latest plugin versions at the time of the upgrade around December 2023.

          John Lengeling added a comment - We have 80+ Jenkins controllers and we are seeing 1-2 controllers deadlock this issue per week.  This started when we upgraded to 2.246.1 + latest plugin versions at the time of the upgrade around December 2023.

          Any update on this? Another occurance with a Java Level threadlock related to WorstChildHealthMetric:

          "class org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber$SCMHeadEventImpl Sun Feb 11 12:42:50 CST 2024 / SCMEvent [#1]":
            waiting to lock monitor 0x00007f596c63fa80 (object 0x00000006a6f6a2b0, a org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject),
            which is held by "Executor #-1 for Built-In Node : executing OrganizationScan[Org1]"
          "Executor #-1 for Built-In Node : executing OrganizationScan[Org1]":
            waiting to lock monitor 0x00007f59308d2f00 (object 0x00000006ce7a3318, a hudson.model.RunMap),
            which is held by "Periodic background build discarder thread"
          "Periodic background build discarder thread":
            waiting to lock monitor 0x00007f596c63fa80 (object 0x00000006a6f6a2b0, a org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject),
            which is held by "Executor #-1 for Built-In Node : executing OrganizationScan[Org1]"
          
          Java stack information for the threads listed above:
          ===================================================
          "class org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber$SCMHeadEventImpl Sun Feb 11 12:42:50 CST 2024 / SCMEvent [#1]":
              at jenkins.branch.MultiBranchProject.getProjectFactory(MultiBranchProject.java:343)
              - waiting to lock <0x00000006a6f6a2b0> (a org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject)
              at jenkins.branch.MultiBranchProject$SCMEventListenerImpl.processHeadUpdate(MultiBranchProject.java:1436)
              at jenkins.branch.MultiBranchProject$SCMEventListenerImpl.onSCMHeadEvent(MultiBranchProject.java:1218)
              at jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:246)
              at jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:229)
              at jenkins.scm.api.SCMEvent$Dispatcher.run(SCMEvent.java:545)
              at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
              at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.21/Executors.java:515)
              at java.util.concurrent.FutureTask.run(java.base@11.0.21/FutureTask.java:264)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.21/ScheduledThreadPoolExecutor.java:304)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.21/ThreadPoolExecutor.java:1128)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.21/ThreadPoolExecutor.java:628)
              at java.lang.Thread.run(java.base@11.0.21/Thread.java:829)
          "Executor #-1 for Built-In Node : executing OrganizationScan[Org1]":
              at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:526)
              - waiting to lock <0x00000006ce7a3318> (a hudson.model.RunMap)
              at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:502)
              at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:435)
              at jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:254)
              at org.jenkinsci.plugins.workflow.job.WorkflowJob.getLastBuild(WorkflowJob.java:242)
              at org.jenkinsci.plugins.workflow.job.WorkflowJob.getLastBuild(WorkflowJob.java:105)
              at hudson.model.Job.getBuildHealthReports(Job.java:1200)
              at hudson.model.Job.getBuildHealth(Job.java:1193)
              at com.cloudbees.hudson.plugins.folder.health.FolderHealthMetric.getHealthReport(FolderHealthMetric.java:48)
              at com.cloudbees.hudson.plugins.folder.health.WorstChildHealthMetric$ReporterImpl.observe(WorstChildHealthMetric.java:86)
              at com.cloudbees.hudson.plugins.folder.AbstractFolder.getBuildHealthReports(AbstractFolder.java:924)
              at java.lang.invoke.DirectMethodHandle$Holder.invokeVirtual(java.base@11.0.21/DirectMethodHandle$Holder)
              at java.lang.invoke.LambdaForm$MH/0x0000000800277040.invoke(java.base@11.0.21/LambdaForm$MH)
              at java.lang.invoke.LambdaForm$MH/0x00000008002a7440.invoke_MT(java.base@11.0.21/LambdaForm$MH)
              at org.kohsuke.stapler.export.MethodProperty.getValue(MethodProperty.java:76)
              at org.kohsuke.stapler.export.ExportInterceptor$1.getValue(ExportInterceptor.java:46)
              at io.jenkins.plugins.generic.event.json.ExportedBeanProcessor$IgnoreURLExportInterceptor.getValue(ExportedBeanProcessor.java:71)
              at org.kohsuke.stapler.export.Property.writeTo(Property.java:136)
              at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:222)
              at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:218)
              at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:218)
              at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:218)
              at org.kohsuke.stapler.export.Model.writeTo(Model.java:193)
              at org.kohsuke.stapler.export.Model.writeTo(Model.java:213)
              at org.kohsuke.stapler.export.Model.writeTo(Model.java:181)
              at io.jenkins.plugins.generic.event.json.ExportedBeanProcessor.processBean(ExportedBeanProcessor.java:43)
              at net.sf.json.JSONObject._fromBean(JSONObject.java:619)
              at net.sf.json.JSONObject.fromObject(JSONObject.java:169)
              at net.sf.json.AbstractJSON._processValue(AbstractJSON.java:250)
              at net.sf.json.JSONObject._processValue(JSONObject.java:2599)
              at net.sf.json.JSONObject.processValue(JSONObject.java:2665)
              at net.sf.json.JSONObject.setInternal(JSONObject.java:2680)
              at net.sf.json.JSONObject.setValue(JSONObject.java:1376)
              at net.sf.json.JSONObject.defaultBeanProcessing(JSONObject.java:708)
              at net.sf.json.JSONObject._fromBean(JSONObject.java:642)
              at net.sf.json.JSONObject.fromObject(JSONObject.java:169)
              at io.jenkins.plugins.generic.event.HttpEventSender$CustomCloseableHttpAsyncClient.sendPost(HttpEventSender.java:90)
              at io.jenkins.plugins.generic.event.HttpEventSender.send(HttpEventSender.java:45)
              at io.jenkins.plugins.generic.event.listener.GenericEventItemListener.onUpdated(GenericEventItemListener.java:118)
              at hudson.model.listeners.ItemListener.lambda$fireOnUpdated$2(ItemListener.java:205)
              at hudson.model.listeners.ItemListener$$Lambda$1754/0x00000008022b3040.accept(Unknown Source)
              at jenkins.util.Listeners.lambda$notify$0(Listeners.java:59)
              at jenkins.util.Listeners$$Lambda$645/0x00000008015afc40.run(Unknown Source)
              at jenkins.util.Listeners.notify(Listeners.java:70)
              at hudson.model.listeners.ItemListener.fireOnUpdated(ItemListener.java:205)
              at com.cloudbees.hudson.plugins.folder.AbstractFolder.save(AbstractFolder.java:1317)
              - locked <0x00000006a6f6a2b0> (a org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject)
              at hudson.BulkChange.commit(BulkChange.java:98)
              at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.completeExisting(OrganizationFolder.java:1427)
              at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.complete(OrganizationFolder.java:1395)
              at jenkins.scm.api.trait.SCMNavigatorRequest.process(SCMNavigatorRequest.java:252)
              at jenkins.scm.api.trait.SCMNavigatorRequest.process(SCMNavigatorRequest.java:202)
              at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:1135)
              at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:535)
              at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:269)
              at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:167)
              at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:920)
              at hudson.model.ResourceController.execute(ResourceController.java:101)
              at hudson.model.Executor.run(Executor.java:442)
          "Periodic background build discarder thread":
              at jenkins.branch.MultiBranchProject.getProjectFactory(MultiBranchProject.java:343)
              - waiting to lock <0x00000006a6f6a2b0> (a org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject)
              at io.jenkins.blueocean.rest.impl.pipeline.MultiBranchPipelineImpl$PipelineFactoryImpl.getPipeline(MultiBranchPipelineImpl.java:276)
              at io.jenkins.blueocean.rest.impl.pipeline.MultiBranchPipelineImpl$PipelineFactoryImpl.resolve(MultiBranchPipelineImpl.java:291)
              at io.jenkins.blueocean.rest.impl.pipeline.OrganizationFolderPipelineImpl$OrganizationFolderFactory.resolve(OrganizationFolderPipelineImpl.java:187)
              at io.jenkins.blueocean.rest.factory.BluePipelineFactory.resolve(BluePipelineFactory.java:75)
              at io.jenkins.blueocean.service.embedded.BlueOceanUrlMapperImpl.getJobResource(BlueOceanUrlMapperImpl.java:80)
              at io.jenkins.blueocean.service.embedded.BlueOceanUrlMapperImpl.getUrl(BlueOceanUrlMapperImpl.java:49)
              at io.jenkins.blueocean.service.embedded.BlueOceanUrlObjectImpl.computeUrl(BlueOceanUrlObjectImpl.java:42)
              at io.jenkins.blueocean.service.embedded.BlueOceanUrlObjectImpl.<init>(BlueOceanUrlObjectImpl.java:21)
              at io.jenkins.blueocean.service.embedded.BlueOceanUrlObjectFactoryImpl.get(BlueOceanUrlObjectFactoryImpl.java:18)
              at io.jenkins.blueocean.service.embedded.TryBlueOceanMenu.createFor(TryBlueOceanMenu.java:50)
              at io.jenkins.blueocean.service.embedded.TryBlueOceanMenu.createFor(TryBlueOceanMenu.java:26)
              at hudson.model.Actionable.createFor(Actionable.java:115)
              at hudson.model.Actionable.getAllActions(Actionable.java:101)
              at hudson.model.Run.onLoad(Run.java:386)
              at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:566)
              - locked <0x0000000692413998> (a java.lang.Object)
              at hudson.model.RunMap.retrieve(RunMap.java:233)
              at hudson.model.RunMap.retrieve(RunMap.java:61)
              at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:657)
              at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:639)
              at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:537)
              - locked <0x00000006ce7a3318> (a hudson.model.RunMap)
              at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:240)
              at org.jenkinsci.plugins.workflow.job.WorkflowJob.getBuildByNumber(WorkflowJob.java:234)
              at org.jenkinsci.plugins.workflow.job.WorkflowJob.getBuildByNumber(WorkflowJob.java:105)
              at jenkins.model.PeepholePermalink.resolve(PeepholePermalink.java:103)
              at hudson.model.Job.getLastSuccessfulBuild(Job.java:943)
              at hudson.tasks.LogRotator.perform(LogRotator.java:152)
              at hudson.model.Job.logRotate(Job.java:473)
              at jenkins.model.JobGlobalBuildDiscarderStrategy.apply(JobGlobalBuildDiscarderStrategy.java:54)
              at jenkins.model.BackgroundGlobalBuildDiscarder.lambda$processJob$0(BackgroundGlobalBuildDiscarder.java:67)
              at jenkins.model.BackgroundGlobalBuildDiscarder$$Lambda$1181/0x00000008019f8440.accept(Unknown Source)
              at java.lang.Iterable.forEach(java.base@11.0.21/Iterable.java:75)
              at jenkins.model.BackgroundGlobalBuildDiscarder.processJob(BackgroundGlobalBuildDiscarder.java:61)
              at jenkins.model.BackgroundGlobalBuildDiscarder.execute(BackgroundGlobalBuildDiscarder.java:55)
              at hudson.model.AsyncPeriodicWork.lambda$doRun$0(AsyncPeriodicWork.java:102)
              at hudson.model.AsyncPeriodicWork$$Lambda$850/0x000000080178d840.run(Unknown Source)
              at java.lang.Thread.run(java.base@11.0.21/Thread.java:829)
          

          Logan Highland added a comment - Any update on this? Another occurance with a Java Level threadlock related to WorstChildHealthMetric: "class org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber$SCMHeadEventImpl Sun Feb 11 12:42:50 CST 2024 / SCMEvent [#1]": waiting to lock monitor 0x00007f596c63fa80 (object 0x00000006a6f6a2b0, a org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject), which is held by "Executor #-1 for Built-In Node : executing OrganizationScan[Org1]" "Executor #-1 for Built-In Node : executing OrganizationScan[Org1]": waiting to lock monitor 0x00007f59308d2f00 (object 0x00000006ce7a3318, a hudson.model.RunMap), which is held by "Periodic background build discarder thread" "Periodic background build discarder thread": waiting to lock monitor 0x00007f596c63fa80 (object 0x00000006a6f6a2b0, a org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject), which is held by "Executor #-1 for Built-In Node : executing OrganizationScan[Org1]" Java stack information for the threads listed above: =================================================== "class org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber$SCMHeadEventImpl Sun Feb 11 12:42:50 CST 2024 / SCMEvent [#1]": at jenkins.branch.MultiBranchProject.getProjectFactory(MultiBranchProject.java:343) - waiting to lock <0x00000006a6f6a2b0> (a org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject) at jenkins.branch.MultiBranchProject$SCMEventListenerImpl.processHeadUpdate(MultiBranchProject.java:1436) at jenkins.branch.MultiBranchProject$SCMEventListenerImpl.onSCMHeadEvent(MultiBranchProject.java:1218) at jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:246) at jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:229) at jenkins.scm.api.SCMEvent$Dispatcher.run(SCMEvent.java:545) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67) at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.21/Executors.java:515) at java.util.concurrent.FutureTask.run(java.base@11.0.21/FutureTask.java:264) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.21/ScheduledThreadPoolExecutor.java:304) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.21/ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.21/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.21/Thread.java:829) "Executor #-1 for Built-In Node : executing OrganizationScan[Org1]": at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:526) - waiting to lock <0x00000006ce7a3318> (a hudson.model.RunMap) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:502) at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:435) at jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:254) at org.jenkinsci.plugins.workflow.job.WorkflowJob.getLastBuild(WorkflowJob.java:242) at org.jenkinsci.plugins.workflow.job.WorkflowJob.getLastBuild(WorkflowJob.java:105) at hudson.model.Job.getBuildHealthReports(Job.java:1200) at hudson.model.Job.getBuildHealth(Job.java:1193) at com.cloudbees.hudson.plugins.folder.health.FolderHealthMetric.getHealthReport(FolderHealthMetric.java:48) at com.cloudbees.hudson.plugins.folder.health.WorstChildHealthMetric$ReporterImpl.observe(WorstChildHealthMetric.java:86) at com.cloudbees.hudson.plugins.folder.AbstractFolder.getBuildHealthReports(AbstractFolder.java:924) at java.lang.invoke.DirectMethodHandle$Holder.invokeVirtual(java.base@11.0.21/DirectMethodHandle$Holder) at java.lang.invoke.LambdaForm$MH/0x0000000800277040.invoke(java.base@11.0.21/LambdaForm$MH) at java.lang.invoke.LambdaForm$MH/0x00000008002a7440.invoke_MT(java.base@11.0.21/LambdaForm$MH) at org.kohsuke.stapler.export.MethodProperty.getValue(MethodProperty.java:76) at org.kohsuke.stapler.export.ExportInterceptor$1.getValue(ExportInterceptor.java:46) at io.jenkins.plugins.generic.event.json.ExportedBeanProcessor$IgnoreURLExportInterceptor.getValue(ExportedBeanProcessor.java:71) at org.kohsuke.stapler.export.Property.writeTo(Property.java:136) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:222) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:218) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:218) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:218) at org.kohsuke.stapler.export.Model.writeTo(Model.java:193) at org.kohsuke.stapler.export.Model.writeTo(Model.java:213) at org.kohsuke.stapler.export.Model.writeTo(Model.java:181) at io.jenkins.plugins.generic.event.json.ExportedBeanProcessor.processBean(ExportedBeanProcessor.java:43) at net.sf.json.JSONObject._fromBean(JSONObject.java:619) at net.sf.json.JSONObject.fromObject(JSONObject.java:169) at net.sf.json.AbstractJSON._processValue(AbstractJSON.java:250) at net.sf.json.JSONObject._processValue(JSONObject.java:2599) at net.sf.json.JSONObject.processValue(JSONObject.java:2665) at net.sf.json.JSONObject.setInternal(JSONObject.java:2680) at net.sf.json.JSONObject.setValue(JSONObject.java:1376) at net.sf.json.JSONObject.defaultBeanProcessing(JSONObject.java:708) at net.sf.json.JSONObject._fromBean(JSONObject.java:642) at net.sf.json.JSONObject.fromObject(JSONObject.java:169) at io.jenkins.plugins.generic.event.HttpEventSender$CustomCloseableHttpAsyncClient.sendPost(HttpEventSender.java:90) at io.jenkins.plugins.generic.event.HttpEventSender.send(HttpEventSender.java:45) at io.jenkins.plugins.generic.event.listener.GenericEventItemListener.onUpdated(GenericEventItemListener.java:118) at hudson.model.listeners.ItemListener.lambda$fireOnUpdated$2(ItemListener.java:205) at hudson.model.listeners.ItemListener$$Lambda$1754/0x00000008022b3040.accept(Unknown Source) at jenkins.util.Listeners.lambda$notify$0(Listeners.java:59) at jenkins.util.Listeners$$Lambda$645/0x00000008015afc40.run(Unknown Source) at jenkins.util.Listeners.notify(Listeners.java:70) at hudson.model.listeners.ItemListener.fireOnUpdated(ItemListener.java:205) at com.cloudbees.hudson.plugins.folder.AbstractFolder.save(AbstractFolder.java:1317) - locked <0x00000006a6f6a2b0> (a org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject) at hudson.BulkChange.commit(BulkChange.java:98) at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.completeExisting(OrganizationFolder.java:1427) at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.complete(OrganizationFolder.java:1395) at jenkins.scm.api.trait.SCMNavigatorRequest.process(SCMNavigatorRequest.java:252) at jenkins.scm.api.trait.SCMNavigatorRequest.process(SCMNavigatorRequest.java:202) at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:1135) at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:535) at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:269) at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:167) at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:920) at hudson.model.ResourceController.execute(ResourceController.java:101) at hudson.model.Executor.run(Executor.java:442) "Periodic background build discarder thread": at jenkins.branch.MultiBranchProject.getProjectFactory(MultiBranchProject.java:343) - waiting to lock <0x00000006a6f6a2b0> (a org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject) at io.jenkins.blueocean.rest.impl.pipeline.MultiBranchPipelineImpl$PipelineFactoryImpl.getPipeline(MultiBranchPipelineImpl.java:276) at io.jenkins.blueocean.rest.impl.pipeline.MultiBranchPipelineImpl$PipelineFactoryImpl.resolve(MultiBranchPipelineImpl.java:291) at io.jenkins.blueocean.rest.impl.pipeline.OrganizationFolderPipelineImpl$OrganizationFolderFactory.resolve(OrganizationFolderPipelineImpl.java:187) at io.jenkins.blueocean.rest.factory.BluePipelineFactory.resolve(BluePipelineFactory.java:75) at io.jenkins.blueocean.service.embedded.BlueOceanUrlMapperImpl.getJobResource(BlueOceanUrlMapperImpl.java:80) at io.jenkins.blueocean.service.embedded.BlueOceanUrlMapperImpl.getUrl(BlueOceanUrlMapperImpl.java:49) at io.jenkins.blueocean.service.embedded.BlueOceanUrlObjectImpl.computeUrl(BlueOceanUrlObjectImpl.java:42) at io.jenkins.blueocean.service.embedded.BlueOceanUrlObjectImpl.<init>(BlueOceanUrlObjectImpl.java:21) at io.jenkins.blueocean.service.embedded.BlueOceanUrlObjectFactoryImpl.get(BlueOceanUrlObjectFactoryImpl.java:18) at io.jenkins.blueocean.service.embedded.TryBlueOceanMenu.createFor(TryBlueOceanMenu.java:50) at io.jenkins.blueocean.service.embedded.TryBlueOceanMenu.createFor(TryBlueOceanMenu.java:26) at hudson.model.Actionable.createFor(Actionable.java:115) at hudson.model.Actionable.getAllActions(Actionable.java:101) at hudson.model.Run.onLoad(Run.java:386) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:566) - locked <0x0000000692413998> (a java.lang.Object) at hudson.model.RunMap.retrieve(RunMap.java:233) at hudson.model.RunMap.retrieve(RunMap.java:61) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:657) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:639) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:537) - locked <0x00000006ce7a3318> (a hudson.model.RunMap) at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:240) at org.jenkinsci.plugins.workflow.job.WorkflowJob.getBuildByNumber(WorkflowJob.java:234) at org.jenkinsci.plugins.workflow.job.WorkflowJob.getBuildByNumber(WorkflowJob.java:105) at jenkins.model.PeepholePermalink.resolve(PeepholePermalink.java:103) at hudson.model.Job.getLastSuccessfulBuild(Job.java:943) at hudson.tasks.LogRotator.perform(LogRotator.java:152) at hudson.model.Job.logRotate(Job.java:473) at jenkins.model.JobGlobalBuildDiscarderStrategy.apply(JobGlobalBuildDiscarderStrategy.java:54) at jenkins.model.BackgroundGlobalBuildDiscarder.lambda$processJob$0(BackgroundGlobalBuildDiscarder.java:67) at jenkins.model.BackgroundGlobalBuildDiscarder$$Lambda$1181/0x00000008019f8440.accept(Unknown Source) at java.lang.Iterable.forEach(java.base@11.0.21/Iterable.java:75) at jenkins.model.BackgroundGlobalBuildDiscarder.processJob(BackgroundGlobalBuildDiscarder.java:61) at jenkins.model.BackgroundGlobalBuildDiscarder.execute(BackgroundGlobalBuildDiscarder.java:55) at hudson.model.AsyncPeriodicWork.lambda$doRun$0(AsyncPeriodicWork.java:102) at hudson.model.AsyncPeriodicWork$$Lambda$850/0x000000080178d840.run(Unknown Source) at java.lang.Thread.run(java.base@11.0.21/Thread.java:829)

          Another unresponsive Jenkins controller with deadlocked threads.  Here is the full threaddump that I hope helps.

          20240221_threaddump.txt

          John Lengeling added a comment - Another unresponsive Jenkins controller with deadlocked threads.  Here is the full threaddump that I hope helps. 20240221_threaddump.txt

          Here is our plugin list:

          {{{{Jenkins: 2.426.3
          OS: Linux - 4.14.328-248.540.amzn2.x86_64
          Java: 11.0.21 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)

          Exclusion:0.15
          Matrix-sorter-plugin:1.3
          Office-365-Connector:4.20.2
          PrioritySorter:5.0.0
          analysis-model-api:11.13.0
          ansicolor:1.0.4
          ant:497.v94e7d9fffa_b_9
          antisamy-markup-formatter:162.v0e6ec0fcfcf6
          any-buildstep:14.ve115ec1484f0
          apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
          artifact-manager-s3:822.vf129d4836c31
          artifactory:4.0.0
          audit-trail:359.va_7c0fb_cc4f1f
          authentication-tokens:1.53.v1c90fd9191a_b_
          authorize-project:1.7.1
          aws-credentials:218.v1b_e9466ec5da_
          aws-global-configuration:128.ve2c5685a_09c3
          aws-java-sdk:1.12.586-413.v6a_6c3a_420126
          aws-java-sdk-cloudformation:1.12.586-413.v6a_6c3a_420126
          aws-java-sdk-codebuild:1.12.586-413.v6a_6c3a_420126
          aws-java-sdk-ec2:1.12.586-413.v6a_6c3a_420126
          aws-java-sdk-ecr:1.12.586-413.v6a_6c3a_420126
          aws-java-sdk-ecs:1.12.586-413.v6a_6c3a_420126
          aws-java-sdk-efs:1.12.586-413.v6a_6c3a_420126
          aws-java-sdk-elasticbeanstalk:1.12.586-413.v6a_6c3a_420126
          aws-java-sdk-iam:1.12.586-413.v6a_6c3a_420126
          aws-java-sdk-kinesis:1.12.586-413.v6a_6c3a_420126
          aws-java-sdk-logs:1.12.586-413.v6a_6c3a_420126
          aws-java-sdk-minimal:1.12.586-413.v6a_6c3a_420126
          aws-java-sdk-secretsmanager:1.12.586-413.v6a_6c3a_420126
          aws-java-sdk-sns:1.12.586-413.v6a_6c3a_420126
          aws-java-sdk-sqs:1.12.586-413.v6a_6c3a_420126
          aws-java-sdk-ssm:1.12.586-413.v6a_6c3a_420126
          aws-secrets-manager-credentials-provider:1.213.vca_3f37306fed
          aws-secrets-manager-secret-source:1.72.v61781b_35c542
          azure-credentials:293.vb_d506148f506
          azure-sdk:157.v855da_0b_eb_dc2
          azure-vm-agents:883.v63c930b_025dc
          badge:1.9.1
          basic-branch-build-strategies:81.v05e333931c7d
          blueocean:1.27.9
          blueocean-autofavorite:1.2.5
          blueocean-bitbucket-pipeline:1.27.9
          blueocean-commons:1.27.9
          blueocean-config:1.27.9
          blueocean-core-js:1.27.9
          blueocean-dashboard:1.27.9
          blueocean-display-url:2.4.2
          blueocean-events:1.27.9
          blueocean-git-pipeline:1.27.9
          blueocean-github-pipeline:1.27.9
          blueocean-i18n:1.27.9
          blueocean-jira:1.27.9
          blueocean-jwt:1.27.9
          blueocean-personalization:1.27.9
          blueocean-pipeline-api-impl:1.27.9
          blueocean-pipeline-editor:1.27.9
          blueocean-pipeline-scm-api:1.27.9
          blueocean-rest:1.27.9
          blueocean-rest-impl:1.27.9
          blueocean-web:1.27.9
          bootstrap5-api:5.3.2-2
          bouncycastle-api:2.29
          branch-api:2.1135.v8de8e7899051
          build-cause-run-condition:0.1
          build-keeper-plugin:19.va_df8a_2c65123
          build-monitor-plugin:1.14-745.ve2023a_305f40
          build-name-setter:2.4.0
          build-timeout:1.31
          build-user-vars-plugin:1.9
          buildresult-trigger:0.18
          built-on-column:1.4
          caffeine-api:3.1.8-133.v17b_1ff2e0599
          categorized-view:1.13
          checks-api:2.0.2
          chucknorris:159.vdfe649cb_9c37
          claim:516.v36293563731d
          cloud-stats:320.v96b_65297a_4b_b_
          cloudbees-bitbucket-branch-source:856.v04c46c86f911
          cloudbees-folder:6.858.v898218f3609d
          clover:4.14.1.577.vb_ff5a_f69b_37b_
          cobertura:1.17
          code-coverage-api:4.99.0
          command-launcher:107.v773860566e2e
          commons-httpclient3-api:3.1-3
          commons-lang3-api:3.13.0-62.v7d18e55f51e2
          commons-text-api:1.11.0-94.v3e1f4a_926e49
          computer-queue-plugin:1.7
          conditional-buildstep:1.4.3
          config-file-provider:959.vcff671a_4518b_
          configuration-as-code:1746.vf1673cfe690a
          configuration-as-code-groovy:1.1
          copy-project-link:106.veb_028794a_844
          copyartifact:722.v0662a_9b_e22a_c
          coverage:1.5.0
          cppcheck:1.26
          cppncss:1.2
          credentials:1309.v8835d63eb_d8a_
          credentials-binding:642.v737c34dea_6c2
          cucumber-reports:5.8.0
          custom-tools-plugin:0.8
          cvs:2.19.1
          dashboard-view:2.495.v07e81500c3f2
          data-tables-api:1.13.8-1
          datetime-constraint:0.1.2
          delivery-pipeline-plugin:1.4.2
          depgraph-view:1.0.5
          description-setter:1.10-CUSTOM
          disable-github-multibranch-status:1.2
          display-url-api:2.200.vb_9327d658781
          docker-commons:439.va_3cb_0a_6a_fb_29
          docker-workflow:572.v950f58993843
          downstream-ext:73.vdda_16e6eb_0da
          dropdown-viewstabbar-plugin:1.7
          dtkit-api:3.0.2
          durable-task:523.va_a_22cf15d5e0
          dynamic-axis:1.0.3
          ec2:1648.vf3d852e00486
          echarts-api:5.4.3-1
          ecutest:2.42
          email-ext:2.102
          embeddable-build-status:412.v09da_db_1dee68
          envinject:2.908.v66a_774b_31d93
          envinject-api:1.199.v3ce31253ed13
          environment-variable-page-decoration:1.3.0
          excludeMatrixParent:1.1
          extended-choice-parameter:376.v2e02857547b_a_
          extended-read-permission:53.v6499940139e5
          extensible-choice-parameter:1.8.1
          external-monitor-job:215.v2e88e894db_f8
          external-workspace-manager:1.3.1
          ez-templates:1.3.5
          favorite:2.4.3
          file-parameters:316.va_83a_1221db_a_7
          fitnesse:1.36
          flexible-publish:0.16.1
          font-awesome-api:6.4.2-1
          forensics-api:2.3.0
          git:5.2.1
          git-client:4.5.0
          git-forensics:2.0.0
          git-parameter:0.9.19
          git-server:114.v068a_c7cc2574
          github:1.37.3.1
          github-api:1.318-461.v7a_c09c9fa_d63
          github-autostatus:3.6.2
          github-branch-source:1752.vc201a_0235d80
          github-issues:1.2.4
          github-oauth:597.ve0c3480fcb_d0
          github-scm-trait-commit-skip:0.4.0
          global-variable-string-parameter:1.2
          google-oauth-plugin:1.319.v5cb_58b_91b_389
          gradle:2.9
          groovy:457.v99900cb_85593
          groovy-postbuild:2.5
          h2-api:11.1.4.199-12.v9f4244395f7a_
          handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953
          htmlpublisher:1.32
          hubot-steps:95.va_30176518a_5a
          instance-identity:185.v303dc7c645f9
          ionicons-api:56.v1b_1c8c49374e
          ivy:2.5
          jackson2-api:2.15.3-372.v309620682326
          jacoco:3.3.5
          jakarta-activation-api:2.0.1-3
          jakarta-mail-api:2.0.1-3
          javadoc:243.vb_b_503b_b_45537
          javax-activation-api:1.2.0-6
          javax-mail-api:1.6.2-9
          jaxb:2.3.9-1
          jdk-tool:73.vddf737284550
          jenkins-design-language:1.27.9
          jenkins-flowdock-plugin:1.1.8
          jenkins-multijob-plugin:623.v03401733c9a_9
          jersey2-api:2.41-133.va_03323b_a_1396
          jfrog:1.5.0
          jira:3.12
          jjwt-api:0.11.5-77.v646c772fddb_0
          jnr-posix-api:3.1.18-1
          job-dsl:1.87
          job-restrictions:0.8
          jobConfigHistory:1229.v3039470161a_d
          jquery:1.12.4-1
          jquery3-api:3.7.1-1
          jsch:0.2.8-65.v052c39de79b_2
          junit:1240.vf9529b_881428
          junit-attachments:205.vc0677977deb_0
          keepSlaveOffline:1.0
          keyboard-shortcuts-plugin:1.4
          label-verifier:1.4
          ldap:711.vb_d1a_491714dc
          locale:314.v22ce953dfe9e
          lockable-resources:1218.va_3dd45e2b_fa_7
          log-parser:2.3.1
          logstash:2.5.0218.v0a_ff8fefc12b_
          mailer:463.vedf8358e006b_
          mapdb-api:1.0.9-28.vf251ce40855d
          mask-passwords:173.v6a_077a_291eb_5
          matrix-auth:3.2.1
          matrix-combinations-parameter:1.3.3
          matrix-project:822.824.v14451b_c0fd42
          maven-plugin:3.23
          measurement-plots:0.1
          mercurial:1260.vdfb_723cdcc81
          metrics:4.2.18-442.v02e107157925
          mina-sshd-api-common:2.11.0-86.v836f585d47fa_
          mina-sshd-api-core:2.11.0-86.v836f585d47fa_
          monitoring:1.95.0
          multi-branch-priority-sorter:1.0
          multibranch-action-triggers:1.8.6
          multibranch-build-strategy-extension:1.1.0
          nested-view:1.33
          next-build-number:1.8
          node-iterator-api:55.v3b_77d4032326
          nodelabelparameter:1.12.0
          oauth-credentials:0.646.v02b_66dc03d2e
          okhttp-api:4.11.0-157.v6852a_a_fa_ec11
          pam-auth:1.10
          parameterized-scheduler:255.v73827fcdf618
          parameterized-trigger:787.v665fcf2a_830b_
          pipeline-aggregator-view:1.15
          pipeline-build-step:516.v8ee60a_81c5b_9
          pipeline-github:2.8-155.8eab375ac9f8
          pipeline-github-lib:42.v0739460cda_c4
          pipeline-githubnotify-step:49.vf37bf92d2bc8
          pipeline-graph-analysis:202.va_d268e64deb_3
          pipeline-groovy-lib:689.veec561a_dee13
          pipeline-input-step:477.v339683a_8d55e
          pipeline-maven:1362.vee39a_d4b_02b_1
          pipeline-maven-api:1362.vee39a_d4b_02b_1
          pipeline-milestone-step:111.v449306f708b_7
          pipeline-model-api:2.2151.ve32c9d209a_3f
          pipeline-model-definition:2.2151.ve32c9d209a_3f
          pipeline-model-extensions:2.2151.ve32c9d209a_3f
          pipeline-multibranch-defaults:2.1
          pipeline-rest-api:2.34
          pipeline-stage-step:305.ve96d0205c1c6
          pipeline-stage-tags-metadata:2.2151.ve32c9d209a_3f
          pipeline-stage-view:2.34
          pipeline-timeline:1.0.3
          pipeline-utility-steps:2.16.0
          plain-credentials:143.v1b_df8b_d3b_e48
          plot:2.1.12
          plugin-usage-plugin:4.2
          plugin-util-api:3.6.0
          pollscm:1.5
          postbuild-task:1.9
          postbuildscript:3.2.0-550.v88192b_d3e922
          preSCMbuildstep:71.v1f2990a_37e27
          prism-api:1.29.0-9
          progress-bar-column-plugin:11.vdef198c2d6c1
          project-build-times:1.2.1
          project-description-setter:1.2
          project-health-report:1.2
          promoted-builds:936.va_571a_a_b_f8da_5
          publish-over:0.22
          publish-over-cifs:0.16
          pubsub-light:1.18
          python:1.3
          random-string-parameter:1.0
          read-only-configurations:1.10
          rebuild:330.v645b_7df10e2a_
          release:2.18
          remote-file:1.24
          resource-disposer:0.23
          rich-text-publisher-plugin:1.5
          role-strategy:689.v731678c3e0eb_
          run-condition:1.7
          saferestart:0.7
          saml:4.429.v9a_781a_61f1da_
          scm-api:683.vb_16722fb_b_80b_
          script-security:1294.v99333c047434
          sectioned-view:1.25
          secure-requester-whitelist:67.vca_a_d9205723f
          seed:2.1.4
          shelve-project-plugin:3.2
          show-build-parameters:1.0
          sidebar-link:2.4.1
          simple-build-for-pipeline:0.2
          simple-theme-plugin:176.v39740c03a_a_f5
          slave-setup:1.16
          snakeyaml-api:2.2-111.vc6598e30cc65
          sse-gateway:1.26
          ssh-agent:346.vda_a_c4f2c8e50
          ssh-credentials:308.ve4497b_ccd8f4
          ssh-slaves:2.916.vd17b_43357ce4
          sshd:3.312.v1c601b_c83b_0e
          statusmonitor:1.3
          structs:325.vcb_307d2a_2782
          subversion:2.17.3
          summary_report:1.15
          test-stability:2.3
          throttle-concurrents:2.14
          timestamper:1.26
          token-macro:384.vf35b_f26814ec
          trilead-api:2.84.v72119de229b_7
          urltrigger:1.02
          validating-string-parameter:183.v3748e79b_9737
          variant:60.v7290fc0eb_b_cd
          versioncolumn:217.v16457a_b_eb_7eb_
          view-job-filters:369.ve0513a_a_f5524
          warnings-ng:10.5.1
          webhook-step:327.vb_58f208a_7783
          workflow-aggregator:596.v8c21c963d92d
          workflow-api:1283.v99c10937efcb_
          workflow-basic-steps:1042.ve7b_140c4a_e0c
          workflow-cps:3812.vc3031a_b_a_c955
          workflow-durable-task-step:1289.v4d3e7b_01546b_
          workflow-job:1360.vc6700e3136f5
          workflow-multibranch:756.v891d88f2cd46
          workflow-remote-loader:1.6
          workflow-scm-step:415.v434365564324
          workflow-step-api:639.v6eca_cd8c04a_a_
          workflow-support:865.v43e78cc44e0d
          ws-cleanup:0.45
          xtrigger-api:1.0
          xunit:3.1.3
          }}}}

          John Lengeling added a comment - Here is our plugin list: {{{{Jenkins: 2.426.3 OS: Linux - 4.14.328-248.540.amzn2.x86_64 Java: 11.0.21 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) — Exclusion:0.15 Matrix-sorter-plugin:1.3 Office-365-Connector:4.20.2 PrioritySorter:5.0.0 analysis-model-api:11.13.0 ansicolor:1.0.4 ant:497.v94e7d9fffa_b_9 antisamy-markup-formatter:162.v0e6ec0fcfcf6 any-buildstep:14.ve115ec1484f0 apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 artifact-manager-s3:822.vf129d4836c31 artifactory:4.0.0 audit-trail:359.va_7c0fb_cc4f1f authentication-tokens:1.53.v1c90fd9191a_b_ authorize-project:1.7.1 aws-credentials:218.v1b_e9466ec5da_ aws-global-configuration:128.ve2c5685a_09c3 aws-java-sdk:1.12.586-413.v6a_6c3a_420126 aws-java-sdk-cloudformation:1.12.586-413.v6a_6c3a_420126 aws-java-sdk-codebuild:1.12.586-413.v6a_6c3a_420126 aws-java-sdk-ec2:1.12.586-413.v6a_6c3a_420126 aws-java-sdk-ecr:1.12.586-413.v6a_6c3a_420126 aws-java-sdk-ecs:1.12.586-413.v6a_6c3a_420126 aws-java-sdk-efs:1.12.586-413.v6a_6c3a_420126 aws-java-sdk-elasticbeanstalk:1.12.586-413.v6a_6c3a_420126 aws-java-sdk-iam:1.12.586-413.v6a_6c3a_420126 aws-java-sdk-kinesis:1.12.586-413.v6a_6c3a_420126 aws-java-sdk-logs:1.12.586-413.v6a_6c3a_420126 aws-java-sdk-minimal:1.12.586-413.v6a_6c3a_420126 aws-java-sdk-secretsmanager:1.12.586-413.v6a_6c3a_420126 aws-java-sdk-sns:1.12.586-413.v6a_6c3a_420126 aws-java-sdk-sqs:1.12.586-413.v6a_6c3a_420126 aws-java-sdk-ssm:1.12.586-413.v6a_6c3a_420126 aws-secrets-manager-credentials-provider:1.213.vca_3f37306fed aws-secrets-manager-secret-source:1.72.v61781b_35c542 azure-credentials:293.vb_d506148f506 azure-sdk:157.v855da_0b_eb_dc2 azure-vm-agents:883.v63c930b_025dc badge:1.9.1 basic-branch-build-strategies:81.v05e333931c7d blueocean:1.27.9 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.27.9 blueocean-commons:1.27.9 blueocean-config:1.27.9 blueocean-core-js:1.27.9 blueocean-dashboard:1.27.9 blueocean-display-url:2.4.2 blueocean-events:1.27.9 blueocean-git-pipeline:1.27.9 blueocean-github-pipeline:1.27.9 blueocean-i18n:1.27.9 blueocean-jira:1.27.9 blueocean-jwt:1.27.9 blueocean-personalization:1.27.9 blueocean-pipeline-api-impl:1.27.9 blueocean-pipeline-editor:1.27.9 blueocean-pipeline-scm-api:1.27.9 blueocean-rest:1.27.9 blueocean-rest-impl:1.27.9 blueocean-web:1.27.9 bootstrap5-api:5.3.2-2 bouncycastle-api:2.29 branch-api:2.1135.v8de8e7899051 build-cause-run-condition:0.1 build-keeper-plugin:19.va_df8a_2c65123 build-monitor-plugin:1.14-745.ve2023a_305f40 build-name-setter:2.4.0 build-timeout:1.31 build-user-vars-plugin:1.9 buildresult-trigger:0.18 built-on-column:1.4 caffeine-api:3.1.8-133.v17b_1ff2e0599 categorized-view:1.13 checks-api:2.0.2 chucknorris:159.vdfe649cb_9c37 claim:516.v36293563731d cloud-stats:320.v96b_65297a_4b_b_ cloudbees-bitbucket-branch-source:856.v04c46c86f911 cloudbees-folder:6.858.v898218f3609d clover:4.14.1.577.vb_ff5a_f69b_37b_ cobertura:1.17 code-coverage-api:4.99.0 command-launcher:107.v773860566e2e commons-httpclient3-api:3.1-3 commons-lang3-api:3.13.0-62.v7d18e55f51e2 commons-text-api:1.11.0-94.v3e1f4a_926e49 computer-queue-plugin:1.7 conditional-buildstep:1.4.3 config-file-provider:959.vcff671a_4518b_ configuration-as-code:1746.vf1673cfe690a configuration-as-code-groovy:1.1 copy-project-link:106.veb_028794a_844 copyartifact:722.v0662a_9b_e22a_c coverage:1.5.0 cppcheck:1.26 cppncss:1.2 credentials:1309.v8835d63eb_d8a_ credentials-binding:642.v737c34dea_6c2 cucumber-reports:5.8.0 custom-tools-plugin:0.8 cvs:2.19.1 dashboard-view:2.495.v07e81500c3f2 data-tables-api:1.13.8-1 datetime-constraint:0.1.2 delivery-pipeline-plugin:1.4.2 depgraph-view:1.0.5 description-setter:1.10-CUSTOM disable-github-multibranch-status:1.2 display-url-api:2.200.vb_9327d658781 docker-commons:439.va_3cb_0a_6a_fb_29 docker-workflow:572.v950f58993843 downstream-ext:73.vdda_16e6eb_0da dropdown-viewstabbar-plugin:1.7 dtkit-api:3.0.2 durable-task:523.va_a_22cf15d5e0 dynamic-axis:1.0.3 ec2:1648.vf3d852e00486 echarts-api:5.4.3-1 ecutest:2.42 email-ext:2.102 embeddable-build-status:412.v09da_db_1dee68 envinject:2.908.v66a_774b_31d93 envinject-api:1.199.v3ce31253ed13 environment-variable-page-decoration:1.3.0 excludeMatrixParent:1.1 extended-choice-parameter:376.v2e02857547b_a_ extended-read-permission:53.v6499940139e5 extensible-choice-parameter:1.8.1 external-monitor-job:215.v2e88e894db_f8 external-workspace-manager:1.3.1 ez-templates:1.3.5 favorite:2.4.3 file-parameters:316.va_83a_1221db_a_7 fitnesse:1.36 flexible-publish:0.16.1 font-awesome-api:6.4.2-1 forensics-api:2.3.0 git:5.2.1 git-client:4.5.0 git-forensics:2.0.0 git-parameter:0.9.19 git-server:114.v068a_c7cc2574 github:1.37.3.1 github-api:1.318-461.v7a_c09c9fa_d63 github-autostatus:3.6.2 github-branch-source:1752.vc201a_0235d80 github-issues:1.2.4 github-oauth:597.ve0c3480fcb_d0 github-scm-trait-commit-skip:0.4.0 global-variable-string-parameter:1.2 google-oauth-plugin:1.319.v5cb_58b_91b_389 gradle:2.9 groovy:457.v99900cb_85593 groovy-postbuild:2.5 h2-api:11.1.4.199-12.v9f4244395f7a_ handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 htmlpublisher:1.32 hubot-steps:95.va_30176518a_5a instance-identity:185.v303dc7c645f9 ionicons-api:56.v1b_1c8c49374e ivy:2.5 jackson2-api:2.15.3-372.v309620682326 jacoco:3.3.5 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javadoc:243.vb_b_503b_b_45537 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.9-1 jdk-tool:73.vddf737284550 jenkins-design-language:1.27.9 jenkins-flowdock-plugin:1.1.8 jenkins-multijob-plugin:623.v03401733c9a_9 jersey2-api:2.41-133.va_03323b_a_1396 jfrog:1.5.0 jira:3.12 jjwt-api:0.11.5-77.v646c772fddb_0 jnr-posix-api:3.1.18-1 job-dsl:1.87 job-restrictions:0.8 jobConfigHistory:1229.v3039470161a_d jquery:1.12.4-1 jquery3-api:3.7.1-1 jsch:0.2.8-65.v052c39de79b_2 junit:1240.vf9529b_881428 junit-attachments:205.vc0677977deb_0 keepSlaveOffline:1.0 keyboard-shortcuts-plugin:1.4 label-verifier:1.4 ldap:711.vb_d1a_491714dc locale:314.v22ce953dfe9e lockable-resources:1218.va_3dd45e2b_fa_7 log-parser:2.3.1 logstash:2.5.0218.v0a_ff8fefc12b_ mailer:463.vedf8358e006b_ mapdb-api:1.0.9-28.vf251ce40855d mask-passwords:173.v6a_077a_291eb_5 matrix-auth:3.2.1 matrix-combinations-parameter:1.3.3 matrix-project:822.824.v14451b_c0fd42 maven-plugin:3.23 measurement-plots:0.1 mercurial:1260.vdfb_723cdcc81 metrics:4.2.18-442.v02e107157925 mina-sshd-api-common:2.11.0-86.v836f585d47fa_ mina-sshd-api-core:2.11.0-86.v836f585d47fa_ monitoring:1.95.0 multi-branch-priority-sorter:1.0 multibranch-action-triggers:1.8.6 multibranch-build-strategy-extension:1.1.0 nested-view:1.33 next-build-number:1.8 node-iterator-api:55.v3b_77d4032326 nodelabelparameter:1.12.0 oauth-credentials:0.646.v02b_66dc03d2e okhttp-api:4.11.0-157.v6852a_a_fa_ec11 pam-auth:1.10 parameterized-scheduler:255.v73827fcdf618 parameterized-trigger:787.v665fcf2a_830b_ pipeline-aggregator-view:1.15 pipeline-build-step:516.v8ee60a_81c5b_9 pipeline-github:2.8-155.8eab375ac9f8 pipeline-github-lib:42.v0739460cda_c4 pipeline-githubnotify-step:49.vf37bf92d2bc8 pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-groovy-lib:689.veec561a_dee13 pipeline-input-step:477.v339683a_8d55e pipeline-maven:1362.vee39a_d4b_02b_1 pipeline-maven-api:1362.vee39a_d4b_02b_1 pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2151.ve32c9d209a_3f pipeline-model-definition:2.2151.ve32c9d209a_3f pipeline-model-extensions:2.2151.ve32c9d209a_3f pipeline-multibranch-defaults:2.1 pipeline-rest-api:2.34 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2151.ve32c9d209a_3f pipeline-stage-view:2.34 pipeline-timeline:1.0.3 pipeline-utility-steps:2.16.0 plain-credentials:143.v1b_df8b_d3b_e48 plot:2.1.12 plugin-usage-plugin:4.2 plugin-util-api:3.6.0 pollscm:1.5 postbuild-task:1.9 postbuildscript:3.2.0-550.v88192b_d3e922 preSCMbuildstep:71.v1f2990a_37e27 prism-api:1.29.0-9 progress-bar-column-plugin:11.vdef198c2d6c1 project-build-times:1.2.1 project-description-setter:1.2 project-health-report:1.2 promoted-builds:936.va_571a_a_b_f8da_5 publish-over:0.22 publish-over-cifs:0.16 pubsub-light:1.18 python:1.3 random-string-parameter:1.0 read-only-configurations:1.10 rebuild:330.v645b_7df10e2a_ release:2.18 remote- file:1.24 resource-disposer:0.23 rich-text-publisher-plugin:1.5 role-strategy:689.v731678c3e0eb_ run-condition:1.7 saferestart:0.7 saml:4.429.v9a_781a_61f1da_ scm-api:683.vb_16722fb_b_80b_ script-security:1294.v99333c047434 sectioned-view:1.25 secure-requester-whitelist:67.vca_a_d9205723f seed:2.1.4 shelve-project-plugin:3.2 show-build-parameters:1.0 sidebar-link:2.4.1 simple-build-for-pipeline:0.2 simple-theme-plugin:176.v39740c03a_a_f5 slave-setup:1.16 snakeyaml-api:2.2-111.vc6598e30cc65 sse-gateway:1.26 ssh-agent:346.vda_a_c4f2c8e50 ssh-credentials:308.ve4497b_ccd8f4 ssh-slaves:2.916.vd17b_43357ce4 sshd:3.312.v1c601b_c83b_0e statusmonitor:1.3 structs:325.vcb_307d2a_2782 subversion:2.17.3 summary_report:1.15 test-stability:2.3 throttle-concurrents:2.14 timestamper:1.26 token-macro:384.vf35b_f26814ec trilead-api:2.84.v72119de229b_7 urltrigger:1.02 validating-string-parameter:183.v3748e79b_9737 variant:60.v7290fc0eb_b_cd versioncolumn:217.v16457a_b_eb_7eb_ view-job-filters:369.ve0513a_a_f5524 warnings-ng:10.5.1 webhook-step:327.vb_58f208a_7783 workflow-aggregator:596.v8c21c963d92d workflow-api:1283.v99c10937efcb_ workflow-basic-steps:1042.ve7b_140c4a_e0c workflow-cps:3812.vc3031a_b_a_c955 workflow-durable-task-step:1289.v4d3e7b_01546b_ workflow-job:1360.vc6700e3136f5 workflow-multibranch:756.v891d88f2cd46 workflow-remote-loader:1.6 workflow-scm-step:415.v434365564324 workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:865.v43e78cc44e0d ws-cleanup:0.45 xtrigger-api:1.0 xunit:3.1.3 }}}}

          Devin Nusbaum added a comment -

          https://plugins.jenkins.io/generic-event/ seems to be present in all stack traces posted here as well as in JENKINS-73715. In particular, it looks like GenericEventItemListener is trying to do too much synchronously from various item listener methods, which are typically called while a monitor is held for the item in question. Even more specifically, the problem seems largely related to the use of ExportedBeanProcessor in generic-event, which is computing (among other things) health checks for the items being processed, which for multibranch projects causes some of their builds to be loaded, which requires various locks, and all of this is happening while the original item is locked. Complex listeners should perform their logic on a background thread, for example by submitting tasks to jenkins.util.Timer.

          The locking in MultiBranchProject.getProjectFactory could perhaps be improved as well to fix some cases of this issue, see JENKINS-74777.

          Devin Nusbaum added a comment - https://plugins.jenkins.io/generic-event/ seems to be present in all stack traces posted here as well as in JENKINS-73715 . In particular, it looks like GenericEventItemListener is trying to do too much synchronously from various item listener methods, which are typically called while a monitor is held for the item in question. Even more specifically, the problem seems largely related to the use of ExportedBeanProcessor in generic-event , which is computing (among other things) health checks for the items being processed, which for multibranch projects causes some of their builds to be loaded, which requires various locks, and all of this is happening while the original item is locked. Complex listeners should perform their logic on a background thread, for example by submitting tasks to jenkins.util.Timer . The locking in MultiBranchProject.getProjectFactory could perhaps be improved as well to fix some cases of this issue, see JENKINS-74777 .

            Unassigned Unassigned
            johnlengeling John Lengeling
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: