-
Bug
-
Resolution: Unresolved
-
Critical
-
Running Jenkins controllers on AWS ECS. Using EFS for Jenkins home storage.
Jenkins Controller: 2.246.1, 2.426.3
github-branch-source-plugin: 1752.vc201a_0235d80
cloudbees-folder-plugin: 6.858.v898218f3609d
-
Powered by SuggestiMate
We are seeing occasional deadlocked/blocked threads on the Github Organization Scans due to possible health checks/WeatherColumn.
From the catalina.log:
Some health checks are reporting as unhealthy: [thread-deadlock : [Executor #-1 for Built-In Node : executing OrganizationScan[CP-Models] locked on hudson.model.RunMap@50446244 (owned by Handling GET /job/CP-Models/ from 192.168.1.143 : Jetty (winstone)-345663 View/index.jelly WeatherColumn/column.jelly): at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:526) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:502) at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:435) at jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:254) at org.jenkinsci.plugins.workflow.job.WorkflowJob.getLastBuild(WorkflowJob.java:242) at org.jenkinsci.plugins.workflow.job.WorkflowJob.getLastBuild(WorkflowJob.java:105) at hudson.model.Job.getBuildHealthReports(Job.java:1200) at hudson.model.Job.getBuildHealth(Job.java:1193) at com.cloudbees.hudson.plugins.folder.health.FolderHealthMetric.getHealthReport(FolderHealthMetric.java:48) at com.cloudbees.hudson.plugins.folder.health.WorstChildHealthMetric$ReporterImpl.observe(WorstChildHealthMetric.java:86) at com.cloudbees.hudson.plugins.folder.AbstractFolder.getBuildHealthReports(AbstractFolder.java:924) at java.base@11.0.21/java.lang.invoke.DirectMethodHandle$Holder.invokeVirtual(DirectMethodHandle$Holder) at java.base@11.0.21/java.lang.invoke.LambdaForm$MH/0x0000000800277840.invoke(LambdaForm$MH) at java.base@11.0.21/java.lang.invoke.LambdaForm$MH/0x00000008002a7440.invoke_MT(LambdaForm$MH) at org.kohsuke.stapler.export.MethodProperty.getValue(MethodProperty.java:76) at org.kohsuke.stapler.export.ExportInterceptor$1.getValue(ExportInterceptor.java:46) at io.jenkins.plugins.generic.event.json.ExportedBeanProcessor$IgnoreURLExportInterceptor.getValue(ExportedBeanProcessor.java:71) at org.kohsuke.stapler.export.Property.writeTo(Property.java:136) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:222) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:218) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:218) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:218) at org.kohsuke.stapler.export.Model.writeTo(Model.java:193) at org.kohsuke.stapler.export.Model.writeTo(Model.java:213) at org.kohsuke.stapler.export.Model.writeTo(Model.java:181) at io.jenkins.plugins.generic.event.json.ExportedBeanProcessor.processBean(ExportedBeanProcessor.java:43) at net.sf.json.JSONObject._fromBean(JSONObject.java:619) at net.sf.json.JSONObject.fromObject(JSONObject.java:169) at net.sf.json.AbstractJSON._processValue(AbstractJSON.java:250) ...
See the attached full catalina.log stack trace. I also attached the thread dump showing the BLOCKED threads.
This looked like the Weather health metric/folder performance issue as documented here by Cloudbees. But we have disabled all health metrics on all folders including Github Organization folders and we still have deadlocks/blocked threads.
How to disable the weather column to resolve instance slowness? (cloudbees.com)
When this occurs, we have to restart the Jenkins controller. This seems to have started after we upgraded to 2.246.1 back in December as we have had rock solid controllers on 2.387.3 which was the previous version. We are currently running 2.246.3 and see the issue.
- 20240205_threaddump.txt
- 679 kB
- 20240221_threaddump.txt
- 2.71 MB
- is duplicated by
-
JENKINS-73715 Deadlock between Queue.maintain and BranchIndexing
-
- Closed
-
- relates to
-
JENKINS-74777 Improve locking around jenkins.branch.MultiBranchProject.getProjectFactory
-
- Closed
-
[JENKINS-72678] Thread Deadlock during GitHub Organization Scan and WeatherColumn
Any update on this? Another occurance with a Java Level threadlock related to WorstChildHealthMetric:
"class org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber$SCMHeadEventImpl Sun Feb 11 12:42:50 CST 2024 / SCMEvent [#1]": waiting to lock monitor 0x00007f596c63fa80 (object 0x00000006a6f6a2b0, a org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject), which is held by "Executor #-1 for Built-In Node : executing OrganizationScan[Org1]" "Executor #-1 for Built-In Node : executing OrganizationScan[Org1]": waiting to lock monitor 0x00007f59308d2f00 (object 0x00000006ce7a3318, a hudson.model.RunMap), which is held by "Periodic background build discarder thread" "Periodic background build discarder thread": waiting to lock monitor 0x00007f596c63fa80 (object 0x00000006a6f6a2b0, a org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject), which is held by "Executor #-1 for Built-In Node : executing OrganizationScan[Org1]" Java stack information for the threads listed above: =================================================== "class org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber$SCMHeadEventImpl Sun Feb 11 12:42:50 CST 2024 / SCMEvent [#1]": at jenkins.branch.MultiBranchProject.getProjectFactory(MultiBranchProject.java:343) - waiting to lock <0x00000006a6f6a2b0> (a org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject) at jenkins.branch.MultiBranchProject$SCMEventListenerImpl.processHeadUpdate(MultiBranchProject.java:1436) at jenkins.branch.MultiBranchProject$SCMEventListenerImpl.onSCMHeadEvent(MultiBranchProject.java:1218) at jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:246) at jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:229) at jenkins.scm.api.SCMEvent$Dispatcher.run(SCMEvent.java:545) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67) at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.21/Executors.java:515) at java.util.concurrent.FutureTask.run(java.base@11.0.21/FutureTask.java:264) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.21/ScheduledThreadPoolExecutor.java:304) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.21/ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.21/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.21/Thread.java:829) "Executor #-1 for Built-In Node : executing OrganizationScan[Org1]": at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:526) - waiting to lock <0x00000006ce7a3318> (a hudson.model.RunMap) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:502) at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:435) at jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:254) at org.jenkinsci.plugins.workflow.job.WorkflowJob.getLastBuild(WorkflowJob.java:242) at org.jenkinsci.plugins.workflow.job.WorkflowJob.getLastBuild(WorkflowJob.java:105) at hudson.model.Job.getBuildHealthReports(Job.java:1200) at hudson.model.Job.getBuildHealth(Job.java:1193) at com.cloudbees.hudson.plugins.folder.health.FolderHealthMetric.getHealthReport(FolderHealthMetric.java:48) at com.cloudbees.hudson.plugins.folder.health.WorstChildHealthMetric$ReporterImpl.observe(WorstChildHealthMetric.java:86) at com.cloudbees.hudson.plugins.folder.AbstractFolder.getBuildHealthReports(AbstractFolder.java:924) at java.lang.invoke.DirectMethodHandle$Holder.invokeVirtual(java.base@11.0.21/DirectMethodHandle$Holder) at java.lang.invoke.LambdaForm$MH/0x0000000800277040.invoke(java.base@11.0.21/LambdaForm$MH) at java.lang.invoke.LambdaForm$MH/0x00000008002a7440.invoke_MT(java.base@11.0.21/LambdaForm$MH) at org.kohsuke.stapler.export.MethodProperty.getValue(MethodProperty.java:76) at org.kohsuke.stapler.export.ExportInterceptor$1.getValue(ExportInterceptor.java:46) at io.jenkins.plugins.generic.event.json.ExportedBeanProcessor$IgnoreURLExportInterceptor.getValue(ExportedBeanProcessor.java:71) at org.kohsuke.stapler.export.Property.writeTo(Property.java:136) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:222) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:218) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:218) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:218) at org.kohsuke.stapler.export.Model.writeTo(Model.java:193) at org.kohsuke.stapler.export.Model.writeTo(Model.java:213) at org.kohsuke.stapler.export.Model.writeTo(Model.java:181) at io.jenkins.plugins.generic.event.json.ExportedBeanProcessor.processBean(ExportedBeanProcessor.java:43) at net.sf.json.JSONObject._fromBean(JSONObject.java:619) at net.sf.json.JSONObject.fromObject(JSONObject.java:169) at net.sf.json.AbstractJSON._processValue(AbstractJSON.java:250) at net.sf.json.JSONObject._processValue(JSONObject.java:2599) at net.sf.json.JSONObject.processValue(JSONObject.java:2665) at net.sf.json.JSONObject.setInternal(JSONObject.java:2680) at net.sf.json.JSONObject.setValue(JSONObject.java:1376) at net.sf.json.JSONObject.defaultBeanProcessing(JSONObject.java:708) at net.sf.json.JSONObject._fromBean(JSONObject.java:642) at net.sf.json.JSONObject.fromObject(JSONObject.java:169) at io.jenkins.plugins.generic.event.HttpEventSender$CustomCloseableHttpAsyncClient.sendPost(HttpEventSender.java:90) at io.jenkins.plugins.generic.event.HttpEventSender.send(HttpEventSender.java:45) at io.jenkins.plugins.generic.event.listener.GenericEventItemListener.onUpdated(GenericEventItemListener.java:118) at hudson.model.listeners.ItemListener.lambda$fireOnUpdated$2(ItemListener.java:205) at hudson.model.listeners.ItemListener$$Lambda$1754/0x00000008022b3040.accept(Unknown Source) at jenkins.util.Listeners.lambda$notify$0(Listeners.java:59) at jenkins.util.Listeners$$Lambda$645/0x00000008015afc40.run(Unknown Source) at jenkins.util.Listeners.notify(Listeners.java:70) at hudson.model.listeners.ItemListener.fireOnUpdated(ItemListener.java:205) at com.cloudbees.hudson.plugins.folder.AbstractFolder.save(AbstractFolder.java:1317) - locked <0x00000006a6f6a2b0> (a org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject) at hudson.BulkChange.commit(BulkChange.java:98) at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.completeExisting(OrganizationFolder.java:1427) at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.complete(OrganizationFolder.java:1395) at jenkins.scm.api.trait.SCMNavigatorRequest.process(SCMNavigatorRequest.java:252) at jenkins.scm.api.trait.SCMNavigatorRequest.process(SCMNavigatorRequest.java:202) at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:1135) at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:535) at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:269) at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:167) at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:920) at hudson.model.ResourceController.execute(ResourceController.java:101) at hudson.model.Executor.run(Executor.java:442) "Periodic background build discarder thread": at jenkins.branch.MultiBranchProject.getProjectFactory(MultiBranchProject.java:343) - waiting to lock <0x00000006a6f6a2b0> (a org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject) at io.jenkins.blueocean.rest.impl.pipeline.MultiBranchPipelineImpl$PipelineFactoryImpl.getPipeline(MultiBranchPipelineImpl.java:276) at io.jenkins.blueocean.rest.impl.pipeline.MultiBranchPipelineImpl$PipelineFactoryImpl.resolve(MultiBranchPipelineImpl.java:291) at io.jenkins.blueocean.rest.impl.pipeline.OrganizationFolderPipelineImpl$OrganizationFolderFactory.resolve(OrganizationFolderPipelineImpl.java:187) at io.jenkins.blueocean.rest.factory.BluePipelineFactory.resolve(BluePipelineFactory.java:75) at io.jenkins.blueocean.service.embedded.BlueOceanUrlMapperImpl.getJobResource(BlueOceanUrlMapperImpl.java:80) at io.jenkins.blueocean.service.embedded.BlueOceanUrlMapperImpl.getUrl(BlueOceanUrlMapperImpl.java:49) at io.jenkins.blueocean.service.embedded.BlueOceanUrlObjectImpl.computeUrl(BlueOceanUrlObjectImpl.java:42) at io.jenkins.blueocean.service.embedded.BlueOceanUrlObjectImpl.<init>(BlueOceanUrlObjectImpl.java:21) at io.jenkins.blueocean.service.embedded.BlueOceanUrlObjectFactoryImpl.get(BlueOceanUrlObjectFactoryImpl.java:18) at io.jenkins.blueocean.service.embedded.TryBlueOceanMenu.createFor(TryBlueOceanMenu.java:50) at io.jenkins.blueocean.service.embedded.TryBlueOceanMenu.createFor(TryBlueOceanMenu.java:26) at hudson.model.Actionable.createFor(Actionable.java:115) at hudson.model.Actionable.getAllActions(Actionable.java:101) at hudson.model.Run.onLoad(Run.java:386) at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:566) - locked <0x0000000692413998> (a java.lang.Object) at hudson.model.RunMap.retrieve(RunMap.java:233) at hudson.model.RunMap.retrieve(RunMap.java:61) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:657) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:639) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:537) - locked <0x00000006ce7a3318> (a hudson.model.RunMap) at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:240) at org.jenkinsci.plugins.workflow.job.WorkflowJob.getBuildByNumber(WorkflowJob.java:234) at org.jenkinsci.plugins.workflow.job.WorkflowJob.getBuildByNumber(WorkflowJob.java:105) at jenkins.model.PeepholePermalink.resolve(PeepholePermalink.java:103) at hudson.model.Job.getLastSuccessfulBuild(Job.java:943) at hudson.tasks.LogRotator.perform(LogRotator.java:152) at hudson.model.Job.logRotate(Job.java:473) at jenkins.model.JobGlobalBuildDiscarderStrategy.apply(JobGlobalBuildDiscarderStrategy.java:54) at jenkins.model.BackgroundGlobalBuildDiscarder.lambda$processJob$0(BackgroundGlobalBuildDiscarder.java:67) at jenkins.model.BackgroundGlobalBuildDiscarder$$Lambda$1181/0x00000008019f8440.accept(Unknown Source) at java.lang.Iterable.forEach(java.base@11.0.21/Iterable.java:75) at jenkins.model.BackgroundGlobalBuildDiscarder.processJob(BackgroundGlobalBuildDiscarder.java:61) at jenkins.model.BackgroundGlobalBuildDiscarder.execute(BackgroundGlobalBuildDiscarder.java:55) at hudson.model.AsyncPeriodicWork.lambda$doRun$0(AsyncPeriodicWork.java:102) at hudson.model.AsyncPeriodicWork$$Lambda$850/0x000000080178d840.run(Unknown Source) at java.lang.Thread.run(java.base@11.0.21/Thread.java:829)
Another unresponsive Jenkins controller with deadlocked threads. Here is the full threaddump that I hope helps.
Here is our plugin list:
{{{{Jenkins: 2.426.3
OS: Linux - 4.14.328-248.540.amzn2.x86_64
Java: 11.0.21 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
—
Exclusion:0.15
Matrix-sorter-plugin:1.3
Office-365-Connector:4.20.2
PrioritySorter:5.0.0
analysis-model-api:11.13.0
ansicolor:1.0.4
ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:162.v0e6ec0fcfcf6
any-buildstep:14.ve115ec1484f0
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
artifact-manager-s3:822.vf129d4836c31
artifactory:4.0.0
audit-trail:359.va_7c0fb_cc4f1f
authentication-tokens:1.53.v1c90fd9191a_b_
authorize-project:1.7.1
aws-credentials:218.v1b_e9466ec5da_
aws-global-configuration:128.ve2c5685a_09c3
aws-java-sdk:1.12.586-413.v6a_6c3a_420126
aws-java-sdk-cloudformation:1.12.586-413.v6a_6c3a_420126
aws-java-sdk-codebuild:1.12.586-413.v6a_6c3a_420126
aws-java-sdk-ec2:1.12.586-413.v6a_6c3a_420126
aws-java-sdk-ecr:1.12.586-413.v6a_6c3a_420126
aws-java-sdk-ecs:1.12.586-413.v6a_6c3a_420126
aws-java-sdk-efs:1.12.586-413.v6a_6c3a_420126
aws-java-sdk-elasticbeanstalk:1.12.586-413.v6a_6c3a_420126
aws-java-sdk-iam:1.12.586-413.v6a_6c3a_420126
aws-java-sdk-kinesis:1.12.586-413.v6a_6c3a_420126
aws-java-sdk-logs:1.12.586-413.v6a_6c3a_420126
aws-java-sdk-minimal:1.12.586-413.v6a_6c3a_420126
aws-java-sdk-secretsmanager:1.12.586-413.v6a_6c3a_420126
aws-java-sdk-sns:1.12.586-413.v6a_6c3a_420126
aws-java-sdk-sqs:1.12.586-413.v6a_6c3a_420126
aws-java-sdk-ssm:1.12.586-413.v6a_6c3a_420126
aws-secrets-manager-credentials-provider:1.213.vca_3f37306fed
aws-secrets-manager-secret-source:1.72.v61781b_35c542
azure-credentials:293.vb_d506148f506
azure-sdk:157.v855da_0b_eb_dc2
azure-vm-agents:883.v63c930b_025dc
badge:1.9.1
basic-branch-build-strategies:81.v05e333931c7d
blueocean:1.27.9
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.9
blueocean-commons:1.27.9
blueocean-config:1.27.9
blueocean-core-js:1.27.9
blueocean-dashboard:1.27.9
blueocean-display-url:2.4.2
blueocean-events:1.27.9
blueocean-git-pipeline:1.27.9
blueocean-github-pipeline:1.27.9
blueocean-i18n:1.27.9
blueocean-jira:1.27.9
blueocean-jwt:1.27.9
blueocean-personalization:1.27.9
blueocean-pipeline-api-impl:1.27.9
blueocean-pipeline-editor:1.27.9
blueocean-pipeline-scm-api:1.27.9
blueocean-rest:1.27.9
blueocean-rest-impl:1.27.9
blueocean-web:1.27.9
bootstrap5-api:5.3.2-2
bouncycastle-api:2.29
branch-api:2.1135.v8de8e7899051
build-cause-run-condition:0.1
build-keeper-plugin:19.va_df8a_2c65123
build-monitor-plugin:1.14-745.ve2023a_305f40
build-name-setter:2.4.0
build-timeout:1.31
build-user-vars-plugin:1.9
buildresult-trigger:0.18
built-on-column:1.4
caffeine-api:3.1.8-133.v17b_1ff2e0599
categorized-view:1.13
checks-api:2.0.2
chucknorris:159.vdfe649cb_9c37
claim:516.v36293563731d
cloud-stats:320.v96b_65297a_4b_b_
cloudbees-bitbucket-branch-source:856.v04c46c86f911
cloudbees-folder:6.858.v898218f3609d
clover:4.14.1.577.vb_ff5a_f69b_37b_
cobertura:1.17
code-coverage-api:4.99.0
command-launcher:107.v773860566e2e
commons-httpclient3-api:3.1-3
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.11.0-94.v3e1f4a_926e49
computer-queue-plugin:1.7
conditional-buildstep:1.4.3
config-file-provider:959.vcff671a_4518b_
configuration-as-code:1746.vf1673cfe690a
configuration-as-code-groovy:1.1
copy-project-link:106.veb_028794a_844
copyartifact:722.v0662a_9b_e22a_c
coverage:1.5.0
cppcheck:1.26
cppncss:1.2
credentials:1309.v8835d63eb_d8a_
credentials-binding:642.v737c34dea_6c2
cucumber-reports:5.8.0
custom-tools-plugin:0.8
cvs:2.19.1
dashboard-view:2.495.v07e81500c3f2
data-tables-api:1.13.8-1
datetime-constraint:0.1.2
delivery-pipeline-plugin:1.4.2
depgraph-view:1.0.5
description-setter:1.10-CUSTOM
disable-github-multibranch-status:1.2
display-url-api:2.200.vb_9327d658781
docker-commons:439.va_3cb_0a_6a_fb_29
docker-workflow:572.v950f58993843
downstream-ext:73.vdda_16e6eb_0da
dropdown-viewstabbar-plugin:1.7
dtkit-api:3.0.2
durable-task:523.va_a_22cf15d5e0
dynamic-axis:1.0.3
ec2:1648.vf3d852e00486
echarts-api:5.4.3-1
ecutest:2.42
email-ext:2.102
embeddable-build-status:412.v09da_db_1dee68
envinject:2.908.v66a_774b_31d93
envinject-api:1.199.v3ce31253ed13
environment-variable-page-decoration:1.3.0
excludeMatrixParent:1.1
extended-choice-parameter:376.v2e02857547b_a_
extended-read-permission:53.v6499940139e5
extensible-choice-parameter:1.8.1
external-monitor-job:215.v2e88e894db_f8
external-workspace-manager:1.3.1
ez-templates:1.3.5
favorite:2.4.3
file-parameters:316.va_83a_1221db_a_7
fitnesse:1.36
flexible-publish:0.16.1
font-awesome-api:6.4.2-1
forensics-api:2.3.0
git:5.2.1
git-client:4.5.0
git-forensics:2.0.0
git-parameter:0.9.19
git-server:114.v068a_c7cc2574
github:1.37.3.1
github-api:1.318-461.v7a_c09c9fa_d63
github-autostatus:3.6.2
github-branch-source:1752.vc201a_0235d80
github-issues:1.2.4
github-oauth:597.ve0c3480fcb_d0
github-scm-trait-commit-skip:0.4.0
global-variable-string-parameter:1.2
google-oauth-plugin:1.319.v5cb_58b_91b_389
gradle:2.9
groovy:457.v99900cb_85593
groovy-postbuild:2.5
h2-api:11.1.4.199-12.v9f4244395f7a_
handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953
htmlpublisher:1.32
hubot-steps:95.va_30176518a_5a
instance-identity:185.v303dc7c645f9
ionicons-api:56.v1b_1c8c49374e
ivy:2.5
jackson2-api:2.15.3-372.v309620682326
jacoco:3.3.5
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javadoc:243.vb_b_503b_b_45537
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jenkins-design-language:1.27.9
jenkins-flowdock-plugin:1.1.8
jenkins-multijob-plugin:623.v03401733c9a_9
jersey2-api:2.41-133.va_03323b_a_1396
jfrog:1.5.0
jira:3.12
jjwt-api:0.11.5-77.v646c772fddb_0
jnr-posix-api:3.1.18-1
job-dsl:1.87
job-restrictions:0.8
jobConfigHistory:1229.v3039470161a_d
jquery:1.12.4-1
jquery3-api:3.7.1-1
jsch:0.2.8-65.v052c39de79b_2
junit:1240.vf9529b_881428
junit-attachments:205.vc0677977deb_0
keepSlaveOffline:1.0
keyboard-shortcuts-plugin:1.4
label-verifier:1.4
ldap:711.vb_d1a_491714dc
locale:314.v22ce953dfe9e
lockable-resources:1218.va_3dd45e2b_fa_7
log-parser:2.3.1
logstash:2.5.0218.v0a_ff8fefc12b_
mailer:463.vedf8358e006b_
mapdb-api:1.0.9-28.vf251ce40855d
mask-passwords:173.v6a_077a_291eb_5
matrix-auth:3.2.1
matrix-combinations-parameter:1.3.3
matrix-project:822.824.v14451b_c0fd42
maven-plugin:3.23
measurement-plots:0.1
mercurial:1260.vdfb_723cdcc81
metrics:4.2.18-442.v02e107157925
mina-sshd-api-common:2.11.0-86.v836f585d47fa_
mina-sshd-api-core:2.11.0-86.v836f585d47fa_
monitoring:1.95.0
multi-branch-priority-sorter:1.0
multibranch-action-triggers:1.8.6
multibranch-build-strategy-extension:1.1.0
nested-view:1.33
next-build-number:1.8
node-iterator-api:55.v3b_77d4032326
nodelabelparameter:1.12.0
oauth-credentials:0.646.v02b_66dc03d2e
okhttp-api:4.11.0-157.v6852a_a_fa_ec11
pam-auth:1.10
parameterized-scheduler:255.v73827fcdf618
parameterized-trigger:787.v665fcf2a_830b_
pipeline-aggregator-view:1.15
pipeline-build-step:516.v8ee60a_81c5b_9
pipeline-github:2.8-155.8eab375ac9f8
pipeline-github-lib:42.v0739460cda_c4
pipeline-githubnotify-step:49.vf37bf92d2bc8
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:689.veec561a_dee13
pipeline-input-step:477.v339683a_8d55e
pipeline-maven:1362.vee39a_d4b_02b_1
pipeline-maven-api:1362.vee39a_d4b_02b_1
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2151.ve32c9d209a_3f
pipeline-model-definition:2.2151.ve32c9d209a_3f
pipeline-model-extensions:2.2151.ve32c9d209a_3f
pipeline-multibranch-defaults:2.1
pipeline-rest-api:2.34
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2151.ve32c9d209a_3f
pipeline-stage-view:2.34
pipeline-timeline:1.0.3
pipeline-utility-steps:2.16.0
plain-credentials:143.v1b_df8b_d3b_e48
plot:2.1.12
plugin-usage-plugin:4.2
plugin-util-api:3.6.0
pollscm:1.5
postbuild-task:1.9
postbuildscript:3.2.0-550.v88192b_d3e922
preSCMbuildstep:71.v1f2990a_37e27
prism-api:1.29.0-9
progress-bar-column-plugin:11.vdef198c2d6c1
project-build-times:1.2.1
project-description-setter:1.2
project-health-report:1.2
promoted-builds:936.va_571a_a_b_f8da_5
publish-over:0.22
publish-over-cifs:0.16
pubsub-light:1.18
python:1.3
random-string-parameter:1.0
read-only-configurations:1.10
rebuild:330.v645b_7df10e2a_
release:2.18
remote-file:1.24
resource-disposer:0.23
rich-text-publisher-plugin:1.5
role-strategy:689.v731678c3e0eb_
run-condition:1.7
saferestart:0.7
saml:4.429.v9a_781a_61f1da_
scm-api:683.vb_16722fb_b_80b_
script-security:1294.v99333c047434
sectioned-view:1.25
secure-requester-whitelist:67.vca_a_d9205723f
seed:2.1.4
shelve-project-plugin:3.2
show-build-parameters:1.0
sidebar-link:2.4.1
simple-build-for-pipeline:0.2
simple-theme-plugin:176.v39740c03a_a_f5
slave-setup:1.16
snakeyaml-api:2.2-111.vc6598e30cc65
sse-gateway:1.26
ssh-agent:346.vda_a_c4f2c8e50
ssh-credentials:308.ve4497b_ccd8f4
ssh-slaves:2.916.vd17b_43357ce4
sshd:3.312.v1c601b_c83b_0e
statusmonitor:1.3
structs:325.vcb_307d2a_2782
subversion:2.17.3
summary_report:1.15
test-stability:2.3
throttle-concurrents:2.14
timestamper:1.26
token-macro:384.vf35b_f26814ec
trilead-api:2.84.v72119de229b_7
urltrigger:1.02
validating-string-parameter:183.v3748e79b_9737
variant:60.v7290fc0eb_b_cd
versioncolumn:217.v16457a_b_eb_7eb_
view-job-filters:369.ve0513a_a_f5524
warnings-ng:10.5.1
webhook-step:327.vb_58f208a_7783
workflow-aggregator:596.v8c21c963d92d
workflow-api:1283.v99c10937efcb_
workflow-basic-steps:1042.ve7b_140c4a_e0c
workflow-cps:3812.vc3031a_b_a_c955
workflow-durable-task-step:1289.v4d3e7b_01546b_
workflow-job:1360.vc6700e3136f5
workflow-multibranch:756.v891d88f2cd46
workflow-remote-loader:1.6
workflow-scm-step:415.v434365564324
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:865.v43e78cc44e0d
ws-cleanup:0.45
xtrigger-api:1.0
xunit:3.1.3
}}}}
https://plugins.jenkins.io/generic-event/ seems to be present in all stack traces posted here as well as in JENKINS-73715. In particular, it looks like GenericEventItemListener is trying to do too much synchronously from various item listener methods, which are typically called while a monitor is held for the item in question. Even more specifically, the problem seems largely related to the use of ExportedBeanProcessor in generic-event, which is computing (among other things) health checks for the items being processed, which for multibranch projects causes some of their builds to be loaded, which requires various locks, and all of this is happening while the original item is locked. Complex listeners should perform their logic on a background thread, for example by submitting tasks to jenkins.util.Timer.
The locking in MultiBranchProject.getProjectFactory could perhaps be improved as well to fix some cases of this issue, see JENKINS-74777.
We have 80+ Jenkins controllers and we are seeing 1-2 controllers deadlock this issue per week. This started when we upgraded to 2.246.1 + latest plugin versions at the time of the upgrade around December 2023.