-
Bug
-
Resolution: Unresolved
-
Critical
-
Jenkins 2.401.1; Jenkins 2.401.3; Jenkins 2.414.1, 2.414.3
JVM 11.0.19+7; 11.0.20+8 (server and agent)
Windows Server 2019
plugins:
Active Directory plugin (active-directory): 2.33
Allure Jenkins Plugin (allure-jenkins-plugin): 2.30.3
OWASP Markup Formatter Plugin (antisamy-markup-formatter): 162.v0e6ec0fcfcf6
Apache HttpComponents Client 4.x API Plugin (apache-httpcomponents-client-4-api): 4.5.14-208.v438351942757
Authorize Project (authorize-project): 1.7.1
Bootstrap 5 API Plugin (bootstrap5-api): 5.3.0-1
bouncycastle API Plugin (bouncycastle-api): 2.29
Branch API Plugin (branch-api): 2.1122.v09cb_8ea_8a_724
Caffeine API Plugin (caffeine-api): 3.1.8-133.v17b_1ff2e0599
Checks API plugin (checks-api): 2.0.0
Folders Plugin (cloudbees-folder): 6.848.ve3b_fd7839a_81
Cobertura Plugin (cobertura): 1.17
Code Coverage API Plugin (code-coverage-api): 4.7.0
Command Agent Launcher Plugin (command-launcher): 107.v773860566e2e
Commons HttpClient 3.x API (commons-httpclient3-api): 3.1-3
commons-lang3 v3.x Jenkins API Plugin (commons-lang3-api): 3.13.0-62.v7d18e55f51e2
commons-text API Plugin (commons-text-api): 1.10.0-68.v0d0b_c439292b_
Compress Artifacts Plugin (compress-artifacts): 98.vb_20f3c77ddf7
Copy Artifact Plugin (copyartifact): 722.v0662a_9b_e22a_c
Credentials Binding Plugin (credentials-binding): 631.v861c06d062b_4
Credentials Plugin (credentials): 1271.v54b_1c2c6388a_
DataTables.net API Plugin (data-tables-api): 1.13.5-1
Display URL API (display-url-api): 2.3.9
Durable Task Plugin (durable-task): 523.va_a_22cf15d5e0
ECharts API Plugin (echarts-api): 5.4.0-5
Email Extension Plugin (email-ext): 2.100
Font Awesome API Plugin (font-awesome-api): 6.4.0-2
Forensics API Plugin (forensics-api): 2.3.0
Git client plugin (git-client): 4.4.0
Git plugin (git): 5.2.0
Instance Identity (instance-identity): 173.va_37c494ec4e5
Ionicons API (ionicons-api): 56.v1b_1c8c49374e
Jackson 2 API Plugin (jackson2-api): 2.15.2-350.v0c2f3f8fc595
Jakarta Activation API (jakarta-activation-api): 2.0.1-3
Jakarta Mail API (jakarta-mail-api): 2.0.1-3
JavaBeans Activation Framework (JAF) API (javax-activation-api): 1.2.0-6
JavaMail API (javax-mail-api): 1.6.2-9
JAXB plugin (jaxb): 2.3.8-1
Oracle Java SE Development Kit Installer Plugin (jdk-tool): 73.vddf737284550
JQuery3 API Plugin (jquery3-api): 3.7.0-1
JSch dependency plugin (jsch): 0.2.8-65.v052c39de79b_2
JUnit Plugin (junit): 1217.v4297208a_a_b_ce
Mailer Plugin (mailer): 463.vedf8358e006b_
MapDB API Plugin (mapdb-api): 1.0.9-28.vf251ce40855d
Mask Passwords Plugin (mask-passwords): 150.vf80d33113e80
Matrix Project Plugin (matrix-project): 808.v5a_b_5f56d6966
Mina SSHD API :: Common (mina-sshd-api-common): 2.10.0-69.v28e3e36d18eb_
Mina SSHD API :: Core (mina-sshd-api-core): 2.10.0-69.v28e3e36d18eb_
NUnit plugin (nunit): 394.v65c09a_8408b_a_
Pipeline: Build Step (pipeline-build-step): 505.v5f0844d8d126
Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 202.va_d268e64deb_3
Pipeline: Groovy Libraries (pipeline-groovy-lib): 671.v07c339c842e8
Pipeline: Input Step (pipeline-input-step): 477.v339683a_8d55e
Pipeline: Milestone Step (pipeline-milestone-step): 111.v449306f708b_7
Pipeline: Model API (pipeline-model-api): 2.2144.v077a_d1928a_40
Pipeline: Declarative (pipeline-model-definition): 2.2144.v077a_d1928a_40
Pipeline: Declarative Extension Points API (pipeline-model-extensions): 2.2144.v077a_d1928a_40
Pipeline: REST API Plugin (pipeline-rest-api): 2.33
Pipeline: Stage Step (pipeline-stage-step): 305.ve96d0205c1c6
Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata): 2.2144.v077a_d1928a_40
Pipeline: Stage View Plugin (pipeline-stage-view): 2.33
Plain Credentials Plugin (plain-credentials): 143.v1b_df8b_d3b_e48
Plugin Utilities API Plugin (plugin-util-api): 3.3.0
Prism API Plugin (prism-api): 1.29.0-7
Resource Disposer Plugin (resource-disposer): 0.23
Role-based Authorization Strategy (role-strategy): 689.v731678c3e0eb_
SCM API Plugin (scm-api): 676.v886669a_199a_a_
Script Security Plugin (script-security): 1273.v66c1964f0dfd
Simple Theme Plugin (simple-theme-plugin): 160.vb_76454b_67900
SnakeYAML API Plugin (snakeyaml-api): 1.33-95.va_b_a_e3e47b_fa_4
SSH Credentials Plugin (ssh-credentials): 308.ve4497b_ccd8f4
SSH server (sshd): 3.312.v1c601b_c83b_0e
Structs Plugin (structs): 325.vcb_307d2a_2782
Subversion Plug-in (subversion): 2.17.3
ThinBackup (thinBackup): 1.18
Token Macro Plugin (token-macro): 384.vf35b_f26814ec
Trilead API Plugin (trilead-api): 2.84.v72119de229b_7
Variant Plugin (variant): 59.vf075fe829ccb
Versions Node Monitors plugin (versioncolumn): 162.v85841b_0790d5
Pipeline (workflow-aggregator): 596.v8c21c963d92d
Pipeline: API (workflow-api): 1267.vd9b_a_ddd9eb_47
Pipeline: Basic Steps (workflow-basic-steps): 1042.ve7b_140c4a_e0c
Pipeline: Groovy (workflow-cps): 3773.v505e0052522c
Pipeline: Nodes and Processes (workflow-durable-task-step): 1289.v4d3e7b_01546b_
Pipeline: Job (workflow-job): 1326.ve643e00e9220
Pipeline: Multibranch (workflow-multibranch): 756.v891d88f2cd46
Pipeline: SCM Step (workflow-scm-step): 415.v434365564324
Pipeline: Step API (workflow-step-api): 639.v6eca_cd8c04a_a_
Pipeline: Supporting APIs (workflow-support): 848.v5a_383b_d14921
Workspace Cleanup Plugin (ws-cleanup): 0.45Jenkins 2.401.1; Jenkins 2.401.3; Jenkins 2.414.1, 2.414.3 JVM 11.0.19+7; 11.0.20+8 (server and agent) Windows Server 2019 plugins: Active Directory plugin (active-directory): 2.33 Allure Jenkins Plugin (allure-jenkins-plugin): 2.30.3 OWASP Markup Formatter Plugin (antisamy-markup-formatter): 162.v0e6ec0fcfcf6 Apache HttpComponents Client 4.x API Plugin (apache-httpcomponents-client-4-api): 4.5.14-208.v438351942757 Authorize Project (authorize-project): 1.7.1 Bootstrap 5 API Plugin (bootstrap5-api): 5.3.0-1 bouncycastle API Plugin (bouncycastle-api): 2.29 Branch API Plugin (branch-api): 2.1122.v09cb_8ea_8a_724 Caffeine API Plugin (caffeine-api): 3.1.8-133.v17b_1ff2e0599 Checks API plugin (checks-api): 2.0.0 Folders Plugin (cloudbees-folder): 6.848.ve3b_fd7839a_81 Cobertura Plugin (cobertura): 1.17 Code Coverage API Plugin (code-coverage-api): 4.7.0 Command Agent Launcher Plugin (command-launcher): 107.v773860566e2e Commons HttpClient 3.x API (commons-httpclient3-api): 3.1-3 commons-lang3 v3.x Jenkins API Plugin (commons-lang3-api): 3.13.0-62.v7d18e55f51e2 commons-text API Plugin (commons-text-api): 1.10.0-68.v0d0b_c439292b_ Compress Artifacts Plugin (compress-artifacts): 98.vb_20f3c77ddf7 Copy Artifact Plugin (copyartifact): 722.v0662a_9b_e22a_c Credentials Binding Plugin (credentials-binding): 631.v861c06d062b_4 Credentials Plugin (credentials): 1271.v54b_1c2c6388a_ DataTables.net API Plugin (data-tables-api): 1.13.5-1 Display URL API (display-url-api): 2.3.9 Durable Task Plugin (durable-task): 523.va_a_22cf15d5e0 ECharts API Plugin (echarts-api): 5.4.0-5 Email Extension Plugin (email-ext): 2.100 Font Awesome API Plugin (font-awesome-api): 6.4.0-2 Forensics API Plugin (forensics-api): 2.3.0 Git client plugin (git-client): 4.4.0 Git plugin (git): 5.2.0 Instance Identity (instance-identity): 173.va_37c494ec4e5 Ionicons API (ionicons-api): 56.v1b_1c8c49374e Jackson 2 API Plugin (jackson2-api): 2.15.2-350.v0c2f3f8fc595 Jakarta Activation API (jakarta-activation-api): 2.0.1-3 Jakarta Mail API (jakarta-mail-api): 2.0.1-3 JavaBeans Activation Framework (JAF) API (javax-activation-api): 1.2.0-6 JavaMail API (javax-mail-api): 1.6.2-9 JAXB plugin (jaxb): 2.3.8-1 Oracle Java SE Development Kit Installer Plugin (jdk-tool): 73.vddf737284550 JQuery3 API Plugin (jquery3-api): 3.7.0-1 JSch dependency plugin (jsch): 0.2.8-65.v052c39de79b_2 JUnit Plugin (junit): 1217.v4297208a_a_b_ce Mailer Plugin (mailer): 463.vedf8358e006b_ MapDB API Plugin (mapdb-api): 1.0.9-28.vf251ce40855d Mask Passwords Plugin (mask-passwords): 150.vf80d33113e80 Matrix Project Plugin (matrix-project): 808.v5a_b_5f56d6966 Mina SSHD API :: Common (mina-sshd-api-common): 2.10.0-69.v28e3e36d18eb_ Mina SSHD API :: Core (mina-sshd-api-core): 2.10.0-69.v28e3e36d18eb_ NUnit plugin (nunit): 394.v65c09a_8408b_a_ Pipeline: Build Step (pipeline-build-step): 505.v5f0844d8d126 Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 202.va_d268e64deb_3 Pipeline: Groovy Libraries (pipeline-groovy-lib): 671.v07c339c842e8 Pipeline: Input Step (pipeline-input-step): 477.v339683a_8d55e Pipeline: Milestone Step (pipeline-milestone-step): 111.v449306f708b_7 Pipeline: Model API (pipeline-model-api): 2.2144.v077a_d1928a_40 Pipeline: Declarative (pipeline-model-definition): 2.2144.v077a_d1928a_40 Pipeline: Declarative Extension Points API (pipeline-model-extensions): 2.2144.v077a_d1928a_40 Pipeline: REST API Plugin (pipeline-rest-api): 2.33 Pipeline: Stage Step (pipeline-stage-step): 305.ve96d0205c1c6 Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata): 2.2144.v077a_d1928a_40 Pipeline: Stage View Plugin (pipeline-stage-view): 2.33 Plain Credentials Plugin (plain-credentials): 143.v1b_df8b_d3b_e48 Plugin Utilities API Plugin (plugin-util-api): 3.3.0 Prism API Plugin (prism-api): 1.29.0-7 Resource Disposer Plugin (resource-disposer): 0.23 Role-based Authorization Strategy (role-strategy): 689.v731678c3e0eb_ SCM API Plugin (scm-api): 676.v886669a_199a_a_ Script Security Plugin (script-security): 1273.v66c1964f0dfd Simple Theme Plugin (simple-theme-plugin): 160.vb_76454b_67900 SnakeYAML API Plugin (snakeyaml-api): 1.33-95.va_b_a_e3e47b_fa_4 SSH Credentials Plugin (ssh-credentials): 308.ve4497b_ccd8f4 SSH server (sshd): 3.312.v1c601b_c83b_0e Structs Plugin (structs): 325.vcb_307d2a_2782 Subversion Plug-in (subversion): 2.17.3 ThinBackup (thinBackup): 1.18 Token Macro Plugin (token-macro): 384.vf35b_f26814ec Trilead API Plugin (trilead-api): 2.84.v72119de229b_7 Variant Plugin (variant): 59.vf075fe829ccb Versions Node Monitors plugin (versioncolumn): 162.v85841b_0790d5 Pipeline (workflow-aggregator): 596.v8c21c963d92d Pipeline: API (workflow-api): 1267.vd9b_a_ddd9eb_47 Pipeline: Basic Steps (workflow-basic-steps): 1042.ve7b_140c4a_e0c Pipeline: Groovy (workflow-cps): 3773.v505e0052522c Pipeline: Nodes and Processes (workflow-durable-task-step): 1289.v4d3e7b_01546b_ Pipeline: Job (workflow-job): 1326.ve643e00e9220 Pipeline: Multibranch (workflow-multibranch): 756.v891d88f2cd46 Pipeline: SCM Step (workflow-scm-step): 415.v434365564324 Pipeline: Step API (workflow-step-api): 639.v6eca_cd8c04a_a_ Pipeline: Supporting APIs (workflow-support): 848.v5a_383b_d14921 Workspace Cleanup Plugin (ws-cleanup): 0.45
Update 2023-10-27 for LTS v2.414.3
with Jenkins v2.414.3 this bug is now so massive:
after each restart service goes into this corrupt state just in a few seconds
-> Jenkins no longer usable for us
=> after downgrade back to 2.414.2 it is usable again
=> but still worse
hint: we use extensively the Jenkins API to update labels and config of nodes (>150 nodes in parallel) to balance jobs, I think we found here a race condition
some minutes after restart Jenkins Server service some pipeline job stuck in state "Still waiting to schedule task" and whole queue deadlock forever
pipeline example:
pipeline {
agent { node { label params.nodename } }
options { buildDiscarder(logRotator(numToKeepStr: '5000', artifactNumToKeepStr: '5000')) }
parameters
{
string (name: 'nodename', defaultValue: '')
string (name: 'commandLine', defaultValue: '')
string (name: 'sourceBranch', defaultValue: 'master', description: '')
}
stages {
stage('Execute') {
when {
beforeAgent true
expression { params.nodename != '' }
}
steps {
bat 'echo hello'
}
}
}
}
parameter example:
nodename: VM-514A56DC0979
node is connected and idling
finally the whole jenkins queue deadlocks
no progress until I restart the Jenkins Server service
but after service restart and some minutes it stuck again in that state
have also an observation what can be related to this:
if I want to change configuration of that node it lead to an serious error in Jenkins
see
pipeline:
pipeline job console log:
parameter values:
node state:
queue deadlock:
thread dump:
Thread #0
at DSL.node(java.util.concurrent.ExecutionException: java.lang.ClassCastException)
at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.LabelScript.run(LabelScript.groovy:42)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inDeclarativeAgent(ModelInterpreter.groovy:594)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.call(ModelInterpreter.groovy:76)
at WorkflowScript.run(WorkflowScript:1)
plugin version:
[JENKINS-71388] pipeline job stuck with "Still waiting to schedule task" but node is idling and then whole queue stay in deadlock state
thread dumps:
VM-514A56DC0979 Common-Cleaner "Common-Cleaner" Id=10 Group=InnocuousThreadGroup TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@22894378 at java.base@11.0.19/java.lang.Object.wait(Native Method) - waiting on java.lang.ref.ReferenceQueue$Lock@22894378 at java.base@11.0.19/java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.base@11.0.19/jdk.internal.ref.CleanerImpl.run(Unknown Source) at java.base@11.0.19/java.lang.Thread.run(Unknown Source) at java.base@11.0.19/jdk.internal.misc.InnocuousThread.run(Unknown Source) IOHub#1: Selector[keys:0, gen:0] / pool-1-thread-1 "IOHub#1: Selector[keys:0, gen:0] / pool-1-thread-1" Id=14 Group=main RUNNABLE (in native) at java.base@11.0.19/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at java.base@11.0.19/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source) at java.base@11.0.19/sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source) at java.base@11.0.19/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source) - locked sun.nio.ch.Util$2@75a901a8 - locked sun.nio.ch.WindowsSelectorImpl@6b1c81f at java.base@11.0.19/sun.nio.ch.SelectorImpl.select(Unknown Source) at app//org.jenkinsci.remoting.protocol.IOHub.run(IOHub.java:460) at java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at app//hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125) at app//hudson.remoting.Engine$1$$Lambda$56/0x0000000100133c40.run(Unknown Source) at java.base@11.0.19/java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@2011559c JNA Cleaner "JNA Cleaner" Id=29 Group=main WAITING on java.lang.ref.ReferenceQueue$Lock@6744fd30 at java.base@11.0.19/java.lang.Object.wait(Native Method) - waiting on java.lang.ref.ReferenceQueue$Lock@6744fd30 at java.base@11.0.19/java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.base@11.0.19/java.lang.ref.ReferenceQueue.remove(Unknown Source) at com.sun.jna.internal.Cleaner$1.run(Cleaner.java:58) main "main" Id=1 Group=main WAITING on hudson.remoting.Engine@5eda1778 at java.base@11.0.19/java.lang.Object.wait(Native Method) - waiting on hudson.remoting.Engine@5eda1778 at java.base@11.0.19/java.lang.Thread.join(Unknown Source) at java.base@11.0.19/java.lang.Thread.join(Unknown Source) at app//hudson.remoting.jnlp.Main.main(Main.java:278) at app//hudson.remoting.jnlp.Main._main(Main.java:271) at app//hudson.remoting.Launcher.run(Launcher.java:386) at app//hudson.remoting.Launcher.main(Launcher.java:297) Ping thread for channel hudson.remoting.Channel@125d7887:JNLP4-connect connection to <<replaced>> "Ping thread for channel hudson.remoting.Channel@125d7887:JNLP4-connect connection to <<replaced>>" Id=25 Group=main TIMED_WAITING at java.base@11.0.19/java.lang.Thread.sleep(Native Method) at java.base@11.0.19/java.lang.Thread.sleep(Unknown Source) at java.base@11.0.19/java.util.concurrent.TimeUnit.sleep(Unknown Source) at app//hudson.remoting.PingThread.run(PingThread.java:93) pool-1-thread-3 "pool-1-thread-3" Id=16 Group=main RUNNABLE (in native) at java.base@11.0.19/sun.nio.ch.SocketDispatcher.read0(Native Method) at java.base@11.0.19/sun.nio.ch.SocketDispatcher.read(Unknown Source) at java.base@11.0.19/sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source) at java.base@11.0.19/sun.nio.ch.IOUtil.read(Unknown Source) at java.base@11.0.19/sun.nio.ch.IOUtil.read(Unknown Source) at java.base@11.0.19/sun.nio.ch.SocketChannelImpl.read(Unknown Source) at app//hudson.remoting.SocketChannelStream$1.read(SocketChannelStream.java:36) at java.base@11.0.19/sun.nio.ch.ChannelInputStream.read(Unknown Source) at java.base@11.0.19/sun.nio.ch.ChannelInputStream.read(Unknown Source) at java.base@11.0.19/sun.nio.ch.ChannelInputStream.read(Unknown Source) - locked sun.nio.ch.ChannelInputStream@53d22102 at java.base@11.0.19/java.nio.channels.Channels$ReadableByteChannelImpl.read(Unknown Source) - locked java.lang.Object@1d9799f3 at app//org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:255) at java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at app//hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125) at app//hudson.remoting.Engine$1$$Lambda$56/0x0000000100133c40.run(Unknown Source) at java.base@11.0.19/java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 2 - java.util.concurrent.locks.ReentrantLock$NonfairSync@15ee4b6e - java.util.concurrent.ThreadPoolExecutor$Worker@6cc4cb38 pool-1-thread-41 for JNLP4-connect connection to <<replaced>> id=171205 "pool-1-thread-41 for JNLP4-connect connection to <<replaced>> id=171205" Id=57 Group=main RUNNABLE at java.management@11.0.19/sun.management.ThreadImpl.dumpThreads0(Native Method) at java.management@11.0.19/sun.management.ThreadImpl.dumpAllThreads(Unknown Source) at java.management@11.0.19/sun.management.ThreadImpl.dumpAllThreads(Unknown Source) at hudson.Functions.getThreadInfos(Functions.java:1482) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:101) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:97) at app//hudson.remoting.UserRequest.perform(UserRequest.java:211) at app//hudson.remoting.UserRequest.perform(UserRequest.java:54) at app//hudson.remoting.Request$2.run(Request.java:377) at app//hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78) at app//hudson.remoting.InterceptingExecutorService$$Lambda$102/0x000000010021e840.call(Unknown Source) at java.base@11.0.19/java.util.concurrent.FutureTask.run(Unknown Source) at java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at app//hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125) at app//hudson.remoting.Engine$1$$Lambda$56/0x0000000100133c40.run(Unknown Source) at java.base@11.0.19/java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@196abe6 pool-1-thread-42 "pool-1-thread-42" Id=58 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@463da82a at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@463da82a at java.base@11.0.19/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.base@11.0.19/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.base@11.0.19/java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.base@11.0.19/java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at app//hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125) at app//hudson.remoting.Engine$1$$Lambda$56/0x0000000100133c40.run(Unknown Source) at java.base@11.0.19/java.lang.Thread.run(Unknown Source) RemoteInvocationHandler [#1] "RemoteInvocationHandler [#1]" Id=18 Group=main TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@255c02a8 at java.base@11.0.19/java.lang.Object.wait(Native Method) - waiting on java.lang.ref.ReferenceQueue$Lock@255c02a8 at java.base@11.0.19/java.lang.ref.ReferenceQueue.remove(Unknown Source) at app//hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:602) at java.base@11.0.19/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base@11.0.19/java.util.concurrent.FutureTask.run(Unknown Source) at app//hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:121) at java.base@11.0.19/java.lang.Thread.run(Unknown Source) Thread-0 "Thread-0" Id=13 Group=main TIMED_WAITING on hudson.remoting.Channel@125d7887 at java.base@11.0.19/java.lang.Object.wait(Native Method) - waiting on hudson.remoting.Channel@125d7887 at app//hudson.remoting.Channel.join(Channel.java:1182) at app//hudson.remoting.Engine.innerRun(Engine.java:844) at app//hudson.remoting.Engine.run(Engine.java:543) Windows IOHub Watcher for IOHub#1: Selector[keys:0, gen:0] / pool-1-thread-2 "Windows IOHub Watcher for IOHub#1: Selector[keys:0, gen:0] / pool-1-thread-2" Id=15 Group=main TIMED_WAITING on java.lang.Object@6ecbd932 at java.base@11.0.19/java.lang.Object.wait(Native Method) - waiting on java.lang.Object@6ecbd932 at app//org.jenkinsci.remoting.protocol.IOHub$IOHubSelectorWatcher.run(IOHub.java:548) at java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at app//hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125) at app//hudson.remoting.Engine$1$$Lambda$56/0x0000000100133c40.run(Unknown Source) at java.base@11.0.19/java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@13ea5b40 Attach Listener "Attach Listener" Id=5 Group=system RUNNABLE Finalizer "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@226bfc21 at java.base@11.0.19/java.lang.Object.wait(Native Method) - waiting on java.lang.ref.ReferenceQueue$Lock@226bfc21 at java.base@11.0.19/java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.base@11.0.19/java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.base@11.0.19/java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) Reference Handler "Reference Handler" Id=2 Group=system RUNNABLE at java.base@11.0.19/java.lang.ref.Reference.waitForReferencePendingList(Native Method) at java.base@11.0.19/java.lang.ref.Reference.processPendingReferences(Unknown Source) at java.base@11.0.19/java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) Signal Dispatcher "Signal Dispatcher" Id=4 Group=system RUNNABLE
here is an additional thread dump of an current blocking job:
Thread #2
at DSL.node(java.util.concurrent.ExecutionException: java.lang.ClassCastException)
at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.LabelScript.run(LabelScript.groovy:42)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inDeclarativeAgent(ModelInterpreter.groovy:594)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(ModelInterpreter.groovy:272)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.stageInput(ModelInterpreter.groovy:356)
Vishnu
added a comment - This is a +1 from me as well, lots of jobs are affected 
from jenkins log: