-
Bug
-
Resolution: Unresolved
-
Blocker
-
HideJenkins core 2.107.3
workflow-aggregator: 2.5
workflow-api: 2.27
workflow-basic-steps: 2.7
workflow-cps: 2.53
workflow-cps-global-lib: 2.9
workflow-durable-task-step: 2.19
workflow-job: 2.21
workflow-multibranch: 2.19
workflow-scm-step: 2.6
workflow-step-api: 2.15
workflow-support: 2.18
pipeline-build-step: 2.7
pipeline-graph-analysis: 1.6
pipeline-input-step: 2.8
pipeline-milestone-step: 1.3.1
pipeline-model-api: 1.2.9
pipeline-model-declarative-agent: 1.1.1
pipeline-model-definition: 1.2.9
pipeline-model-extensions: 1.2.9
pipeline-rest-api: 2.10
pipeline-stage-step: 2.3
pipeline-stage-tags-metadata: 1.2.9
pipeline-stage-view: 2.10
pipeline-utility-steps: 2.1.0ShowJenkins core 2.107.3 workflow-aggregator: 2.5 workflow-api: 2.27 workflow-basic-steps: 2.7 workflow-cps: 2.53 workflow-cps-global-lib: 2.9 workflow-durable-task-step: 2.19 workflow-job: 2.21 workflow-multibranch: 2.19 workflow-scm-step: 2.6 workflow-step-api: 2.15 workflow-support: 2.18 pipeline-build-step: 2.7 pipeline-graph-analysis: 1.6 pipeline-input-step: 2.8 pipeline-milestone-step: 1.3.1 pipeline-model-api: 1.2.9 pipeline-model-declarative-agent: 1.1.1 pipeline-model-definition: 1.2.9 pipeline-model-extensions: 1.2.9 pipeline-rest-api: 2.10 pipeline-stage-step: 2.3 pipeline-stage-tags-metadata: 1.2.9 pipeline-stage-view: 2.10 pipeline-utility-steps: 2.1.0
-
Powered by SuggestiMate
Three times in the last two weeks, we've had our Jenkins server stop responding to requests. When I check syslog, I see errors like this:
Jun 30 16:07:18 jenkins [jenkins]: Jun 30, 2018 4:07:18 PM org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0
Jun 30 16:07:18 jenkins [jenkins]: INFO: Running CpsFlowExecutionOwner[project/263:project #263] unresponsive for 5 sec
Jun 30 16:07:18 jenkins [jenkins]: Jun 30, 2018 4:07:18 PM org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0
Jun 30 16:07:18 jenkins [jenkins]: INFO: Running CpsFlowExecutionOwner[project/368:project #368] unresponsive for 5 sec
Jun 30 16:07:18 jenkins [jenkins]: Jun 30, 2018 4:07:18 PM org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0
Jun 30 16:07:18 jenkins [jenkins]: INFO: Running CpsFlowExecutionOwner[project/318:project #318] unresponsive for 5 sec
These seem to persist indefinitely and there don't seem to be any other relevant messages in the log. The Web UI just hangs until nginx times out.
The Java process will then refuse to stop when I try to restart the service and I have to kill it with kill -9.
[JENKINS-52362] Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Version: Jenkins 2.355 with latest plugins.
Problem:
- Some job cannot finish even if it's actually done, and keeps logging same output
- Nodes of those jobs report unresponsive for 5 sec/ 10 sec... ,and the time can reset to 5 sec(means it's not a dead lock)
- Other job on those node cannot start or finish
- jstack jenkins, jobs are waiting for a thread. And there is tee in this thread backtrace. (I forgot to snapshot/save it...)
Reason:
- We use tee in pipeline
- Maybe cmd in tee does not close fd properly, or unstable network(packet loss for EOF), or tee self has bugs
- Such huge and infinite outputs occupy the lock for too long.
- The node start unresponsive, and the unresponsive time will reset when lock is get.
- Jobs on those node cannot start or finish.
After we remove all tee in all jobs, the problem disappeared. But the reason might be different in other user's reports.
If you got tee in your backtrace, might be the tee problem, you should try to remove it.
Report by: Aliyun PolarDB Testing team.
We are having this same problem, I'm not sure if it's related to the CPS issue of the original subject or not. Several times a day, jobs will just stop, at various spots in their pipeline. These are all declarative pipelines, Linux controller, windows agent. There is plenty of disk space.
Jenkins 2.346.2
mostly all updated plugins
that -Xmx256m looks suspiciously low
This is from the jenkins.xml:
<executable>/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre/bin</executable> --> <arguments>-Xrs -Xmx256m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -Djsse.enableSNIExtension=false -jar "%BASE%\jenkins.war" --httpPort=8080</arguments>
memory:
$ free -h total used free shared buff/cache available Mem: 7.6G 3.6G 1.2G 256M 2.9G 3.5G Swap: 1.0G 345M 678M
Should we bump up the -Xmx to 4g?
mcascone , Not sure whether you have already fixed it. Try updating to openjdk 11
touseef We're experiencing the same issue. Do you mean updating the jdk version will help us fix the issue?
Which specific version helps? The latest one?
The issue is still present with Jenkins 2.401.1 running with Java Temurin 17.0.8.
Installed plugins:
com/coravy/hudson/plugins/github/github/1.37.3/github-1.37.3.hpi io/jenkins/blueocean/blueocean-bitbucket-pipeline/1.27.5/blueocean-bitbucket-pipeline-1.27.5.hpi io/jenkins/blueocean/blueocean-commons/1.27.5/blueocean-commons-1.27.5.hpi io/jenkins/blueocean/blueocean-config/1.27.5/blueocean-config-1.27.5.hpi io/jenkins/blueocean/blueocean-core-js/1.27.5/blueocean-core-js-1.27.5.hpi io/jenkins/blueocean/blueocean-dashboard/1.27.5/blueocean-dashboard-1.27.5.hpi io/jenkins/blueocean/blueocean-events/1.27.5/blueocean-events-1.27.5.hpi io/jenkins/blueocean/blueocean-git-pipeline/1.27.5/blueocean-git-pipeline-1.27.5.hpi io/jenkins/blueocean/blueocean-github-pipeline/1.27.5/blueocean-github-pipeline-1.27.5.hpi io/jenkins/blueocean/blueocean-i18n/1.27.5/blueocean-i18n-1.27.5.hpi io/jenkins/blueocean/blueocean-jwt/1.27.5/blueocean-jwt-1.27.5.hpi io/jenkins/blueocean/blueocean-personalization/1.27.5/blueocean-personalization-1.27.5.hpi io/jenkins/blueocean/blueocean-pipeline-api-impl/1.27.5/blueocean-pipeline-api-impl-1.27.5.hpi io/jenkins/blueocean/blueocean-pipeline-editor/1.27.5/blueocean-pipeline-editor-1.27.5.hpi io/jenkins/blueocean/blueocean-pipeline-scm-api/1.27.5/blueocean-pipeline-scm-api-1.27.5.hpi io/jenkins/blueocean/blueocean-rest-impl/1.27.5/blueocean-rest-impl-1.27.5.hpi io/jenkins/blueocean/blueocean-rest/1.27.5/blueocean-rest-1.27.5.hpi io/jenkins/blueocean/blueocean-web/1.27.5/blueocean-web-1.27.5.hpi io/jenkins/blueocean/blueocean/1.27.5/blueocean-1.27.5.hpi io/jenkins/blueocean/jenkins-design-language/1.27.5/jenkins-design-language-1.27.5.hpi io/jenkins/configuration-as-code/1670.v564dc8b_982d0/configuration-as-code-1670.v564dc8b_982d0.hpi io/jenkins/plugins/bootstrap5-api/5.2.2-4/bootstrap5-api-5.2.2-4.hpi io/jenkins/plugins/caffeine-api/3.1.6-115.vb_8b_b_328e59d8/caffeine-api-3.1.6-115.vb_8b_b_328e59d8.hpi io/jenkins/plugins/checks-api/1.8.1/checks-api-1.8.1.hpi io/jenkins/plugins/commons-lang3-api/3.13.0-62.v7d18e55f51e2/commons-lang3-api-3.13.0-62.v7d18e55f51e2.hpi io/jenkins/plugins/commons-text-api/1.10.0-36.vc008c8fcda_7b_/commons-text-api-1.10.0-36.vc008c8fcda_7b_.hpi io/jenkins/plugins/configuration-as-code-groovy/1.1/configuration-as-code-groovy-1.1.hpi io/jenkins/plugins/data-tables-api/1.13.5-1/data-tables-api-1.13.5-1.hpi io/jenkins/plugins/echarts-api/5.4.0-1/echarts-api-5.4.0-1.hpi io/jenkins/plugins/font-awesome-api/6.3.0-2/font-awesome-api-6.3.0-2.hpi io/jenkins/plugins/ionicons-api/56.v1b_1c8c49374e/ionicons-api-56.v1b_1c8c49374e.hpi io/jenkins/plugins/jakarta-activation-api/2.0.1-3/jakarta-activation-api-2.0.1-3.hpi io/jenkins/plugins/jakarta-mail-api/2.0.1-3/jakarta-mail-api-2.0.1-3.hpi io/jenkins/plugins/javax-activation-api/1.2.0-6/javax-activation-api-1.2.0-6.hpi io/jenkins/plugins/jaxb/2.3.8-1/jaxb-2.3.8-1.hpi io/jenkins/plugins/jjwt-api/0.11.5-77.v646c772fddb_0/jjwt-api-0.11.5-77.v646c772fddb_0.hpi io/jenkins/plugins/jquery3-api/3.7.0-1/jquery3-api-3.7.0-1.hpi io/jenkins/plugins/mina-sshd-api/mina-sshd-api-common/2.10.0-69.v28e3e36d18eb_/mina-sshd-api-common-2.10.0-69.v28e3e36d18eb_.hpi io/jenkins/plugins/mina-sshd-api/mina-sshd-api-core/2.10.0-69.v28e3e36d18eb_/mina-sshd-api-core-2.10.0-69.v28e3e36d18eb_.hpi io/jenkins/plugins/okhttp-api/4.11.0-157.v6852a_a_fa_ec11/okhttp-api-4.11.0-157.v6852a_a_fa_ec11.hpi io/jenkins/plugins/pipeline-graph-view/191.vc6da_9d3eb_70a/pipeline-graph-view-191.vc6da_9d3eb_70a.hpi io/jenkins/plugins/pipeline-groovy-lib/671.v07c339c842e8/pipeline-groovy-lib-671.v07c339c842e8.hpi io/jenkins/plugins/plugin-util-api/3.2.1/plugin-util-api-3.2.1.hpi io/jenkins/plugins/popper2-api/2.11.6-1/popper2-api-2.11.6-1.hpi io/jenkins/plugins/snakeyaml-api/1.33-95.va_b_a_e3e47b_fa_4/snakeyaml-api-1.33-95.va_b_a_e3e47b_fa_4.hpi org/6wind/jenkins/lockable-resources/1185.v0c528656ce04/lockable-resources-1185.v0c528656ce04.hpi org/jenkins-ci/modules/instance-identity/173.va_37c494ec4e5/instance-identity-173.va_37c494ec4e5.hpi org/jenkins-ci/plugins/antisamy-markup-formatter/159.v25b_c67cd35fb_/antisamy-markup-formatter-159.v25b_c67cd35fb_.hpi org/jenkins-ci/plugins/apache-httpcomponents-client-4-api/4.5.14-150.v7a_b_9d17134a_5/apache-httpcomponents-client-4-api-4.5.14-150.v7a_b_9d17134a_5.hpi org/jenkins-ci/plugins/authentication-tokens/1.53.v1c90fd9191a_b_/authentication-tokens-1.53.v1c90fd9191a_b_.hpi org/jenkins-ci/plugins/authorize-project/1.7.1/authorize-project-1.7.1.hpi org/jenkins-ci/plugins/badge/1.9.1/badge-1.9.1.hpi org/jenkins-ci/plugins/blueocean-display-url/2.4.1/blueocean-display-url-2.4.1.hpi org/jenkins-ci/plugins/bouncycastle-api/2.27/bouncycastle-api-2.27.hpi org/jenkins-ci/plugins/branch-api/2.1092.vda_3c2a_a_f0c11/branch-api-2.1092.vda_3c2a_a_f0c11.hpi org/jenkins-ci/plugins/build-with-parameters/76.v9382db_f78962/build-with-parameters-76.v9382db_f78962.hpi org/jenkins-ci/plugins/cloudbees-bitbucket-branch-source/825.va_6a_dc46a_f97d/cloudbees-bitbucket-branch-source-825.va_6a_dc46a_f97d.hpi org/jenkins-ci/plugins/cloudbees-folder/6.846.v23698686f0f6/cloudbees-folder-6.846.v23698686f0f6.hpi org/jenkins-ci/plugins/credentials-binding/604.vb_64480b_c56ca_/credentials-binding-604.vb_64480b_c56ca_.hpi org/jenkins-ci/plugins/credentials/1271.v54b_1c2c6388a_/credentials-1271.v54b_1c2c6388a_.hpi org/jenkins-ci/plugins/display-url-api/2.3.8/display-url-api-2.3.8.hpi org/jenkins-ci/plugins/durable-task/500.v8927d9fd99d8/durable-task-500.v8927d9fd99d8.hpi org/jenkins-ci/plugins/git-client/4.4.0/git-client-4.4.0.hpi org/jenkins-ci/plugins/git/5.2.0/git-5.2.0.hpi org/jenkins-ci/plugins/github-api/1.314-431.v78d72a_3fe4c3/github-api-1.314-431.v78d72a_3fe4c3.hpi org/jenkins-ci/plugins/github-branch-source/1703.vd5a_2b_29c6cdc/github-branch-source-1703.vd5a_2b_29c6cdc.hpi org/jenkins-ci/plugins/handy-uri-templates-2-api/2.1.8-22.v77d5b_75e6953/handy-uri-templates-2-api-2.1.8-22.v77d5b_75e6953.hpi org/jenkins-ci/plugins/htmlpublisher/1.31/htmlpublisher-1.31.hpi org/jenkins-ci/plugins/http_request/1.18/http_request-1.18.hpi org/jenkins-ci/plugins/jackson2-api/2.15.2-350.v0c2f3f8fc595/jackson2-api-2.15.2-350.v0c2f3f8fc595.hpi org/jenkins-ci/plugins/job-dsl/1.84/job-dsl-1.84.hpi org/jenkins-ci/plugins/junit/1202.v79a_986785076/junit-1202.v79a_986785076.hpi org/jenkins-ci/plugins/mailer/463.vedf8358e006b_/mailer-463.vedf8358e006b_.hpi org/jenkins-ci/plugins/matrix-auth/3.1.10/matrix-auth-3.1.10.hpi org/jenkins-ci/plugins/matrix-project/789.v57a_725b_63c79/matrix-project-789.v57a_725b_63c79.hpi org/jenkins-ci/plugins/metrics/4.2.18-442.v02e107157925/metrics-4.2.18-442.v02e107157925.hpi org/jenkins-ci/plugins/parameterized-trigger/2.46/parameterized-trigger-2.46.hpi org/jenkins-ci/plugins/pipeline-build-step/491.v1fec530da_858/pipeline-build-step-491.v1fec530da_858.hpi org/jenkins-ci/plugins/pipeline-graph-analysis/202.va_d268e64deb_3/pipeline-graph-analysis-202.va_d268e64deb_3.hpi org/jenkins-ci/plugins/pipeline-input-step/468.va_5db_051498a_4/pipeline-input-step-468.va_5db_051498a_4.hpi org/jenkins-ci/plugins/pipeline-milestone-step/101.vd572fef9d926/pipeline-milestone-step-101.vd572fef9d926.hpi org/jenkins-ci/plugins/pipeline-stage-step/305.ve96d0205c1c6/pipeline-stage-step-305.ve96d0205c1c6.hpi org/jenkins-ci/plugins/pipeline-stage-view/pipeline-rest-api/2.33/pipeline-rest-api-2.33.hpi org/jenkins-ci/plugins/pipeline-stage-view/pipeline-stage-view/2.33/pipeline-stage-view-2.33.hpi org/jenkins-ci/plugins/pipeline-utility-steps/2.16.0/pipeline-utility-steps-2.16.0.hpi org/jenkins-ci/plugins/plain-credentials/143.v1b_df8b_d3b_e48/plain-credentials-143.v1b_df8b_d3b_e48.hpi org/jenkins-ci/plugins/pubsub-light/1.16/pubsub-light-1.16.hpi org/jenkins-ci/plugins/resource-disposer/0.21/resource-disposer-0.21.hpi org/jenkins-ci/plugins/scm-api/676.v886669a_199a_a_/scm-api-676.v886669a_199a_a_.hpi org/jenkins-ci/plugins/script-security/1251.vfe552ed55f8d/script-security-1251.vfe552ed55f8d.hpi org/jenkins-ci/plugins/sse-gateway/1.24/sse-gateway-1.24.hpi org/jenkins-ci/plugins/ssh-credentials/305.v8f4381501156/ssh-credentials-305.v8f4381501156.hpi org/jenkins-ci/plugins/ssh-steps/2.0.65.vd26b_5b_9b_de4d/ssh-steps-2.0.65.vd26b_5b_9b_de4d.hpi org/jenkins-ci/plugins/stashNotifier/1.28/stashNotifier-1.28.hpi org/jenkins-ci/plugins/structs/324.va_f5d6774f3a_d/structs-324.va_f5d6774f3a_d.hpi org/jenkins-ci/plugins/swarm/3.40/swarm-3.40.hpi org/jenkins-ci/plugins/timestamper/1.26/timestamper-1.26.hpi org/jenkins-ci/plugins/token-macro/383.v36161104b_002/token-macro-383.v36161104b_002.hpi org/jenkins-ci/plugins/trilead-api/2.84.v72119de229b_7/trilead-api-2.84.v72119de229b_7.hpi org/jenkins-ci/plugins/variant/59.vf075fe829ccb/variant-59.vf075fe829ccb.hpi org/jenkins-ci/plugins/webhook-step/173.vfa_b_93560b_977/webhook-step-173.vfa_b_93560b_977.hpi org/jenkins-ci/plugins/workflow/workflow-aggregator/596.v8c21c963d92d/workflow-aggregator-596.v8c21c963d92d.hpi org/jenkins-ci/plugins/workflow/workflow-api/1215.v2b_ee3e1b_dd39/workflow-api-1215.v2b_ee3e1b_dd39.hpi org/jenkins-ci/plugins/workflow/workflow-basic-steps/1042.ve7b_140c4a_e0c/workflow-basic-steps-1042.ve7b_140c4a_e0c.hpi org/jenkins-ci/plugins/workflow/workflow-cps/3731.ve4b_5b_857b_a_d3/workflow-cps-3731.ve4b_5b_857b_a_d3.hpi org/jenkins-ci/plugins/workflow/workflow-durable-task-step/1247.v7f9dfea_b_4fd0/workflow-durable-task-step-1247.v7f9dfea_b_4fd0.hpi org/jenkins-ci/plugins/workflow/workflow-job/1308.v58d48a_763b_31/workflow-job-1308.v58d48a_763b_31.hpi org/jenkins-ci/plugins/workflow/workflow-multibranch/746.v05814d19c001/workflow-multibranch-746.v05814d19c001.hpi org/jenkins-ci/plugins/workflow/workflow-scm-step/415.v434365564324/workflow-scm-step-415.v434365564324.hpi org/jenkins-ci/plugins/workflow/workflow-step-api/639.v6eca_cd8c04a_a_/workflow-step-api-639.v6eca_cd8c04a_a_.hpi org/jenkins-ci/plugins/workflow/workflow-support/839.v35e2736cfd5c/workflow-support-839.v35e2736cfd5c.hpi org/jenkins-ci/plugins/ws-cleanup/0.45/ws-cleanup-0.45.hpi org/jenkinsci/plugins/pipeline-model-api/2.2144.v077a_d1928a_40/pipeline-model-api-2.2144.v077a_d1928a_40.hpi org/jenkinsci/plugins/pipeline-model-definition/2.2144.v077a_d1928a_40/pipeline-model-definition-2.2144.v077a_d1928a_40.hpi org/jenkinsci/plugins/pipeline-model-extensions/2.2144.v077a_d1928a_40/pipeline-model-extensions-2.2144.v077a_d1928a_40.hpi org/jenkinsci/plugins/pipeline-stage-tags-metadata/2.2144.v077a_d1928a_40/pipeline-stage-tags-metadata-2.2144.v077a_d1928a_40.hpi org/jvnet/hudson/plugins/favorite/2.3.1/favorite-2.3.1.hpi org/jvnet/hudson/plugins/thinBackup/1.18/thinBackup-1.18.hpi
When we saw those log messages (as in the summary of this ticket) we had an issue with the call to build history of an agent, see ticket JENKINS-72138. Maybe the blocking of threads is related to this.
Same here, several different pipelines appear in the log with the "unresponsive for..." message
Dec 21 15:12:42 <host> jenkins[2263991]: 2023-12-21 13:12:42.225+0000 [id=1525] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecution[Owner[<jenkins-pipeline-name> #36]] unresponsive for 1 day 4 hr Dec 21 15:12:42 <host> jenkins[2263991]: 2023-12-21 13:12:42.227+0000 [id=1525] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecution[Owner[<jenkins-pipeline-name> #34]] unresponsive for 3 days 5 hr
Even though I aborted some of them manually they still keep appearing on the log
I started seeing it recently NOT following core/plugins upgrade
Jenkins version: 2.429
Java version: openjdk 11.0.21
Jenkins is running on GCP e2-highmem-4 (4-CPU, 32-RAM) with "-Xms2G -Xmx16G -XX:GCTimeRatio=14 -XX:SoftRefLRUPolicyMSPerMB=50 -XX:+UseG1GC"
markwaite any idea what can be the root-cause or how to troubleshoot this issue?
This issue was reported by many users (for the first time in 2018)
benipeled asked:
any idea what can be the root-cause or how to troubleshoot this issue?
No idea from me and no suggestions on how to troubleshoot the issue.
Is it possible, that the error message looks different with Jenkins 2.479.2 and Java 21?
2024-12-10 08:19:47.008+0000 [id=127127] WARNING o.j.p.w.cps.CpsFlowExecution#blocksRestart: Not blocking restart due to problem checking running steps in CpsFlowExecution[Owner[SOME_JOB_NAME #232]] java.util.concurrent.TimeoutException: Waited 1 seconds (plus 66615 nanoseconds delay) for SettableFuture@7b535cf5[status=PENDING] at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:534) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:119) at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.blocksRestart(CpsFlowExecution.java:1028) at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun$2.blocksRestart(WorkflowRun.java:407) at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun$2.displayCell(WorkflowRun.java:410) at hudson.model.Executor.isDisplayCell(Executor.java:685) at hudson.model.Computer.getDisplayExecutors(Computer.java:1016) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125) at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314) at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185) at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75) at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83) at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57) at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51) at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80) at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:76) at org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:102) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:271) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:271) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:102) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:271) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:102) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:121) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:98) at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:174) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:271) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:100) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:271) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:100) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:271) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:121) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:121) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:98) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:67) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:55) at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:62) at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:42) at org.kohsuke.stapler.Facet$1.dispatch(Facet.java:230) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:800) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:938) at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:369) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:61) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:800) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:938) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:871) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:938) at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:369) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:61) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:800) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:938) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:721) at org.kohsuke.stapler.Stapler.service(Stapler.java:253) at Jenkins Main ClassLoader//jakarta.servlet.http.HttpServlet.service(HttpServlet.java:587) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.servlet.ServletHolder.handle(ServletHolder.java:765) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1668) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:204) at io.jenkins.servlet.FilterChainWrapper$2.doFilter(FilterChainWrapper.java:53) at PluginClassLoader for sse-gateway//org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248) at io.jenkins.servlet.FilterWrapper$1.doFilter(FilterWrapper.java:42) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:201) at io.jenkins.servlet.FilterChainWrapper$2.doFilter(FilterChainWrapper.java:53) at PluginClassLoader for blueocean-web//io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) at io.jenkins.servlet.FilterWrapper$1.doFilter(FilterWrapper.java:42) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:201) at io.jenkins.servlet.FilterChainWrapper$2.doFilter(FilterChainWrapper.java:53) at PluginClassLoader for blueocean-jwt//io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60) at io.jenkins.servlet.FilterWrapper$1.doFilter(FilterWrapper.java:42) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:201) at io.jenkins.servlet.FilterChainWrapper$2.doFilter(FilterChainWrapper.java:53) at PluginClassLoader for metrics//jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at io.jenkins.servlet.FilterWrapper$1.doFilter(FilterWrapper.java:42) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:201) at jenkins.util.HttpServletFilter$1.doFilter(HttpServletFilter.java:77) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:201) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:207) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.servlet.FilterHolder.doFilter(FilterHolder.java:202) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1638) at jenkins.ErrorAttributeFilter.doFilter(ErrorAttributeFilter.java:29) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.servlet.FilterHolder.doFilter(FilterHolder.java:202) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1638) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:154) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.servlet.FilterHolder.doFilter(FilterHolder.java:202) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1638) at hudson.security.ChainedServletFilter2$1.doFilter(ChainedServletFilter2.java:94) at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52) at hudson.security.ChainedServletFilter2$1.doFilter(ChainedServletFilter2.java:99) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54) at hudson.security.ChainedServletFilter2$1.doFilter(ChainedServletFilter2.java:99) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120) at hudson.security.ChainedServletFilter2$1.doFilter(ChainedServletFilter2.java:99) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100) at hudson.security.ChainedServletFilter2$1.doFilter(ChainedServletFilter2.java:99) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:110) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101) at hudson.security.ChainedServletFilter2$1.doFilter(ChainedServletFilter2.java:99) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221) at hudson.security.ChainedServletFilter2$1.doFilter(ChainedServletFilter2.java:99) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:98) at hudson.security.ChainedServletFilter2$1.doFilter(ChainedServletFilter2.java:99) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63) at hudson.security.ChainedServletFilter2$1.doFilter(ChainedServletFilter2.java:99) at hudson.security.ChainedServletFilter2.doFilter(ChainedServletFilter2.java:111) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:173) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.servlet.FilterHolder.doFilter(FilterHolder.java:202) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1638) at org.kohsuke.stapler.UncaughtExceptionFilter.doFilter(UncaughtExceptionFilter.java:26) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.servlet.FilterHolder.doFilter(FilterHolder.java:202) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1638) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.servlet.FilterHolder.doFilter(FilterHolder.java:202) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1638) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:31) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.servlet.FilterHolder.doFilter(FilterHolder.java:202) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1638) at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.servlet.FilterHolder.doFilter(FilterHolder.java:202) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1638) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.servlet.ServletHandler.doHandle(ServletHandler.java:526) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.ScopedHandler.handle(ScopedHandler.java:127) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.security.SecurityHandler.handle(SecurityHandler.java:574) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.HandlerWrapper.handle(HandlerWrapper.java:124) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.ScopedHandler.nextHandle(ScopedHandler.java:197) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.SessionHandler.doHandle(SessionHandler.java:609) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.ScopedHandler.nextHandle(ScopedHandler.java:195) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.ContextHandler.doHandle(ContextHandler.java:1035) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.ScopedHandler.nextScope(ScopedHandler.java:164) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.servlet.ServletHandler.doScope(ServletHandler.java:483) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.ScopedHandler.nextScope(ScopedHandler.java:162) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.SessionHandler.doScope(SessionHandler.java:586) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.ScopedHandler.nextScope(ScopedHandler.java:162) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.ContextHandler.doScope(ContextHandler.java:956) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.ScopedHandler.handle(ScopedHandler.java:125) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.ContextHandler.handle(ContextHandler.java:1694) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1576) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.HttpChannel.dispatch(HttpChannel.java:738) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.HttpChannel.handle(HttpChannel.java:511) at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.ContextHandler$CoreContextHandler$CoreToNestedHandler.handle(ContextHandler.java:2862) at Jenkins Main ClassLoader//org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1060) at Jenkins Main ClassLoader//org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:597) at Jenkins Main ClassLoader//org.eclipse.jetty.server.Server.handle(Server.java:181) at Jenkins Main ClassLoader//org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:661) at Jenkins Main ClassLoader//org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:406) at Jenkins Main ClassLoader//org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322) at Jenkins Main ClassLoader//org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99) at Jenkins Main ClassLoader//org.eclipse.jetty.io.ssl.SslConnection$SslEndPoint.onFillable(SslConnection.java:574) at Jenkins Main ClassLoader//org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:390) at Jenkins Main ClassLoader//org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:150) at Jenkins Main ClassLoader//org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99) at Jenkins Main ClassLoader//org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) at Jenkins Main ClassLoader//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478) at Jenkins Main ClassLoader//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441) at Jenkins Main ClassLoader//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293) at Jenkins Main ClassLoader//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201) at Jenkins Main ClassLoader//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311) at Jenkins Main ClassLoader//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979) at Jenkins Main ClassLoader//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209) at Jenkins Main ClassLoader//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164) at java.base/java.lang.Thread.run(Thread.java:1583)
Edit: Root cause was the performance plugin running perfReport on ~240mb worth of test data.
We're experiencing a similar issue for the past 2 days.
Attached is a thread dump. ganthore-threads.dump![](/images/icons/link_attachment_7.gif)
We're running on core 2.344 and latest pipelines plugins.