Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-76278

High amount of Manage Old Data causes high CPU

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • core
    • None

      We have a lot of Manage Old Data count (over 30000 indicated in the /manage page)

      We use the Docker image `jenkins/jenkins:2.528.1-lts-jdk21` and the /var/jenkins_home is mounted/stored to persistent data.

      Upon pressing the Manage Old Data button, the controller see high CPU and needs to be restarted due to not responding for 40 minutes.

      Not much to see in the logs when it occurs. There's one thing here that looks more like a symptom:

       

      Nov 7, 2025 @ 13:29:38.923
2025-11-07 20:29:27.076+0000 [id=2500] WARNING hudson.model.RunMap#retrieve: could not load /var/jenkins_home/jobs/pull-requests/jobs/myapp-redacted/branches/PR-1/builds/18
Nov 7, 2025 @ 13:29:38.923com.thoughtworks.xstream.security.InputManipulationException: Denial of Service attack assumed. Adding elements to collections or maps exceeds 5 seconds.
Nov 7, 2025 @ 13:29:38.923 at hudson.util.XStream2SecurityUtils.checkForCollectionDoSAttack(XStream2SecurityUtils.java:54)
Nov 7, 2025 @ 13:29:38.923 at hudson.util.RobustMapConverter.putCurrentEntryIntoMap(RobustMapConverter.java:94)
Nov 7, 2025 @ 13:29:38.923Caused: jenkins.util.xstream.CriticalXStreamException: 
Nov 7, 2025 @ 13:29:38.923[LF]> ---- Debugging information ----
Nov 7, 2025 @ 13:29:38.923[LF]> cause-exception : com.thoughtworks.xstream.security.InputManipulationException
Nov 7, 2025 @ 13:29:38.923[LF]> cause-message : Denial of Service attack assumed. Adding elements to collections or maps exceeds 5 seconds.
Nov 7, 2025 @ 13:29:38.923[LF]> class : java.util.HashMap
Nov 7, 2025 @ 13:29:38.923[LF]> required-type : java.util.HashMap
Nov 7, 2025 @ 13:29:38.923[LF]> converter-type : hudson.util.RobustMapConverter
Nov 7, 2025 @ 13:29:38.923[LF]> path : /flow-build/actions/net.masterthought.jenkins.SafeArchiveServingRunAction/fileChecksums/entry[98]
Nov 7, 2025 @ 13:29:38.923[LF]> line number : 3731
Nov 7, 2025 @ 13:29:38.923[LF]> class[1] : net.masterthought.jenkins.SafeArchiveServingRunAction
Nov 7, 2025 @ 13:29:38.923[LF]> required-type[1] : net.masterthought.jenkins.SafeArchiveServingRunAction
Nov 7, 2025 @ 13:29:38.923[LF]> converter-type[1] : hudson.util.RobustReflectionConverter
Nov 7, 2025 @ 13:29:38.923[LF]> class[2] : java.util.concurrent.CopyOnWriteArrayList
Nov 7, 2025 @ 13:29:38.923[LF]> required-type[2] : java.util.concurrent.CopyOnWriteArrayList
Nov 7, 2025 @ 13:29:38.923[LF]> converter-type[2] : hudson.util.RobustCollectionConverter
Nov 7, 2025 @ 13:29:38.923[LF]> class[3] : org.jenkinsci.plugins.workflow.job.WorkflowRun
Nov 7, 2025 @ 13:29:38.923[LF]> required-type[3] : org.jenkinsci.plugins.workflow.job.WorkflowRun
Nov 7, 2025 @ 13:29:38.923[LF]> version : 2.528.1

       
      Did the issue occur after an upgrade of Jenkins or any plugins?

      It seems highly likely that it is related to an upgrade of Jenkins and plugins.

      I have tested the previous version we use, and the one we moved to in our staging instance.

      Previous version info:

      Jenkins: 2.516.1
OS: Linux - 5.10.245-241.976.amzn2.x86_64
Java: 21.0.8 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
—
allure-jenkins-plugin:2.32.0
amazon-ecs:1.49
ansible:524.v9fa_a_4c989224
ansicolor:1.0.6
ant:513.vde9e7b_a_0da_0f
antisamy-markup-formatter:173.v680e3a_b_69ff3
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
apache-httpcomponents-client-5-api:5.5-150.veb_76e719855b_
artifactory:4.0.8
asm-api:9.8-163.vb_2a_96d3f9c3c
atlassian-jira-software-cloud:2.0.15
authentication-tokens:1.131.v7199556c3004
authorize-project:2.0.0
aws-credentials:248.v78a_dcfc9db_ff
aws-java-sdk:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-api-gateway:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-autoscaling:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-cloudformation:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-cloudfront:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-cloudwatch:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-codebuild:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-codedeploy:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ec2:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ecr:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ecs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-efs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-elasticbeanstalk:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-elasticloadbalancingv2:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-iam:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-kinesis:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-lambda:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-logs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-minimal:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-organizations:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-secretsmanager:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-sns:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-sqs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ssm:1.12.780-480.v4a_0819121a_9e
aws-java-sdk2-core:2.31.63-54.vcf1e5a_c56c49
aws-java-sdk2-ec2:2.31.63-54.vcf1e5a_c56c49
aws-secrets-manager-credentials-provider:1.214.va_0a_d8268d068
badge:2.9.1
basic-branch-build-strategies:228.v68c089762a_db_
blueocean:1.27.21
blueocean-bitbucket-pipeline:1.27.21
blueocean-commons:1.27.21
blueocean-config:1.27.21
blueocean-core-js:1.27.21
blueocean-dashboard:1.27.21
blueocean-display-url:2.4.4
blueocean-events:1.27.21
blueocean-git-pipeline:1.27.21
blueocean-github-pipeline:1.27.21
blueocean-i18n:1.27.21
blueocean-jwt:1.27.21
blueocean-personalization:1.27.21
blueocean-pipeline-api-impl:1.27.21
blueocean-pipeline-editor:1.27.21
blueocean-pipeline-scm-api:1.27.21
blueocean-rest:1.27.21
blueocean-rest-impl:1.27.21
blueocean-web:1.27.21
bootstrap5-api:5.3.7-2
bouncycastle-api:2.30.1.80-261.v00c0e2618ec3
branch-api:2.1235.v04e86c7ce54c
caffeine-api:3.2.2-178.v353b_8428ed56
cctray-xml:127.vc433f5b_3b_9e7
checks-api:373.vfe7645102093
cloudbees-bitbucket-branch-source:936.4.4
cloudbees-folder:6.1037.v4cb_8573b_72a_a_
clover:4.14.2.628.ve412b_714e733
command-launcher:123.v37cfdc92ef67
commons-collections4-api:4.5.0-8.va_d5448ef9011
commons-compress-api:1.27.1-3
commons-lang3-api:3.18.0-98.v3a_674c06072d
commons-text-api:1.14.0-194.v804a_dc3a_1b_d8
config-file-provider:988.v0461fcc2b_9d1
configuration-as-code:1985.vdda_32d0c4ea_b_
configuration-as-code-secret-ssm:1.0.1
copyartifact:770.va_6c69e063442
credentials:1419.v2337d1ceceef
credentials-binding:702.vfe613e537e88
cucumber-reports:5.9.0
display-url-api:2.209.v582ed814ff2f
docker-commons:457.v0f62a_94f11a_3
docker-workflow:621.va_73f881d9232
durable-task:595.ve87b_f1318d67
ec2:1985.v20a_803fd248b_
echarts-api:6.0.0-1
eddsa-api:0.3.0.1-19.vc432d923e5ee
emoji-symbols-api:16.0-26.v9818ff7423f0
extended-choice-parameter:388.ve7b_d0b_920e10
extended-read-permission:61.vf24570ff3b_e9
favorite:2.237.v79163ca_8b_892
flaky-test-handler:1.3.168.v9b_0b_fd020db_8
font-awesome-api:7.0.0-1
git:5.7.0
git-client:6.2.0
github:1.44.0
github-api:1.321-488.v9b_c0da_9533f8
github-branch-source:1834.v857721ea_74c6
github-checks:602.v264a_83610da_6
gradle:2.15
groovy-postbuild:272.v52a_03efb_8653
gson-api:2.13.1-153.vb_3d0c48a_a_b_4a_
handy-uri-templates-2-api:2.1.8-36.v85e4cb_234a_13
htmlpublisher:427
ignore-committer-strategy:37.v0d3157c4a_ef8
instance-identity:203.v15e81a_1b_7a_38
ionicons-api:94.vcc3065403257
jackson2-api:2.19.2-408.v18248a_324cfe
jakarta-activation-api:2.1.3-2
jakarta-mail-api:2.1.3-2
javadoc:327.vdfe586651ee0
javax-activation-api:1.2.0-8
javax-mail-api:1.6.2-11
jaxb:2.3.9-133.vb_ec76a_73f706
jdk-tool:83.v417146707a_3d
jenkins-design-language:1.27.21
jersey2-api:2.47-165.ve7809a_3e87e0
jira:3.18
jira-steps:2.0.180.vccfe35b_5910d
jjwt-api:0.11.5-120.v0268cf544b_89
job-dsl:1.93
job-restrictions:216.v0732cc7fb_076
jobcacher:724.veb_f3d7221449
joda-time-api:2.14.0-149.v1c3ce991d1b_9
jquery3-api:3.7.1-3
jsch:0.2.16-95.v3eecb_55fa_b_78
json-api:20250517-163.v1c5da_e99c775
json-path-api:2.9.0-178.vca_b_c71881321
jsoup:1.21.1-58.vfc578e6e2610
junit:1335.v6b_a_a_e18534e1
mailer:515.vd788654779b_1
matrix-auth:3.2.6
matrix-project:849.v0cd64ed7e531
maven-plugin:3.26
metrics:4.2.32-481.v75f035fdc894
mina-sshd-api-common:2.15.0-161.vb_200831a_c15b_
mina-sshd-api-core:2.15.0-161.vb_200831a_c15b_
mina-sshd-api-scp:2.15.0-161.vb_200831a_c15b_
node-iterator-api:72.vc90e81737df1
okhttp-api:4.11.0-189.v976fa_d3379d6
oss-symbols-api:392.v27a_482d90083
parameterized-scheduler:285.ve611986d4c48
performance:988.v5699a_d7c9162
pipeline-aws:1.45
pipeline-build-step:571.v08a_fffd4b_0ce
pipeline-graph-analysis:241.vc3d48fb_b_2582
pipeline-graph-view:617.v1cb_13e8d4825
pipeline-groovy-lib:752.vdddedf804e72
pipeline-input-step:532.v9e7466cb_4406
pipeline-milestone-step:138.v78ca_76831a_43
pipeline-model-api:2.2258.v4e96d2b_da_f9b_
pipeline-model-definition:2.2258.v4e96d2b_da_f9b_
pipeline-model-extensions:2.2258.v4e96d2b_da_f9b_
pipeline-rest-api:2.38
pipeline-stage-step:322.vecffa_99f371c
pipeline-stage-tags-metadata:2.2258.v4e96d2b_da_f9b_
pipeline-stage-view:2.38
pipeline-utility-steps:2.19.0
plain-credentials:199.v9f8e1f741799
plugin-util-api:6.1.0
prism-api:1.30.0-1
prisma-cloud-jenkins-plugin:33.01.137
pubsub-light:1.19
resource-disposer:0.25
run-condition:243.v3c3f94e46a_8b_
saml:4.569.vcff838e19ed1
sauce-ondemand:2.2.0
scm-api:707.v749f968369d4
script-security:1378.vf25626395f49
skip-notifications-trait:542.vf379a_767ced7
slack:794.va_5081e5ea_423
snakeyaml-api:2.3-125.v4d77857a_b_402
snyk-security-scanner:5.0.1
sonar:2.18
sse-gateway:1.28
ssh-agent:386.v36cc0c7582f0
ssh-credentials:361.vb_f6760818e8c
sshd:3.353.v2b_d33c46e970
stashNotifier:1.516.v6d49910f22a_5
structs:350.v3b_30f09f2363
testng-plugin:940.vc244edffc2ec
timestamper:1.30
token-macro:477.vd4f0dc3cb_cf1
trilead-api:2.209.v0e69b_c43c245
variant:70.va_d9f17f859e0
view-job-filters:401.va_809f6a_b_0c26
workflow-aggregator:608.v67378e9d3db_1
workflow-api:1382.veca_a_efe062fa_
workflow-basic-steps:1079.vce64b_a_929c5a_
workflow-cps:4173.v4a_1fd14533f1
workflow-durable-task-step:1444.v10879b_0a_6b_7b_
workflow-job:1540.v295eccc9778f
workflow-multibranch:810.v6b_6e77da_7058
workflow-scm-step:437.v05a_f66b_e5ef8
workflow-step-api:706.v518c5dcb_24c0
workflow-support:976.vb_d9493c2eb_09
ws-cleanup:0.48

       

       

      In the rolled back state, there is no Old data to clean up. We have only a couple jobs setup in our staging instance though, but it is enough to show some evidence of a large amount of Old Data upon upgrading.

      Upon upgrading again, I see that most of the old data entries are on a per-build basis, e.g.:

       

      org.jenkinsci.plugins.workflow.job.WorkflowRun myorgfolderjob » myrepo » main #2 CannotResolveClassException: libraryPath, CannotResolveClassException: libraryPath

       

       

       

      Since this shows up for almost every build of that job (14 of 20 builds), that would easily explain the 40k+ results in our prod instance if its flagging every historical build.

      Does downgrading resolve the issue?

      In our staging instance (which has only a couple jobs for testing), the issue does not exist (rather than being "resolved"), in the sense that it there are no results in the Manage Old Data page.

      Provide a complete step-by-step description of how to reproduce the problem.

      Have a Jenkins setup with a lot of build history using the old version (specified in the first question's answer).
      Spin up a new Jenkins with core/plugin upgrades. This is done via a Docker image replacement with the new versions, not with the UI.

      Go to the <jenkins-url>/manage page.
      See the Manage Old Data button and its result count:

      Click on the button/page.

      It will time out.

      The controller node it self goes into a high CPU state processing all these results, forcing a reboot.

      An additional issue seen upon rebooting is that Jenkins fails to initialize sometimes with a 500 error, it takes a around 3 reboot attempts before Jenkins launches into the UI without this error. But I'd consider this issue out of scope:

      HTTP ERROR 500 jakarta.servlet.ServletException: org.apache.commons.jelly.JellyTagException: jar:file:/var/jenkins_home/war/WEB-INF/lib/jenkins-core-2.528.1.jar!/lib/layout/layout.jelly:99:72: <st:include> Could not initialize class org.codehaus.groovy.runtime.InvokerHelper
URI: /
STATUS: 500
MESSAGE: jakarta.servlet.ServletException: org.apache.commons.jelly.JellyTagException: jar:file:/var/jenkins_home/war/WEB-INF/lib/jenkins-core-2.528.1.jar!/lib/layout/layout.jelly:99:72: <st:include> Could not initialize class org.codehaus.groovy.runtime.InvokerHelper
SERVLET: Stapler
CAUSED BY: org.apache.commons.jelly.JellyTagException: jar:file:/var/jenkins_home/war/WEB-INF/lib/jenkins-core-2.528.1.jar!/lib/layout/layout.jelly:99:72: <st:include> Could not initialize class org.codehaus.groovy.runtime.InvokerHelper
CAUSED BY: java.lang.NoClassDefFoundError: Could not initialize class org.codehaus.groovy.runtime.InvokerHelper
CAUSED BY: java.lang.ExceptionInInitializerError: Exception java.lang.NoClassDefFoundError: Could not initialize class groovy.lang.GroovySystem [in thread "GrapeHack.hack"]
Caused by:org.apache.commons.jelly.JellyTagException: jar:file:/var/jenkins_home/war/WEB-INF/lib/jenkins-core-2.528.1.jar!/lib/layout/layout.jelly:99:72: <st:include> Could not initialize class org.codehaus.groovy.runtime.InvokerHelper
at org.apache.commons.jelly.impl.TagScript.handleException(TagScript.java:744)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:291)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:271)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:100)
...

      Does the issue occur with a new Jenkins installation not reusing any old configuration or data? Try to reproduce the problem with a pristine Jenkins installation with as little customization as possible.

      This issue is based on old config/data, so this doesn't apply to a fresh installation.
      The closest thing to this would be the staging setup described earlier, with that version and set of plugins. In other words, one could setup a job and run some test builds and might be able to see the old data manifest, but it wouldn't be at a high enough scale to cause stability issues.

      If the problem occurs in interaction with other systems (e.g. SCM), include their version and other relevant configuration.

      n/a

      If the Jenkins UI no longer responds, get a thread dump

      This is something that can only reproduced on our prod instance, which we avoid incurring unnecessary downtime on. If it happens again, I will run the dump.

       

        1. screenshot-1.png
          screenshot-1.png
          22 kB

            Unassigned Unassigned
            sgka Sg Ka
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: