-
Bug
-
Resolution: Fixed
-
Major
-
Jenkins 2.107.2
ace-editor-1.1
analysis-core-1.95
ansicolor-0.5.2
ant-1.8
antisamy-markup-formatter-1.5
apache-httpcomponents-client-4-api-4.5.3-2.1
artifactory-2.15.1
authentication-tokens-1.3
badge-1.4
blueocean-1.5.0
blueocean-autofavorite-1.2.2
blueocean-bitbucket-pipeline-1.5.0
blueocean-commons-1.5.0
blueocean-config-1.5.0
blueocean-core-js-1.5.0
blueocean-dashboard-1.5.0
blueocean-display-url-2.2.0
blueocean-events-1.5.0
blueocean-git-pipeline-1.5.0
blueocean-github-pipeline-1.5.0
blueocean-i18n-1.5.0
blueocean-jira-1.5.0
blueocean-jwt-1.5.0
blueocean-personalization-1.5.0
blueocean-pipeline-api-impl-1.5.0
blueocean-pipeline-editor-1.5.0
blueocean-pipeline-scm-api-1.5.0
blueocean-rest-1.5.0
blueocean-rest-impl-1.5.0
blueocean-web-1.5.0
bouncycastle-api-2.16.2
branch-api-2.0.19
build-timeout-1.19
build-user-vars-plugin-1.5
buildtriggerbadge-2.9
checkstyle-3.50
cli-commander-0.3
cloudbees-bitbucket-branch-source-2.2.10
cloudbees-folder-6.4
clover-4.9.0
cobertura-1.12
command-launcher-1.2
conditional-buildstep-1.3.6
config-file-provider-2.18
copyartifact-1.39.1
credentials-2.1.16
credentials-binding-1.16
cucumber-reports-3.16.0
cvs-2.14
dashboard-view-2.9.11
display-url-api-2.2.0
docker-commons-1.11
docker-workflow-1.15.1
durable-task-1.22
email-ext-2.62
embeddable-build-status-1.9
envinject-2.1.5
envinject-api-1.5
extended-read-permission-2.0
external-monitor-job-1.7
favorite-2.3.1
findbugs-4.72
git-3.8.0
git-client-2.7.1
git-server-1.7
github-1.29.0
github-api-1.90
github-branch-source-2.3.3
github-oauth-0.29
gradle-1.28
groovy-2.0
groovy-postbuild-2.4
handlebars-1.1.1
handy-uri-templates-2-api-2.1.6-1.0
htmlpublisher-1.16
http_request-1.8.22
ibm-application-security-1.1.2
icon-shim-2.0.3
ivy-1.28
jackson2-api-2.8.11.1
jacoco-3.0.1
javadoc-1.4
jenkins-design-language-1.5.0
jira-2.5.1
job-dsl-1.69
jquery-detached-1.2.1
jsch-0.1.54.2
junit-1.24
ldap-1.20
lockable-resources-2.2
mailer-1.21
mapdb-api-1.0.9.0
matrix-auth-2.2
matrix-project-1.13
maven-plugin-3.1.2
mercurial-2.3
metrics-3.1.2.11
metrics-graphite-3.0.0
modernstatus-1.2
momentjs-1.1.1
monitoring-1.72.0
pam-auth-1.3
parameterized-trigger-2.35.2
pegdown-formatter-1.3
performance-3.8
pipeline-build-step-2.7
pipeline-github-2.0
pipeline-github-lib-1.0
pipeline-graph-analysis-1.6
pipeline-input-step-2.8
pipeline-maven-3.5.3
pipeline-milestone-step-1.3.1
pipeline-model-api-1.2.9
pipeline-model-declarative-agent-1.1.1
pipeline-model-definition-1.2.9
pipeline-model-extensions-1.2.9
pipeline-rest-api-2.10
pipeline-stage-step-2.3
pipeline-stage-tags-metadata-1.2.9
pipeline-stage-view-2.10
pipeline-utility-steps-2.0.2
plain-credentials-1.4
pmd-3.50
pubsub-light-1.12
rebuild-1.28
resource-disposer-0.8
run-condition-1.0
scm-api-2.2.6
script-security-1.44
sidebar-link-1.9.1
simple-theme-plugin-0.4
slack-2.3
sse-gateway-1.15
ssh-agent-1.15
ssh-credentials-1.13
ssh-slaves-1.26
structs-1.14
subversion-2.10.5
support-core-2.46
swarm-3.12
tap-2.2.1
timestamper-1.8.9
token-macro-2.5
translation-1.16
variant-1.1
view-job-filters-1.27
webhook-step-1.3
windows-slaves-1.3.1
workflow-aggregator-2.5
workflow-api-2.27
workflow-basic-steps-2.6
workflow-cps-2.48
workflow-cps-global-lib-2.9
workflow-durable-task-step-2.19
workflow-job-2.19
workflow-multibranch-2.17
workflow-scm-step-2.6
workflow-step-api-2.14
workflow-support-2.18
ws-cleanup-0.34
xunit-1.102Jenkins 2.107.2 ace-editor-1.1 analysis-core-1.95 ansicolor-0.5.2 ant-1.8 antisamy-markup-formatter-1.5 apache-httpcomponents-client-4-api-4.5.3-2.1 artifactory-2.15.1 authentication-tokens-1.3 badge-1.4 blueocean-1.5.0 blueocean-autofavorite-1.2.2 blueocean-bitbucket-pipeline-1.5.0 blueocean-commons-1.5.0 blueocean-config-1.5.0 blueocean-core-js-1.5.0 blueocean-dashboard-1.5.0 blueocean-display-url-2.2.0 blueocean-events-1.5.0 blueocean-git-pipeline-1.5.0 blueocean-github-pipeline-1.5.0 blueocean-i18n-1.5.0 blueocean-jira-1.5.0 blueocean-jwt-1.5.0 blueocean-personalization-1.5.0 blueocean-pipeline-api-impl-1.5.0 blueocean-pipeline-editor-1.5.0 blueocean-pipeline-scm-api-1.5.0 blueocean-rest-1.5.0 blueocean-rest-impl-1.5.0 blueocean-web-1.5.0 bouncycastle-api-2.16.2 branch-api-2.0.19 build-timeout-1.19 build-user-vars-plugin-1.5 buildtriggerbadge-2.9 checkstyle-3.50 cli-commander-0.3 cloudbees-bitbucket-branch-source-2.2.10 cloudbees-folder-6.4 clover-4.9.0 cobertura-1.12 command-launcher-1.2 conditional-buildstep-1.3.6 config-file-provider-2.18 copyartifact-1.39.1 credentials-2.1.16 credentials-binding-1.16 cucumber-reports-3.16.0 cvs-2.14 dashboard-view-2.9.11 display-url-api-2.2.0 docker-commons-1.11 docker-workflow-1.15.1 durable-task-1.22 email-ext-2.62 embeddable-build-status-1.9 envinject-2.1.5 envinject-api-1.5 extended-read-permission-2.0 external-monitor-job-1.7 favorite-2.3.1 findbugs-4.72 git-3.8.0 git-client-2.7.1 git-server-1.7 github-1.29.0 github-api-1.90 github-branch-source-2.3.3 github-oauth-0.29 gradle-1.28 groovy-2.0 groovy-postbuild-2.4 handlebars-1.1.1 handy-uri-templates-2-api-2.1.6-1.0 htmlpublisher-1.16 http_request-1.8.22 ibm-application-security-1.1.2 icon-shim-2.0.3 ivy-1.28 jackson2-api-2.8.11.1 jacoco-3.0.1 javadoc-1.4 jenkins-design-language-1.5.0 jira-2.5.1 job-dsl-1.69 jquery-detached-1.2.1 jsch-0.1.54.2 junit-1.24 ldap-1.20 lockable-resources-2.2 mailer-1.21 mapdb-api-1.0.9.0 matrix-auth-2.2 matrix-project-1.13 maven-plugin-3.1.2 mercurial-2.3 metrics-3.1.2.11 metrics-graphite-3.0.0 modernstatus-1.2 momentjs-1.1.1 monitoring-1.72.0 pam-auth-1.3 parameterized-trigger-2.35.2 pegdown-formatter-1.3 performance-3.8 pipeline-build-step-2.7 pipeline-github-2.0 pipeline-github-lib-1.0 pipeline-graph-analysis-1.6 pipeline-input-step-2.8 pipeline-maven-3.5.3 pipeline-milestone-step-1.3.1 pipeline-model-api-1.2.9 pipeline-model-declarative-agent-1.1.1 pipeline-model-definition-1.2.9 pipeline-model-extensions-1.2.9 pipeline-rest-api-2.10 pipeline-stage-step-2.3 pipeline-stage-tags-metadata-1.2.9 pipeline-stage-view-2.10 pipeline-utility-steps-2.0.2 plain-credentials-1.4 pmd-3.50 pubsub-light-1.12 rebuild-1.28 resource-disposer-0.8 run-condition-1.0 scm-api-2.2.6 script-security-1.44 sidebar-link-1.9.1 simple-theme-plugin-0.4 slack-2.3 sse-gateway-1.15 ssh-agent-1.15 ssh-credentials-1.13 ssh-slaves-1.26 structs-1.14 subversion-2.10.5 support-core-2.46 swarm-3.12 tap-2.2.1 timestamper-1.8.9 token-macro-2.5 translation-1.16 variant-1.1 view-job-filters-1.27 webhook-step-1.3 windows-slaves-1.3.1 workflow-aggregator-2.5 workflow-api-2.27 workflow-basic-steps-2.6 workflow-cps-2.48 workflow-cps-global-lib-2.9 workflow-durable-task-step-2.19 workflow-job-2.19 workflow-multibranch-2.17 workflow-scm-step-2.6 workflow-step-api-2.14 workflow-support-2.18 ws-cleanup-0.34 xunit-1.102
We started running out of memory in our JVM (Xmx 8G) and when looking at Melody's memory (heap) histogram (JENKINS_URL/monitoring?part=heaphisto) the top two items were:
Class | Size (Kb) | % size | Instances | % instances | Source |
---|---|---|---|---|---|
org.jenkinsci.plugins.ssegateway.sse.EventDispatcher$Retry | 2,890,399 | 44 | 92,492,793 | 43 | |
java.util.concurrent.ConcurrentLinkedQueue$Node | 2,167,981 | 33 | 92,500,553 | 43 |
77% (and growing as we were researching the problem) of the memory was being used by these two items.
I have two support bundles from this time and an .hprof as well.
I can either screen share with someone or if you can tell me how to analyze these files I would be happy to.
- is related to
-
JENKINS-45488 Pipeline: StackOverflowError at io.jenkins.blueocean.events.PipelineEventListener$StageEventPublisher.getParentBlock(PipelineEventListener.java:110)
-
- Closed
-
- relates to
-
JENKINS-58684 Linux max task resource exhaustion after excessive timer thread creation
-
- Resolved
-
After many tests and cleanup scripts, we still we able to find some EventDispatcher$Retry elements in the heap but no subscribers in the GuavaPubSubBus or EventDispatchers in the HttpSessions. Some additional verifications in the plugin made me notice that as AsyncEventDispatcher was being used, it is possible that the EventDispatchers were still being referenced by some asyncContexts or threads that were not completed or released. Finally I also added a dispatcher.stop() call in my cleanup script, and it looks like there are no traces of leaked EventDispatcher$Retry clases in the heap anymore, but we still keep observing our instance and analyzing heap dumps, it is very soon to confirm.
Just in case it can help, this is the cleanup script we are running daily: