-
Bug
-
Resolution: Unresolved
-
Major
-
Jenkins 1.618
Plugins:
active-directory 1.41 true false
analysis-core 1.72 true false
ansicolor 0.4.1 true false
ant 1.2 true false
antisamy-markup-formatter 1.3 true true
async-http-client 1.7.8 true false
audit-trail 2.1 true false
authentication-tokens 1.1 true false
build-flow-plugin 0.18 true false
build-name-setter 1.3 true false
build-pipeline-plugin 1.4.7 true false
build-timeout 1.14.1 true false
build-user-vars-plugin 1.4 true false
build-view-column 0.2 true false
buildgraph-view 1.1.1 true false
built-on-column 1.1 true false
claim 2.7 true false
collapsing-console-sections 1.4.1 true false
conditional-buildstep 1.3.3 true false
configurationslicing 1.41 true false
copyartifact 1.35.2 true false
credentials 1.22 true true
credentials-binding 1.4 true false
cvs 2.12 true true
dashboard-view 2.9.5 false false
description-setter 1.10 true false
docker-build-publish 1.0 true false
docker-commons 1.1 true false
email-ext 2.40.5 true false
embeddable-build-status 1.7 true false
env-to-ant-properties-file 1.0-SNAPSHOT (private-09/02/2013 15:40-pmckeown) true false
envinject 1.91.3 true false
extended-choice-parameter 0.50 true false
external-monitor-job 1.4 true true
file-url-annotator 1.1-SNAPSHOT (private-09/18/2012 00:08-pmckeown) true false
findbugs 4.61 true false
ghprb 1.24.8 true false
git 2.3.5 true false
git-client 1.17.1 true false
github 1.11.3 true false
github-api 1.68 true false
global-variable-string-parameter 1.2 true false
gradle 1.24 true false
groovy 1.25 true false
groovy-postbuild 2.2 true false
hipchat 0.1.9 true false
instant-messaging 1.35 true false
jabber 1.35 true false
javadoc 1.3 true true
jclouds-jenkins 2.8 true false
jenkins-flowdock-plugin 1.1.7 true false
jenkinswalldisplay-pom 0.6.2 false false
job-dsl 1.35 true false
jobConfigHistory 2.11 true false
jquery 1.11.2-0 true false
junit 1.6 true true
kpp-management-plugin 1.0.0 true false
ldap 1.11 true true
leastload 1.0.3 true false
m2release 0.14.0 true false
mailer 1.15 true true
mapdb-api 1.0.6.0 true false
mask-passwords 2.7.3 true false
matrix-auth 1.2 true true
matrix-project 1.4.1 true false
maven-plugin 2.10 true true
maven-repo-cleaner 1.2 true false
mercurial 1.54 true false
metrics 3.0.11 true false
monitoring 1.56.0 true false
multiple-scms 0.4 true false
naginator 1.15 true false
nested-view 1.14 true false
node-iterator-api 1.5 true false
nodelabelparameter 1.5.1 true false
openid4java 0.9.8.0 true false
pam-auth 1.2 true true
parameterized-trigger 2.27 true false
perforce 1.3.35 true false
plain-credentials 1.1 true false
postbuild-task 1.8 true false
promoted-builds 2.21 true false
publish-over-ssh 1.11-SNAPSHOT (private-05/22/2013 23:45-pmckeown) true false
rebuild 1.25 true false
rfc163-jenkins 0.1.12-SNAPSHOT (private-c8923176-lxu) true false
role-strategy 2.2.0 true false
run-condition 1.0 true false
scm-api 0.2 true false
script-security 1.14 true true
show-build-parameters 1.0 true false
sonar 2.2.1 true false
ssh-agent 1.7 true false
ssh-credentials 1.11 true true
ssh-slaves 1.9 true true
subversion 2.5.1 true true
support-core 2.25 true false
suppress-stack-trace 1.3 true false
template-project 1.4-SNAPSHOT (private-02/27/2012 22:17-pmckeown) true false
text-finder 1.10 true false
throttle-concurrents 1.8.4 true false
timestamper 1.6.2 true false
token-macro 1.10 true false
translation 1.12 true true
validating-string-parameter 2.3 true false
view-job-filters 1.26 true false
warnings 4.48 true false
windows-slaves 1.1 true true
workflow-step-api 1.7 true false
ws-cleanup 0.26 true falseJenkins 1.618 Plugins: active-directory 1.41 true false analysis-core 1.72 true false ansicolor 0.4.1 true false ant 1.2 true false antisamy-markup-formatter 1.3 true true async-http-client 1.7.8 true false audit-trail 2.1 true false authentication-tokens 1.1 true false build-flow-plugin 0.18 true false build-name-setter 1.3 true false build-pipeline-plugin 1.4.7 true false build-timeout 1.14.1 true false build-user-vars-plugin 1.4 true false build-view-column 0.2 true false buildgraph-view 1.1.1 true false built-on-column 1.1 true false claim 2.7 true false collapsing-console-sections 1.4.1 true false conditional-buildstep 1.3.3 true false configurationslicing 1.41 true false copyartifact 1.35.2 true false credentials 1.22 true true credentials-binding 1.4 true false cvs 2.12 true true dashboard-view 2.9.5 false false description-setter 1.10 true false docker-build-publish 1.0 true false docker-commons 1.1 true false email-ext 2.40.5 true false embeddable-build-status 1.7 true false env-to-ant-properties-file 1.0-SNAPSHOT (private-09/02/2013 15:40-pmckeown) true false envinject 1.91.3 true false extended-choice-parameter 0.50 true false external-monitor-job 1.4 true true file-url-annotator 1.1-SNAPSHOT (private-09/18/2012 00:08-pmckeown) true false findbugs 4.61 true false ghprb 1.24.8 true false git 2.3.5 true false git-client 1.17.1 true false github 1.11.3 true false github-api 1.68 true false global-variable-string-parameter 1.2 true false gradle 1.24 true false groovy 1.25 true false groovy-postbuild 2.2 true false hipchat 0.1.9 true false instant-messaging 1.35 true false jabber 1.35 true false javadoc 1.3 true true jclouds-jenkins 2.8 true false jenkins-flowdock-plugin 1.1.7 true false jenkinswalldisplay-pom 0.6.2 false false job-dsl 1.35 true false jobConfigHistory 2.11 true false jquery 1.11.2-0 true false junit 1.6 true true kpp-management-plugin 1.0.0 true false ldap 1.11 true true leastload 1.0.3 true false m2release 0.14.0 true false mailer 1.15 true true mapdb-api 1.0.6.0 true false mask-passwords 2.7.3 true false matrix-auth 1.2 true true matrix-project 1.4.1 true false maven-plugin 2.10 true true maven-repo-cleaner 1.2 true false mercurial 1.54 true false metrics 3.0.11 true false monitoring 1.56.0 true false multiple-scms 0.4 true false naginator 1.15 true false nested-view 1.14 true false node-iterator-api 1.5 true false nodelabelparameter 1.5.1 true false openid4java 0.9.8.0 true false pam-auth 1.2 true true parameterized-trigger 2.27 true false perforce 1.3.35 true false plain-credentials 1.1 true false postbuild-task 1.8 true false promoted-builds 2.21 true false publish-over-ssh 1.11-SNAPSHOT (private-05/22/2013 23:45-pmckeown) true false rebuild 1.25 true false rfc163-jenkins 0.1.12-SNAPSHOT (private-c8923176-lxu) true false role-strategy 2.2.0 true false run-condition 1.0 true false scm-api 0.2 true false script-security 1.14 true true show-build-parameters 1.0 true false sonar 2.2.1 true false ssh-agent 1.7 true false ssh-credentials 1.11 true true ssh-slaves 1.9 true true subversion 2.5.1 true true support-core 2.25 true false suppress-stack-trace 1.3 true false template-project 1.4-SNAPSHOT (private-02/27/2012 22:17-pmckeown) true false text-finder 1.10 true false throttle-concurrents 1.8.4 true false timestamper 1.6.2 true false token-macro 1.10 true false translation 1.12 true true validating-string-parameter 2.3 true false view-job-filters 1.26 true false warnings 4.48 true false windows-slaves 1.1 true true workflow-step-api 1.7 true false ws-cleanup 0.26 true false
We run a Jenkins master server with over 3500 jobs defined. We've been running an 8GB heap size for some time, we average arorund 4-6GB used and collections tend to happen reasonably. We recently updated to Jenkins 1.618 (from 1.594) and in the past week our master server has hung in garbage collection twice. Oddly upon restart, it will sometimes immediately hang again.
We took a heap dump of todays failure to see if we had a reference leak or misbehaving plugin and couldn't find anything obvious. However, we have a huge amount of memory allocated to retaining strings. Java.lang.string/char consume %70 of our memory. While many of the strings seem to be job descriptions, recent changelists etc (things found on the job details page) there is a huge number of strings dedicated to what looks like errors and stack traces I'd normally expect to find in a log (see the attached file for a few examples associated with: org.apache.commons.jelly.JellyTagException).
I'm not sure if this is normal behavior or not (never analyzed these part of Jenkins before). We're still trying to figure out what retains these strings. Given that we think this may be "normal" we've bumped our heap to 12GB's to see if we've just gotten so big we need more memory.
I've also attached the monitoring graphs (as images) of relevant monitors. I can, if it's useful, upload the 8GB heap dump somewhere.