Details
-
Bug
-
Status: Resolved (View Workflow)
-
Major
-
Resolution: Fixed
-
System Properties
Name ↓ Value
awt.toolkit sun.awt.X11.XToolkit
executable-war /foobar/jenkins/jenkins.war
file.encoding UTF-8
file.encoding.pkg sun.io
file.separator /
guice.disable.misplaced.annotation.check true
hudson.diyChunking true
hudson.upstreamCulprits true
java.awt.graphicsenv sun.awt.X11GraphicsEnvironment
java.awt.headless true
java.awt.printerjob sun.print.PSPrinterJob
java.class.path /foobar/jenkins/jenkins.war
java.class.version 51.0
java.endorsed.dirs /foobar/dist/jdk/jdk1.7.0_02/jre/lib/endorsed
java.ext.dirs /foobar/dist/jdk/jdk1.7.0_02/jre/lib/ext:/usr/java/packages/lib/ext
java.home /foobar/dist/jdk/jdk1.7.0_02/jre
java.io.tmpdir /foobar/scratch
java.library.path /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
java.runtime.name Java(TM) SE Runtime Environment
java.runtime.version 1.7.0_02-b13
java.specification.name Java Platform API Specification
java.specification.vendor Oracle Corporation
java.specification.version 1.7
java.vendor Oracle Corporationa
java.vendor.url http://java.oracle.com/
java.vendor.url.bug http://bugreport.sun.com/bugreport/
java.version 1.7.0_02
java.vm.info mixed mode
java.vm.name Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.name Java Virtual Machine Specification
java.vm.specification.vendor Oracle Corporation
java.vm.specification.version 1.7
java.vm.vendor Oracle Corporation
java.vm.version 22.0-b10
jna.platform.library.path /usr/lib64:/lib64:/usr/lib:/lib
line.separator
mail.smtp.sendpartial true
mail.smtps.sendpartial true
os.arch amd64
os.name Linux
os.version 2.6.18-164.15.1.el5
path.separator :
securerandom.source file:/dev/./urandom
sun.arch.data.model 64
sun.boot.class.path /foobar/dist/jdk/jdk1.7.0_02/jre/lib/resources.jar:/foobar/dist/jdk/jdk1.7.0_02/jre/lib/rt.jar:/foobar/dist/jdk/jdk1.7.0_02/jre/lib/sunrsasign.jar:/foobar/dist/jdk/jdk1.7.0_02/jre/lib/jsse.jar:/foobar/dist/jdk/jdk1.7.0_02/jre/lib/jce.jar:/foobar/dist/jdk/jdk1.7.0_02/jre/lib/charsets.jar:/foobar/dist/jdk/jdk1.7.0_02/jre/classes
sun.boot.library.path /foobar/dist/jdk/jdk1.7.0_02/jre/lib/amd64
sun.cpu.endian little
sun.cpu.isalist
sun.io.unicode.encoding UnicodeLittle
sun.java.command /foobar/jenkins/jenkins.war --httpPort=8088
sun.java.launcher SUN_STANDARD
sun.jnu.encoding UTF-8
sun.management.compiler HotSpot 64-Bit Tiered Compilers
sun.os.patch.level unknown
svnkit.http.methods Digest,Basic,NTLM,Negotiate
svnkit.ssh2.persistent false
user.country FI
user.dir /foobar/jenkins
user.home /foobar/baz
user.language fi
user.name baz
user.timezone Europe/Helsinki
Environment Variables
Name ↓ Value
G_BROKEN_FILENAMES 1
HISTSIZE 1000
HOME /foobar/baz
HUDSON_HOME /foobar/jenkins
INPUTRC /etc/inputrc
JAVA_HOME /foobar/dist/jdk/current
JENKINS_HOME /foobar/jenkins
JETTYENV_HOME /foobar/baz/env-jetty
LANG en_US.UTF-8
LC_ADDRESS fi_FI.UTF-8
LC_ALL fi_FI.utf8
LC_CTYPE fi_FI.UTF-8
LC_IDENTIFICATION fi_FI.UTF-8
LC_MEASUREMENT fi_FI.UTF-8
LC_MONETARY fi_FI.UTF-8
LC_NAME fi_FI.UTF-8
LC_PAPER fi_FI.UTF-8
LC_TELEPHONE fi_FI.UTF-8
LESSOPEN |/usr/bin/lesspipe.sh %s
LOGNAME baz
LS_COLORS
M2_HOME /foobar/dist/maven/current
MAIL /var/spool/mail/baz
MAVEN_OPTS -XX:+UseCompressedOops -Djava.security.egd=file:///dev/urandom -Xmx512m
MVN /foobar/dist/maven/current/bin/mvn
NLSPATH /usr/dt/lib/nls/msg/%L/%N.cat
PAGER less
PATH /opt/CollabNet_Subversion/bin:/opt/CollabNet_Subversion/bin:/foobar/dist/jdk/current/bin:/foobar/dist/maven/current/bin:/foobar/baz/env-jetty/bin:/foobar/baz/env-jetty/dist/jdk/current/bin:/foobar/baz/env-fab/bin:/foobar/baz/env-apache/bin:/foobar/baz/bin:/foobar/baz/python/bin:/foobar/dist/jdk/current/bin:/foobar/dist/maven/current/bin:/opt/CollabNet_Subversion/bin/:/foobar/scala-2.9.1.final/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/symcli/bin
PWD /foobar/jenkins
READLINK readlink -f
SHELL /bin/bash
SHLVL 1
SSH_TTY /dev/pts/2
TERM xterm-256color
TMP /foobar/baz/scratch
TMPDIR /foobar/baz/scratch
USER baz
USE_SSL on
XFILESEARCHPATH /usr/dt/app-defaults/%L/Dt
Plugins
Name ↓ Version Enabled Pinned
ant 1.1 true false
javadoc 1.0 true false
timestamper 1.2.2 true false
greenballs 1.11 true false
maven-plugin 1.455 true false
instant-messaging 1.21 true false
ircbot 2.18 true false
join 1.13 true false
rsync 1.0-SNAPSHOT (private-02/04/2011 11:58-mpe) true false
PrioritySorter 1.3 false false
claim 1.7 true false
radiatorviewplugin 1.13 true false
build-pipeline-plugin 1.2.3 true false
BlameSubversion 1.25 true false
downstream-buildview 1.6 true false
audit-trail 1.7 true false
cvs 1.6 true false
port-allocator 1.5 true false
disk-usage 0.15 true false
m2-extra-steps 1.1.7 true false
subversion 1.34 true false
ssh-slaves 0.21 true false
next-build-number 1.0 true false
email-ext 2.18 true false
git 1.1.16 true false
translation 1.8System Properties Name ↓ Value awt.toolkit sun.awt.X11.XToolkit executable-war /foobar/jenkins/jenkins.war file.encoding UTF-8 file.encoding.pkg sun.io file.separator / guice.disable.misplaced.annotation.check true hudson.diyChunking true hudson.upstreamCulprits true java.awt.graphicsenv sun.awt.X11GraphicsEnvironment java.awt.headless true java.awt.printerjob sun.print.PSPrinterJob java.class.path /foobar/jenkins/jenkins.war java.class.version 51.0 java.endorsed.dirs /foobar/dist/jdk/jdk1.7.0_02/jre/lib/endorsed java.ext.dirs /foobar/dist/jdk/jdk1.7.0_02/jre/lib/ext:/usr/java/packages/lib/ext java.home /foobar/dist/jdk/jdk1.7.0_02/jre java.io.tmpdir /foobar/scratch java.library.path /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib java.runtime.name Java(TM) SE Runtime Environment java.runtime.version 1.7.0_02-b13 java.specification.name Java Platform API Specification java.specification.vendor Oracle Corporation java.specification.version 1.7 java.vendor Oracle Corporationa java.vendor.url http://java.oracle.com/ java.vendor.url.bug http://bugreport.sun.com/bugreport/ java.version 1.7.0_02 java.vm.info mixed mode java.vm.name Java HotSpot(TM) 64-Bit Server VM java.vm.specification.name Java Virtual Machine Specification java.vm.specification.vendor Oracle Corporation java.vm.specification.version 1.7 java.vm.vendor Oracle Corporation java.vm.version 22.0-b10 jna.platform.library.path /usr/lib64:/lib64:/usr/lib:/lib line.separator mail.smtp.sendpartial true mail.smtps.sendpartial true os.arch amd64 os.name Linux os.version 2.6.18-164.15.1.el5 path.separator : securerandom.source file:/dev/./urandom sun.arch.data.model 64 sun.boot.class.path /foobar/dist/jdk/jdk1.7.0_02/jre/lib/resources.jar:/foobar/dist/jdk/jdk1.7.0_02/jre/lib/rt.jar:/foobar/dist/jdk/jdk1.7.0_02/jre/lib/sunrsasign.jar:/foobar/dist/jdk/jdk1.7.0_02/jre/lib/jsse.jar:/foobar/dist/jdk/jdk1.7.0_02/jre/lib/jce.jar:/foobar/dist/jdk/jdk1.7.0_02/jre/lib/charsets.jar:/foobar/dist/jdk/jdk1.7.0_02/jre/classes sun.boot.library.path /foobar/dist/jdk/jdk1.7.0_02/jre/lib/amd64 sun.cpu.endian little sun.cpu.isalist sun.io.unicode.encoding UnicodeLittle sun.java.command /foobar/jenkins/jenkins.war --httpPort=8088 sun.java.launcher SUN_STANDARD sun.jnu.encoding UTF-8 sun.management.compiler HotSpot 64-Bit Tiered Compilers sun.os.patch.level unknown svnkit.http.methods Digest,Basic,NTLM,Negotiate svnkit.ssh2.persistent false user.country FI user.dir /foobar/jenkins user.home /foobar/baz user.language fi user.name baz user.timezone Europe/Helsinki Environment Variables Name ↓ Value G_BROKEN_FILENAMES 1 HISTSIZE 1000 HOME /foobar/baz HUDSON_HOME /foobar/jenkins INPUTRC /etc/inputrc JAVA_HOME /foobar/dist/jdk/current JENKINS_HOME /foobar/jenkins JETTYENV_HOME /foobar/baz/env-jetty LANG en_US.UTF-8 LC_ADDRESS fi_FI.UTF-8 LC_ALL fi_FI.utf8 LC_CTYPE fi_FI.UTF-8 LC_IDENTIFICATION fi_FI.UTF-8 LC_MEASUREMENT fi_FI.UTF-8 LC_MONETARY fi_FI.UTF-8 LC_NAME fi_FI.UTF-8 LC_PAPER fi_FI.UTF-8 LC_TELEPHONE fi_FI.UTF-8 LESSOPEN |/usr/bin/lesspipe.sh %s LOGNAME baz LS_COLORS M2_HOME /foobar/dist/maven/current MAIL /var/spool/mail/baz MAVEN_OPTS -XX:+UseCompressedOops -Djava.security.egd= file:///dev/urandom -Xmx512m MVN /foobar/dist/maven/current/bin/mvn NLSPATH /usr/dt/lib/nls/msg/%L/%N.cat PAGER less PATH /opt/CollabNet_Subversion/bin:/opt/CollabNet_Subversion/bin:/foobar/dist/jdk/current/bin:/foobar/dist/maven/current/bin:/foobar/baz/env-jetty/bin:/foobar/baz/env-jetty/dist/jdk/current/bin:/foobar/baz/env-fab/bin:/foobar/baz/env-apache/bin:/foobar/baz/bin:/foobar/baz/python/bin:/foobar/dist/jdk/current/bin:/foobar/dist/maven/current/bin:/opt/CollabNet_Subversion/bin/:/foobar/scala-2.9.1.final/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/symcli/bin PWD /foobar/jenkins READLINK readlink -f SHELL /bin/bash SHLVL 1 SSH_TTY /dev/pts/2 TERM xterm-256color TMP /foobar/baz/scratch TMPDIR /foobar/baz/scratch USER baz USE_SSL on XFILESEARCHPATH /usr/dt/app-defaults/%L/Dt Plugins Name ↓ Version Enabled Pinned ant 1.1 true false javadoc 1.0 true false timestamper 1.2.2 true false greenballs 1.11 true false maven-plugin 1.455 true false instant-messaging 1.21 true false ircbot 2.18 true false join 1.13 true false rsync 1.0-SNAPSHOT (private-02/04/2011 11:58-mpe) true false PrioritySorter 1.3 false false claim 1.7 true false radiatorviewplugin 1.13 true false build-pipeline-plugin 1.2.3 true false BlameSubversion 1.25 true false downstream-buildview 1.6 true false audit-trail 1.7 true false cvs 1.6 true false port-allocator 1.5 true false disk-usage 0.15 true false m2-extra-steps 1.1.7 true false subversion 1.34 true false ssh-slaves 0.21 true false next-build-number 1.0 true false email-ext 2.18 true false git 1.1.16 true false translation 1.8
Description
We are seeing repeatable very heavy lock congestion with FingerPrint.save.
Jenkins apparently slides down to a state where a lot of threads have locked / are competing for a lock on FingerPrint instance and are competing for a lock on AnnotationMapper (a singleton). Everything grings to a halt.
Attachments
Issue Links
- is related to
-
-
- Resolved
-
-
-
- Resolved
-
-
JENKINS-19561 Unsafe & inefficient concurrency in XStream
-
- Fixed but Unreleased
-
-
-
- Resolved
-
Activity
Eric Denman
added a comment - @jglick thanks! Filed JENKINS-17412 
Mikko Peltonen
added a comment - We are still seeing lots of congestion with Jenkins 1.517 and our builds may still stay tens of minutes in "Waiting for Jenkins to finish collecting data" -phase.
The place has changed though, now thread dumps show lots of these:
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.Collections$SynchronizedMap.get(Collections.java:2031)
- waiting to lock <0x000000070c6dcfa8> (a java.util.Collections$SynchronizedMap)
at com.thoughtworks.xstream.core.DefaultConverterLookup.lookupConverterForType(DefaultConverterLookup.java:49)
Mikko Peltonen
added a comment - We are still seeing lots of congestion with Jenkins 1.517 and our builds may still stay tens of minutes in "Waiting for Jenkins to finish collecting data" -phase.
The place has changed though, now thread dumps show lots of these:
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.Collections$SynchronizedMap.get(Collections.java:2031)
waiting to lock <0x000000070c6dcfa8> (a java.util.Collections$SynchronizedMap)
at com.thoughtworks.xstream.core.DefaultConverterLookup.lookupConverterForType(DefaultConverterLookup.java:49)
Mikko Peltonen
added a comment - Attached relevant parts of the whole thread dump. @mp3 whatever you are seeing is a distinct bug. Please file separately and use JIRA’s “is related to” link as needed.
(I would have suspected a regression from JENKINS-18775, but that is much newer than 1.517.)
@edenman: separate issue please. (Can use JIRA’s “related” link if appropriate.) Your thread dump anyway does not show Jenkins being blocked, but rather excessively busy doing a couple different things: loading fingerprints; and loading historical build records (perhaps after having discarded them under memory pressure).