-
Bug
-
Resolution: Fixed
-
Critical
-
Hide[chris@fen-vz-jenkins ~]$ java -version
java version "1.6.0_22"
OpenJDK Runtime Environment (IcedTea6 1.10.8) (rhel-1.27.1.10.8.el5_8-i386)
OpenJDK Server VM (build 20.0-b11, mixed mode)
[chris@fen-vz-jenkins ~]$ uname -a
Linux fen-vz-jenkins.fen.aptivate.org 2.6.32-7-pve #1 SMP Mon Feb 13 07:33:21 CET 2012 i686 i686 i386 GNU/Linux
[chris@fen-vz-jenkins ~]$ cat /etc/issue
CentOS release 5.8 (Final)Show[ chris@fen-vz-jenkins ~]$ java -version java version "1.6.0_22" OpenJDK Runtime Environment (IcedTea6 1.10.8) (rhel-1.27.1.10.8.el5_8-i386) OpenJDK Server VM (build 20.0-b11, mixed mode) [ chris@fen-vz-jenkins ~]$ uname -a Linux fen-vz-jenkins.fen.aptivate.org 2.6.32-7-pve #1 SMP Mon Feb 13 07:33:21 CET 2012 i686 i686 i386 GNU/Linux [ chris@fen-vz-jenkins ~]$ cat /etc/issue CentOS release 5.8 (Final)
-
Powered by SuggestiMate
Jenkins starts using 100% CPU after a few days. Using jstack I see several threads trying to write compressed output, and apparently not changing over time:
- java.util.zip.Deflater.deflate(byte[], int, int) @bci=55, line=322 (Compiled frame; information may be imprecise)
- java.util.zip.DeflaterOutputStream.deflate() @bci=14, line=176 (Compiled frame)
- java.util.zip.DeflaterOutputStream.write(byte[], int, int) @bci=108, line=135 (Compiled frame)
- java.util.zip.GZIPOutputStream.write(byte[], int, int) @bci=4, line=89 (Compiled frame)
- org.kohsuke.stapler.compression.FilterServletOutputStream.write(byte[], int, int) @bci=7, line=31 (Compiled frame)
- sun.nio.cs.StreamEncoder.writeBytes() @bci=120, line=220 (Interpreted frame)
- sun.nio.cs.StreamEncoder.implClose() @bci=84, line=315 (Interpreted frame)
- sun.nio.cs.StreamEncoder.close() @bci=18, line=148 (Interpreted frame)
- java.io.OutputStreamWriter.close() @bci=4, line=233 (Interpreted frame)
- java.io.PrintWriter.close() @bci=21, line=312 (Interpreted frame)
- org.kohsuke.stapler.compression.CompressionFilter.reportException(java.lang.Exception, javax.servlet.http.HttpServletResponse) @bci=112, line=77 (Interpreted frame)
- org.kohsuke.stapler.compression.CompressionFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=57, line=53 (Compiled frame)
- winstone.FilterConfiguration.execute(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=25, line=194 (Compiled frame)
- winstone.RequestDispatcher.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=48, line=366 (Compiled frame)
- hudson.util.CharacterEncodingFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=43, line=81 (Compiled frame)
- winstone.FilterConfiguration.execute(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=25, line=194 (Compiled frame)
- winstone.RequestDispatcher.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=48, line=366 (Compiled frame)
- winstone.RequestDispatcher.forward(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=483, line=331 (Compiled frame)
- winstone.RequestHandlerThread.processRequest(winstone.WebAppConfiguration, winstone.WinstoneRequest, winstone.WinstoneResponse, java.lang.String) @bci=38, line=215 (Compiled frame)
- winstone.RequestHandlerThread.run() @bci=631, line=138 (Compiled frame)
- java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=471 (Interpreted frame)
- java.util.concurrent.FutureTask$Sync.innerRun() @bci=29, line=334 (Interpreted frame)
- java.util.concurrent.FutureTask.run() @bci=4, line=166 (Interpreted frame)
- winstone.BoundedExecutorService$1.run() @bci=4, line=77 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=679 (Interpreted frame)
I'm suspecting but not 100% sure that these threads are in an infinite loop (livelocked). I'm struggling to see what other threads might be doing this.
This JVM was not started with debugging enabled to attach a debugger for analysis. I've enabled it now. Stack traces attached as files below.
- depends on
-
JENKINS-19473 failed to archive slave artifacts. Unexpected end of ZLIB input stream
-
- Closed
-
- is duplicated by
-
JENKINS-14361 100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException
-
- Resolved
-
-
JENKINS-18513 Slowness in Jenkins
-
- Resolved
-
-
JENKINS-19346 100% on one CPU
-
- Resolved
-
- is related to
-
JENKINS-13625 ERR_CONTENT_DECODING_FAILED returned on testResults and console output after Jenkins reload
-
- Closed
-
-
JENKINS-17349 CPU on master node keeps increasing and never come down
-
- Resolved
-
[JENKINS-14362] 100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException
On 1.511 this is still happening, even though I set:
<arguments>-Xrs -Xmx1024m -XX:MaxPermSize=512m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "%BASE%\jenkins.war" --httpPort=8081 --prefix=/jenkins -Dorg.kohsuke.stapler.compression.CompressionFilter.disabled=true</arguments>
Since my previous comment I have noticed two incidents of 100% cpu usage from a zip compression thread that appears "stuck". Either org.kohsuke.stapler.compression.CompressionFilter.disabled=true reduces the frequency ; a different path results in the same behavior ; or the reduction in frequency is all by chance.
You should create a thread dump from your vm to enrich this report. Configuration is mentioned in http://devophuman.blogspot.com/2013/04/jenkins-is-going-wild.html. The VisualVM is part of Oracles JDK...
Experiencing this daily as well. Fairly frustrating.
Jenkins 1.514
java version "1.6.0_27"
OpenJDK Runtime Environment (IcedTea6 1.12.3) (6b27-1.12.3-0ubuntu1~12.04.1)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
Stack trace of looping thread:
java.lang.Thread.State: RUNNABLE at java.util.zip.Deflater.deflateBytes(Native Method) at java.util.zip.Deflater.deflate(Deflater.java:322) - locked <0x000000008cbec4d8> (a java.util.zip.ZStreamRef) at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:176) at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:135) at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:89) - locked <0x000000008cbec490> (a java.util.zip.GZIPOutputStream) at org.kohsuke.stapler.compression.FilterServletOutputStream.write(FilterServletOutputStream.java:31) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:220) at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:315) at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:148) - locked <0x000000008cc1a478> (a java.io.OutputStreamWriter) at java.io.OutputStreamWriter.close(OutputStreamWriter.java:233) at java.io.PrintWriter.close(PrintWriter.java:312) - locked <0x000000008cc1a478> (a java.io.OutputStreamWriter) at org.kohsuke.stapler.compression.CompressionFilter.reportException(CompressionFilter.java:78) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:227) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150)
Thread dump of this issue occuring on 1.516:
Thread Dump AWT-Windows "AWT-Windows" Id=217 Group=main RUNNABLE (in native) at sun.awt.windows.WToolkit.eventLoop(Native Method) at sun.awt.windows.WToolkit.run(WToolkit.java:299) at java.lang.Thread.run(Thread.java:722) bot1-input "bot1-input" Id=167 Group=main RUNNABLE (in native) at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at sun.security.ssl.InputRecord.read(InputRecord.java:480) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) - locked java.lang.Object@362d4a04 at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) - locked sun.security.ssl.AppInputStream@57566865 at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177) - locked java.io.InputStreamReader@69dfebae at java.io.InputStreamReader.read(InputStreamReader.java:184) at java.io.BufferedReader.fill(BufferedReader.java:154) at java.io.BufferedReader.readLine(BufferedReader.java:317) - locked java.io.InputStreamReader@69dfebae at java.io.BufferedReader.readLine(BufferedReader.java:382) at org.pircbotx.InputThread.run(InputThread.java:83) bot1-output "bot1-output" Id=168 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6bea22cf at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6bea22cf at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.pircbotx.OutputThread.run(OutputThread.java:101) Channel reader thread: serial-server-1 "Channel reader thread: serial-server-1" Id=146 Group=main WAITING on java.lang.Object@61e52512 at java.lang.Object.wait(Native Method) - waiting on java.lang.Object@61e52512 at java.lang.Object.wait(Object.java:503) at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:144) at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2272) at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2565) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2575) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1315) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) Channel reader thread: ubuntu-test-VM-3 "Channel reader thread: ubuntu-test-VM-3" Id=148 Group=main WAITING on java.lang.Object@1c5e0a0a at java.lang.Object.wait(Native Method) - waiting on java.lang.Object@1c5e0a0a at java.lang.Object.wait(Object.java:503) at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:144) at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2272) at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2565) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2575) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1315) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) ConnectorThread:[http-8081] "ConnectorThread:[http-8081]" Id=12 Group=main RUNNABLE (in native) at java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method) at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:125) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:183) - locked java.net.SocksSocketImpl@773d4576 at java.net.ServerSocket.implAccept(ServerSocket.java:522) at java.net.ServerSocket.accept(ServerSocket.java:490) at winstone.HttpListener.run(HttpListener.java:139) at java.lang.Thread.run(Thread.java:722) DestroyJavaVM "DestroyJavaVM" Id=15 Group=main RUNNABLE Executor #0 for D0-Debian-Slave-1 "Executor #0 for D0-Debian-Slave-1" Id=86 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for D0-Debian-Slave-2 "Executor #0 for D0-Debian-Slave-2" Id=91 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for D1-debian-FreshInstall-1 "Executor #0 for D1-debian-FreshInstall-1" Id=116 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for D2-Debian-Slave-1 "Executor #0 for D2-Debian-Slave-1" Id=111 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for D2-Debian-Slave-2 "Executor #0 for D2-Debian-Slave-2" Id=106 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for D3-Debian-Slave-1 "Executor #0 for D3-Debian-Slave-1" Id=96 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for D3-Debian-Slave-2 "Executor #0 for D3-Debian-Slave-2" Id=101 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for master "Executor #0 for master" Id=38 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for serial-server-1 "Executor #0 for serial-server-1" Id=48 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for ubuntu-test-VM-1 "Executor #0 for ubuntu-test-VM-1" Id=64 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for ubuntu-test-VM-2 "Executor #0 for ubuntu-test-VM-2" Id=75 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for ubuntu-test-VM-3 "Executor #0 for ubuntu-test-VM-3" Id=53 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for D0-Debian-Slave-1 "Executor #1 for D0-Debian-Slave-1" Id=87 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for D0-Debian-Slave-2 "Executor #1 for D0-Debian-Slave-2" Id=92 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for D2-Debian-Slave-1 "Executor #1 for D2-Debian-Slave-1" Id=112 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for D2-Debian-Slave-2 "Executor #1 for D2-Debian-Slave-2" Id=107 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for D3-Debian-Slave-1 "Executor #1 for D3-Debian-Slave-1" Id=97 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for D3-Debian-Slave-2 "Executor #1 for D3-Debian-Slave-2" Id=102 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for master "Executor #1 for master" Id=39 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for serial-server-1 "Executor #1 for serial-server-1" Id=49 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for ubuntu-test-VM-1 "Executor #1 for ubuntu-test-VM-1" Id=65 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for ubuntu-test-VM-2 "Executor #1 for ubuntu-test-VM-2" Id=76 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for ubuntu-test-VM-3 "Executor #1 for ubuntu-test-VM-3" Id=54 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for D0-Debian-Slave-1 "Executor #2 for D0-Debian-Slave-1" Id=88 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for D0-Debian-Slave-2 "Executor #2 for D0-Debian-Slave-2" Id=93 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for D2-Debian-Slave-1 "Executor #2 for D2-Debian-Slave-1" Id=113 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for D2-Debian-Slave-2 "Executor #2 for D2-Debian-Slave-2" Id=108 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for D3-Debian-Slave-1 "Executor #2 for D3-Debian-Slave-1" Id=98 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for D3-Debian-Slave-2 "Executor #2 for D3-Debian-Slave-2" Id=103 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for master "Executor #2 for master" Id=40 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for serial-server-1 "Executor #2 for serial-server-1" Id=50 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for ubuntu-test-VM-1 "Executor #2 for ubuntu-test-VM-1" Id=66 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for ubuntu-test-VM-2 "Executor #2 for ubuntu-test-VM-2" Id=77 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for ubuntu-test-VM-3 "Executor #2 for ubuntu-test-VM-3" Id=55 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for D0-Debian-Slave-1 "Executor #3 for D0-Debian-Slave-1" Id=89 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for D0-Debian-Slave-2 "Executor #3 for D0-Debian-Slave-2" Id=94 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for D2-Debian-Slave-1 "Executor #3 for D2-Debian-Slave-1" Id=114 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for D2-Debian-Slave-2 "Executor #3 for D2-Debian-Slave-2" Id=109 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for D3-Debian-Slave-1 "Executor #3 for D3-Debian-Slave-1" Id=99 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for D3-Debian-Slave-2 "Executor #3 for D3-Debian-Slave-2" Id=104 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for master "Executor #3 for master" Id=41 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for serial-server-1 "Executor #3 for serial-server-1" Id=51 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for ubuntu-test-VM-1 "Executor #3 for ubuntu-test-VM-1" Id=67 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for ubuntu-test-VM-2 "Executor #3 for ubuntu-test-VM-2" Id=78 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for ubuntu-test-VM-3 "Executor #3 for ubuntu-test-VM-3" Id=56 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for D0-Debian-Slave-1 "Executor #4 for D0-Debian-Slave-1" Id=90 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for D0-Debian-Slave-2 "Executor #4 for D0-Debian-Slave-2" Id=95 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for D2-Debian-Slave-1 "Executor #4 for D2-Debian-Slave-1" Id=115 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for D2-Debian-Slave-2 "Executor #4 for D2-Debian-Slave-2" Id=110 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for D3-Debian-Slave-1 "Executor #4 for D3-Debian-Slave-1" Id=100 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for D3-Debian-Slave-2 "Executor #4 for D3-Debian-Slave-2" Id=105 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for master "Executor #4 for master" Id=42 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for ubuntu-test-VM-1 "Executor #4 for ubuntu-test-VM-1" Id=68 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for ubuntu-test-VM-2 "Executor #4 for ubuntu-test-VM-2" Id=79 Group=main TIMED_WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at hudson.util.OneShotEvent.block(OneShotEvent.java:86) at hudson.model.Queue.pop(Queue.java:884) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for ubuntu-test-VM-3 "Executor #4 for ubuntu-test-VM-3" Id=57 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #5 for master "Executor #5 for master" Id=43 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #5 for ubuntu-test-VM-1 "Executor #5 for ubuntu-test-VM-1" Id=69 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #5 for ubuntu-test-VM-2 "Executor #5 for ubuntu-test-VM-2" Id=80 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #5 for ubuntu-test-VM-3 "Executor #5 for ubuntu-test-VM-3" Id=58 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #6 for master "Executor #6 for master" Id=44 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #6 for ubuntu-test-VM-1 "Executor #6 for ubuntu-test-VM-1" Id=70 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #6 for ubuntu-test-VM-2 "Executor #6 for ubuntu-test-VM-2" Id=81 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #6 for ubuntu-test-VM-3 "Executor #6 for ubuntu-test-VM-3" Id=59 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #7 for master "Executor #7 for master" Id=45 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #7 for ubuntu-test-VM-1 "Executor #7 for ubuntu-test-VM-1" Id=71 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #7 for ubuntu-test-VM-2 "Executor #7 for ubuntu-test-VM-2" Id=82 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #7 for ubuntu-test-VM-3 "Executor #7 for ubuntu-test-VM-3" Id=60 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #8 for master "Executor #8 for master" Id=46 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #8 for ubuntu-test-VM-1 "Executor #8 for ubuntu-test-VM-1" Id=72 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #8 for ubuntu-test-VM-2 "Executor #8 for ubuntu-test-VM-2" Id=83 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #8 for ubuntu-test-VM-3 "Executor #8 for ubuntu-test-VM-3" Id=61 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #9 for master "Executor #9 for master" Id=47 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #9 for ubuntu-test-VM-1 "Executor #9 for ubuntu-test-VM-1" Id=73 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #9 for ubuntu-test-VM-2 "Executor #9 for ubuntu-test-VM-2" Id=84 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #9 for ubuntu-test-VM-3 "Executor #9 for ubuntu-test-VM-3" Id=62 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Handling GET /jenkins//job/Hardware_Stability_Tests_Sequoia/api/json : RequestHandlerThread[#2] "Handling GET /jenkins//job/Hardware_Stability_Tests_Sequoia/api/json : RequestHandlerThread[#2]" Id=25 Group=main RUNNABLE at java.lang.String.intern(Native Method) at java.lang.Class.searchMethods(Class.java:2670) at java.lang.Class.getMethod0(Class.java:2694) at java.lang.Class.getMethod(Class.java:1622) at org.apache.commons.logging.LogFactory.directGetContextClassLoader(LogFactory.java:896) at org.apache.commons.logging.LogFactory$1.run(LogFactory.java:862) at java.security.AccessController.doPrivileged(Native Method) at org.apache.commons.logging.LogFactory.getContextClassLoaderInternal(LogFactory.java:859) at org.apache.commons.logging.LogFactory.getFactory(LogFactory.java:423) at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:685) at org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:942) at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:274) at org.apache.commons.digester.SetNextRule.end(SetNextRule.java:220) at org.apache.commons.digester.Rule.end(Rule.java:257) at org.apache.commons.digester.Digester.endElement(Digester.java:1345) at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:606) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1742) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2900) at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:607) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:489) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:835) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764) at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:123) at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1210) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:568) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43) at hudson.model.AbstractBuild.calcChangeSet(AbstractBuild.java:922) at hudson.model.AbstractBuild.getChangeSet(AbstractBuild.java:896) at hudson.model.AbstractBuild.hasParticipant(AbstractBuild.java:461) at hudson.model.AbstractProject.hasParticipant(AbstractProject.java:1581) at hudson.model.User.getProjects(User.java:449) at hudson.scm.MailAddressResolverImpl.findMailAddressFor(MailAddressResolverImpl.java:21) at hudson.tasks.MailAddressResolver.resolve(MailAddressResolver.java:112) at hudson.tasks.Mailer$UserProperty.getAddress(Mailer.java:532) at sun.reflect.GeneratedMethodAccessor242.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.kohsuke.stapler.export.MethodProperty.getValue(MethodProperty.java:66) at org.kohsuke.stapler.export.Property.writeTo(Property.java:114) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:187) at org.kohsuke.stapler.export.Property.writeValue(Property.java:232) at org.kohsuke.stapler.export.Property.writeValue(Property.java:182) at org.kohsuke.stapler.export.Property.writeValue(Property.java:137) at org.kohsuke.stapler.export.Property.writeTo(Property.java:114) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:187) at org.kohsuke.stapler.export.Property.writeValue(Property.java:232) at org.kohsuke.stapler.export.Property.writeValue(Property.java:182) at org.kohsuke.stapler.export.Property.writeValue(Property.java:137) at org.kohsuke.stapler.export.Property.writeTo(Property.java:114) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:187) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182) at org.kohsuke.stapler.export.Property.writeValue(Property.java:232) at org.kohsuke.stapler.export.Property.writeValue(Property.java:137) at org.kohsuke.stapler.export.Property.writeTo(Property.java:114) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:187) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182) at org.kohsuke.stapler.export.Model.writeTo(Model.java:154) at org.kohsuke.stapler.ResponseImpl.serveExposedBean(ResponseImpl.java:264) at hudson.model.Api.doJson(Api.java:193) at sun.reflect.GeneratedMethodAccessor236.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:288) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:151) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:90) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:203) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:583) at org.kohsuke.stapler.Stapler.service(Stapler.java:214) at javax.servlet.http.HttpServlet.service(HttpServlet.java:45) at winstone.ServletConfiguration.execute(ServletConfiguration.java:248) at winstone.RequestDispatcher.forward(RequestDispatcher.java:333) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:36) at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:103) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:32) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:124) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:64) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:227) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@128d38f3 Handling GET /jenkins/threadDump : RequestHandlerThread[#7] "Handling GET /jenkins/threadDump : RequestHandlerThread[#7]" Id=33 Group=main TIMED_WAITING on java.util.concurrent.FutureTask$Sync@81e7b1d at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.FutureTask$Sync@81e7b1d at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1033) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326) at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:257) at java.util.concurrent.FutureTask.get(FutureTask.java:119) at hudson.remoting.LocalChannel$2.get(LocalChannel.java:81) at jenkins.model.Jenkins.getAllThreadDumps(Jenkins.java:2865) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258) at org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104) at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83) at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57) at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51) at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80) at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74) at org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61) at org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsIterator(ExpressionSupport.java:94) at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:89) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53) at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:583) at org.kohsuke.stapler.Stapler.service(Stapler.java:214) at javax.servlet.http.HttpServlet.service(HttpServlet.java:45) at winstone.ServletConfiguration.execute(ServletConfiguration.java:248) at winstone.RequestDispatcher.forward(RequestDispatcher.java:333) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:36) at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:103) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:32) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:124) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:64) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:227) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@567bd2f9 IM-Reconnector-Thread "IM-Reconnector-Thread" Id=21 Group=main WAITING on java.util.concurrent.Semaphore$NonfairSync@4c423981 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.Semaphore$NonfairSync@4c423981 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) at java.util.concurrent.Semaphore.acquire(Semaphore.java:317) at hudson.plugins.im.IMConnectionProvider$ConnectorRunnable.run(IMConnectionProvider.java:160) at java.lang.Thread.run(Thread.java:722) javamelody jenkins "javamelody jenkins" Id=23 Group=main TIMED_WAITING on java.util.TaskQueue@5062da8 at java.lang.Object.wait(Native Method) - waiting on java.util.TaskQueue@5062da8 at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) Jenkins cron thread "Jenkins cron thread" Id=16 Group=main TIMED_WAITING on java.util.TaskQueue@fcd6b44 at java.lang.Object.wait(Native Method) - waiting on java.util.TaskQueue@fcd6b44 at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) Jenkins UDP 33848 monitoring thread "Jenkins UDP 33848 monitoring thread" Id=36 Group=main RUNNABLE (in native) at java.net.TwoStacksPlainDatagramSocketImpl.receive0(Native Method) - locked java.net.TwoStacksPlainDatagramSocketImpl@62ce7b37 at java.net.TwoStacksPlainDatagramSocketImpl.receive(TwoStacksPlainDatagramSocketImpl.java:90) - locked java.net.TwoStacksPlainDatagramSocketImpl@62ce7b37 at java.net.DatagramSocket.receive(DatagramSocket.java:786) - locked java.net.DatagramPacket@4945a2e3 - locked java.net.MulticastSocket@7ac90aa9 at hudson.UDPBroadcastThread.run(UDPBroadcastThread.java:82) JmDNS(FITBPWEB.local.).State.Timer "JmDNS(FITBPWEB.local.).State.Timer" Id=126 Group=main TIMED_WAITING on java.util.TaskQueue@187672e7 at java.lang.Object.wait(Native Method) - waiting on java.util.TaskQueue@187672e7 at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) JmDNS(FITBPWEB.local.).Timer "JmDNS(FITBPWEB.local.).Timer" Id=125 Group=main TIMED_WAITING on java.util.TaskQueue@871a83d at java.lang.Object.wait(Native Method) - waiting on java.util.TaskQueue@871a83d at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) jrobin jenkins "jrobin jenkins" Id=27 Group=main TIMED_WAITING on java.util.TaskQueue@57f4be7d at java.lang.Object.wait(Native Method) - waiting on java.util.TaskQueue@57f4be7d at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) LauncherControlThread[ControlPort=-1] "LauncherControlThread[ControlPort=-1]" Id=13 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at winstone.Launcher.run(Launcher.java:265) at java.lang.Thread.run(Thread.java:722) NioSocketAcceptor-1 "NioSocketAcceptor-1" Id=34 Group=main RUNNABLE (in native) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:295) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:277) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:158) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@3298e7c - locked java.util.Collections$UnmodifiableSet@43662b65 - locked sun.nio.ch.WindowsSelectorImpl@5ee2fc3e at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) at org.apache.mina.transport.socket.nio.NioSocketAcceptor.select(NioSocketAcceptor.java:238) at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:432) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@6ea937ab Ping thread for channel hudson.remoting.Channel@246e3870:serial-server-1 "Ping thread for channel hudson.remoting.Channel@246e3870:serial-server-1" Id=150 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) Ping thread for channel hudson.remoting.Channel@660da176:ubuntu-test-VM-3 "Ping thread for channel hudson.remoting.Channel@660da176:ubuntu-test-VM-3" Id=158 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) pool-1-svnkit-thread-1 "pool-1-svnkit-thread-1" Id=149 Group=main WAITING on java.util.concurrent.SynchronousQueue$TransferStack@3a898c5c at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@3a898c5c at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) pool-11-thread-1 "pool-11-thread-1" Id=159 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58937b62 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58937b62 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) pool-15-thread-1 "pool-15-thread-1" Id=118 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7527b2f7 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7527b2f7 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) pool-3-thread-12 "pool-3-thread-12" Id=222 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.jenkinsci.plugins.vSphereCloudLauncher.launch(vSphereCloudLauncher.java:197) at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:223) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@6de7a4d pool-3-thread-16 "pool-3-thread-16" Id=238 Group=main RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446) at hudson.Functions.getThreadInfos(Functions.java:1061) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92) at hudson.remoting.LocalChannel$1.call(LocalChannel.java:52) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@32a5f72a pool-4-thread-1 "pool-4-thread-1" Id=22 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@589f732b at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@589f732b at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) pool-5-thread-1 "pool-5-thread-1" Id=235 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@341ce778 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@341ce778 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) pool-7-thread-1 "pool-7-thread-1" Id=225 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ea58288 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ea58288 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) pool-7-thread-2 "pool-7-thread-2" Id=226 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ea58288 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ea58288 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) pool-7-thread-3 "pool-7-thread-3" Id=241 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ea58288 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ea58288 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) RequestHandlerThread[#14] "RequestHandlerThread[#14]" Id=206 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) RequestHandlerThread[#16] "RequestHandlerThread[#16]" Id=208 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) RequestHandlerThread[#20] "RequestHandlerThread[#20]" Id=212 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) RequestHandlerThread[#22] "RequestHandlerThread[#22]" Id=214 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) RequestHandlerThread[#23] "RequestHandlerThread[#23]" Id=243 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) RequestHandlerThread[#3] "RequestHandlerThread[#3]" Id=26 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) RequestHandlerThread[#6] "RequestHandlerThread[#6]" Id=32 Group=main RUNNABLE at java.util.zip.Deflater.deflateBytes(Native Method) at java.util.zip.Deflater.deflate(Deflater.java:427) - locked java.util.zip.ZStreamRef@924b115 at java.util.zip.Deflater.deflate(Deflater.java:350) at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:251) at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:211) at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:146) - locked java.util.zip.GZIPOutputStream@446b5a75 at org.kohsuke.stapler.compression.FilterServletOutputStream.write(FilterServletOutputStream.java:31) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:316) at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:149) - locked java.io.OutputStreamWriter@325c9870 at java.io.OutputStreamWriter.close(OutputStreamWriter.java:233) at java.io.PrintWriter.close(PrintWriter.java:339) - locked java.io.OutputStreamWriter@325c9870 at org.kohsuke.stapler.compression.CompressionFilter.reportException(CompressionFilter.java:78) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:227) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@51cbc1b5 SocketListener(FITBPWEB.local.) "SocketListener(FITBPWEB.local.)" Id=124 Group=main RUNNABLE (in native) at java.net.TwoStacksPlainDatagramSocketImpl.receive0(Native Method) - locked java.net.TwoStacksPlainDatagramSocketImpl@5ddc411f at java.net.TwoStacksPlainDatagramSocketImpl.receive(TwoStacksPlainDatagramSocketImpl.java:90) - locked java.net.TwoStacksPlainDatagramSocketImpl@5ddc411f at java.net.DatagramSocket.receive(DatagramSocket.java:786) - locked java.net.DatagramPacket@2f5cf3c8 - locked java.net.MulticastSocket@7f88e7c2 at javax.jmdns.impl.SocketListener.run(SocketListener.java:41) stderr copier for remote agent on serial-server-1 "stderr copier for remote agent on serial-server-1" Id=142 Group=main WAITING on java.lang.Object@3d0a76c2 at java.lang.Object.wait(Native Method) - waiting on java.lang.Object@3d0a76c2 at java.lang.Object.wait(Object.java:503) at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:213) at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:171) at hudson.util.StreamCopyThread.run(StreamCopyThread.java:60) stderr copier for remote agent on ubuntu-test-VM-3 "stderr copier for remote agent on ubuntu-test-VM-3" Id=145 Group=main WAITING on java.lang.Object@ae4f2f4 at java.lang.Object.wait(Native Method) - waiting on java.lang.Object@ae4f2f4 at java.lang.Object.wait(Object.java:503) at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:213) at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:171) at hudson.util.StreamCopyThread.run(StreamCopyThread.java:60) TCP slave agent listener port=0 "TCP slave agent listener port=0" Id=35 Group=main RUNNABLE (in native) at java.net.DualStackPlainSocketImpl.accept0(Native Method) at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:121) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:183) - locked java.net.SocksSocketImpl@24db1c39 at java.net.ServerSocket.implAccept(ServerSocket.java:522) at java.net.ServerSocket.accept(ServerSocket.java:490) at hudson.TcpSlaveAgentListener.run(TcpSlaveAgentListener.java:91) Thread-10 "Thread-10" Id=127 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@43f53914 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@43f53914 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Thread-11 "Thread-11" Id=140 Group=main WAITING on com.trilead.ssh2.channel.Channel@79fad9dc at java.lang.Object.wait(Native Method) - waiting on com.trilead.ssh2.channel.Channel@79fad9dc at java.lang.Object.wait(Object.java:503) at com.trilead.ssh2.channel.FifoBuffer.read(FifoBuffer.java:210) at com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:961) at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58) at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:70) at com.trilead.ssh2.StreamGobbler$GobblerThread.run(StreamGobbler.java:48) Thread-12 "Thread-12" Id=141 Group=main WAITING on com.trilead.ssh2.channel.Channel@79fad9dc at java.lang.Object.wait(Native Method) - waiting on com.trilead.ssh2.channel.Channel@79fad9dc at java.lang.Object.wait(Object.java:503) at com.trilead.ssh2.channel.FifoBuffer.read(FifoBuffer.java:210) at com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:961) at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58) at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:70) at com.trilead.ssh2.StreamGobbler$GobblerThread.run(StreamGobbler.java:48) Thread-13 "Thread-13" Id=143 Group=main RUNNABLE at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at com.trilead.ssh2.channel.FifoBuffer.read(FifoBuffer.java:210) at com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:961) at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58) at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:70) at com.trilead.ssh2.StreamGobbler$GobblerThread.run(StreamGobbler.java:48) Thread-14 "Thread-14" Id=144 Group=main RUNNABLE at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at com.trilead.ssh2.channel.FifoBuffer.read(FifoBuffer.java:210) at com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:961) at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58) at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:70) at com.trilead.ssh2.StreamGobbler$GobblerThread.run(StreamGobbler.java:48) Thread-21 "Thread-21" Id=187 Group=main RUNNABLE (in native) at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) - locked java.io.BufferedInputStream@20d1ed39 at com.sun.jndi.ldap.Connection.run(Connection.java:849) at java.lang.Thread.run(Thread.java:722) Thread-6 "Thread-6" Id=120 Group=main RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at com.trilead.ssh2.crypto.cipher.CipherInputStream.fill_buffer(CipherInputStream.java:41) at com.trilead.ssh2.crypto.cipher.CipherInputStream.internal_read(CipherInputStream.java:52) at com.trilead.ssh2.crypto.cipher.CipherInputStream.getBlock(CipherInputStream.java:79) at com.trilead.ssh2.crypto.cipher.CipherInputStream.read(CipherInputStream.java:108) at com.trilead.ssh2.transport.TransportConnection.receiveMessage(TransportConnection.java:232) at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:677) at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:475) at java.lang.Thread.run(Thread.java:722) Thread-7 "Thread-7" Id=121 Group=main RUNNABLE (in native) at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at com.trilead.ssh2.crypto.cipher.CipherInputStream.fill_buffer(CipherInputStream.java:41) at com.trilead.ssh2.crypto.cipher.CipherInputStream.internal_read(CipherInputStream.java:52) at com.trilead.ssh2.crypto.cipher.CipherInputStream.getBlock(CipherInputStream.java:79) at com.trilead.ssh2.crypto.cipher.CipherInputStream.read(CipherInputStream.java:108) at com.trilead.ssh2.transport.TransportConnection.receiveMessage(TransportConnection.java:232) at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:677) at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:475) at java.lang.Thread.run(Thread.java:722) ViewJob reload thread "ViewJob reload thread" Id=30 Group=main TIMED_WAITING on java.util.LinkedHashSet@5d1bfae4 at java.lang.Object.wait(Native Method) - waiting on java.util.LinkedHashSet@5d1bfae4 at hudson.model.ViewJob$ReloadThread.getNext(ViewJob.java:171) at hudson.model.ViewJob$ReloadThread.run(ViewJob.java:188) WinstoneHostConfigurationMgmt:default "WinstoneHostConfigurationMgmt:default" Id=11 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at winstone.HostConfiguration.run(HostConfiguration.java:174) at java.lang.Thread.run(Thread.java:722) Attach Listener "Attach Listener" Id=4 Group=system RUNNABLE Finalizer "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@22be5a1d at java.lang.Object.wait(Native Method) - waiting on java.lang.ref.ReferenceQueue$Lock@22be5a1d at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177) Java2D Disposer "Java2D Disposer" Id=9 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@579b8322 at java.lang.Object.wait(Native Method) - waiting on java.lang.ref.ReferenceQueue$Lock@579b8322 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) at sun.java2d.Disposer.run(Disposer.java:145) at java.lang.Thread.run(Thread.java:722) Reference Handler "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@7f0e37bc at java.lang.Object.wait(Native Method) - waiting on java.lang.ref.Reference$Lock@7f0e37bc at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) D0-Debian-Slave-2 N/A offline D2-Debian-Slave-1 N/A offline D0-Debian-Slave-1 N/A offline D2-Debian-Slave-2 N/A offline serial-server-1 Channel reader thread: channel "Channel reader thread: channel" Id=9 Group=main RUNNABLE (in native) at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:242) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read(BufferedInputStream.java:254) - locked java.io.BufferedInputStream@1348b49 at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2266) at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2559) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2569) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1315) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) main "main" Id=1 Group=main WAITING on hudson.remoting.Channel@6a21b2 at java.lang.Object.wait(Native Method) - waiting on hudson.remoting.Channel@6a21b2 at java.lang.Object.wait(Object.java:503) at hudson.remoting.Channel.join(Channel.java:800) at hudson.remoting.Launcher.main(Launcher.java:484) at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:430) at hudson.remoting.Launcher.run(Launcher.java:221) at hudson.remoting.Launcher.main(Launcher.java:180) Ping thread for channel hudson.remoting.Channel@6a21b2:channel "Ping thread for channel hudson.remoting.Channel@6a21b2:channel" Id=10 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) Pipe writer thread: channel "Pipe writer thread: channel" Id=12 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d66f88 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d66f88 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) pool-1-thread-10 "pool-1-thread-10" Id=22 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@17e21b3 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@17e21b3 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) pool-1-thread-9 "pool-1-thread-9" Id=21 Group=main RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446) at hudson.Functions.getThreadInfos(Functions.java:1061) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@1a970 Finalizer "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@a49259 at java.lang.Object.wait(Native Method) - waiting on java.lang.ref.ReferenceQueue$Lock@a49259 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177) Reference Handler "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@842f23 at java.lang.Object.wait(Native Method) - waiting on java.lang.ref.Reference$Lock@842f23 at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) Signal Dispatcher "Signal Dispatcher" Id=4 Group=system RUNNABLE D3-Debian-Slave-2 N/A offline ubuntu-test-VM-2 N/A offline ubuntu-test-VM-1 N/A offline D1-debian-FreshInstall-1 N/A offline D3-Debian-Slave-1 N/A offline ubuntu-test-VM-3 Channel reader thread: channel "Channel reader thread: channel" Id=8 Group=main RUNNABLE (in native) at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:236) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read(BufferedInputStream.java:254) - locked java.io.BufferedInputStream@12986ef at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2265) at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2558) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2568) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) main "main" Id=1 Group=main WAITING on hudson.remoting.Channel@18d107f at java.lang.Object.wait(Native Method) - waiting on hudson.remoting.Channel@18d107f at java.lang.Object.wait(Object.java:502) at hudson.remoting.Channel.join(Channel.java:800) at hudson.remoting.Launcher.main(Launcher.java:484) at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:430) at hudson.remoting.Launcher.run(Launcher.java:221) at hudson.remoting.Launcher.main(Launcher.java:180) Ping thread for channel hudson.remoting.Channel@18d107f:channel "Ping thread for channel hudson.remoting.Channel@18d107f:channel" Id=9 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) Pipe writer thread: channel "Pipe writer thread: channel" Id=11 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f6d156 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f6d156 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) pool-1-thread-3 "pool-1-thread-3" Id=14 Group=main RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:392) at hudson.Functions.getThreadInfos(Functions.java:1061) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@5e305f pool-1-thread-4 "pool-1-thread-4" Id=15 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7a7e40 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@7a7e40 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:453) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:352) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) Finalizer "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@159cf26 at java.lang.Object.wait(Native Method) - waiting on java.lang.ref.ReferenceQueue$Lock@159cf26 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177) Reference Handler "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@12efed9 at java.lang.Object.wait(Native Method) - waiting on java.lang.ref.Reference$Lock@12efed9 at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) Signal Dispatcher "Signal Dispatcher" Id=4 Group=system RUNNABLE
I saw this for the first time today, and saw it twice. Three things have changed over the past week that could possibly affect this. I'm wondering how many people who have this problem use an NFS share, for I've seen a lot of file contention issues since the change to NFS.
1. (Few weeks ago)$JENKINS_HOME is now an NFS share
2. (Last night)Updated to 1.516
3. (Last night)Modified github plugin to use a, already-defined, ThreadFactory (see below):
-private transient final SequentialExecutionQueue queue = new SequentialExecutionQueue(Executors.newCachedThreadPool());
+private transient final SequentialExecutionQueue queue = new SequentialExecutionQueue(Computer.threadPoolForRemoting);
@walterk82 indeed -Dorg.kohsuke.stapler.compression.CompressionFilter.disabled=true really only turns off the specialized error reporting, not the filter as a whole. For that you need to use -Dorg.kohsuke.stapler.jelly.DefaultScriptInvoker.compress=false from what I can tell.
Looking around for alternate implementations of GZIPOutputStream. JZlib looks most promising. Swapping this in for the native JRE implementation (zlib) should be trivial enough, but I need someone who is able to reproduce this bug and is willing to build Jenkins from sources (or give me a target Jenkins release version that I can build an alternate binary of) and compare the JZlib performance, so see if the hypothesis is valid that a bug in zlib is causing this issue.
Created http://repo.jenkins-ci.org/releases/org/jenkins-ci/main/jenkins-war/1.513.JENKINS-14362-jzlib/jenkins-war-1.513.JENKINS-14362-jzlib.war based on 1.513 if anyone wants to give it a whirl.
guykisel: could you please wrap your stacktrace inside a {quote}, so this ticket remains readable?
EDIT: Thanks a lot
Upgrading priority since this has been reported to necessitate Jenkins restarts to avoid excess load after a few days.
I've been using the "Monitoring" plugin to terminate the corresponding thread, instead of restarting Jenkins. The thread will be named something like "requestHandlerThread[#xyz]" and will most likely have the most cpu/user time by a long shot.
@jglick - Can you build a 1.516 WAR with the fix? If so,I can test that Friday morning. I don't know what specifically is triggering the issue in our instance, but it repeatedly occurs during a specific segment of our build cycle.
With the jenkins-war-1.516.JENKINS-14362-jzlib.war running, we did not see the hanging threads or CPUs @ 100%. Good sign. The next time I'll be able to test again will be Tuesday morning.
I'll also gladly install a 1.518 build w/ this fix as 1.518 has the css issue resolved @jglick - Let me know if you build one.
jenkins-war-1.516.JENKINS-14362-jzlib.war doesn't resolve this issue for me. CPU usage still peaks and Jenkins becomes locked out. Here's my stack trace:
2013-06-17 12:37:37 Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.45-b01-451 mixed mode): "Attach Listener" daemon prio=9 tid=7ff14f801000 nid=0x11c9b5000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE "RequestHandlerThread[#12]" daemon prio=5 tid=7ff14e806000 nid=0x11f57b000 waiting on condition [11f57a000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <77d8c4f40> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917) at java.lang.Thread.run(Thread.java:680) "Handling GET /adjuncts/cb752a56/lib/layout/menu_right_arrow.png : RequestHandlerThread[#11]" daemon prio=5 tid=7ff14e9ed000 nid=0x11f926000 runnable [11f922000] java.lang.Thread.State: RUNNABLE at java.io.File.<init>(File.java:318) at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:996) at sun.misc.URLClassPath$FileLoader.findResource(URLClassPath.java:966) at sun.misc.URLClassPath.findResource(URLClassPath.java:146) at java.net.URLClassLoader$2.run(URLClassLoader.java:385) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findResource(URLClassLoader.java:382) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at jenkins.ClassLoaderReflectionToolkit.findResource(ClassLoaderReflectionToolkit.java:52) at hudson.PluginManager$UberClassLoader.findResource(PluginManager.java:972) at java.lang.ClassLoader.getResource(ClassLoader.java:1002) at org.kohsuke.stapler.framework.adjunct.AdjunctManager.allowResourceToBeServed(AdjunctManager.java:198) at org.kohsuke.stapler.framework.adjunct.AdjunctManager.doDynamic(AdjunctManager.java:165) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:288) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:151) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:90) at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:363) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:583) at org.kohsuke.stapler.Stapler.service(Stapler.java:214) at javax.servlet.http.HttpServlet.service(HttpServlet.java:45) at winstone.ServletConfiguration.execute(ServletConfiguration.java:248) at winstone.RequestDispatcher.forward(RequestDispatcher.java:333) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:36) at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:103) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:32) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:124) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:64) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:227) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:680) "RequestHandlerThread[#10]" daemon prio=5 tid=7ff14ff93000 nid=0x11c2f1000 waiting on condition [11c2f0000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <77d8c4f40> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917) at java.lang.Thread.run(Thread.java:680) "RequestHandlerThread[#9]" daemon prio=5 tid=7ff14ff92800 nid=0x11c8b2000 waiting on condition [11c8b1000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <77d8c4f40> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917) at java.lang.Thread.run(Thread.java:680) "pool-10-thread-1" prio=5 tid=7ff14ff37800 nid=0x11f823000 waiting on condition [11f822000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <77e07af48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917) at java.lang.Thread.run(Thread.java:680) "JmDNS(Cornelius-Builder-Monkey-local.local.).State.Timer" prio=5 tid=7ff14cc94000 nid=0x12037d000 in Object.wait() [12037c000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <77e5be5c0> (a java.util.TaskQueue) at java.util.TimerThread.mainLoop(Timer.java:509) - locked <77e5be5c0> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "JmDNS(Cornelius-Builder-Monkey-local.local.).Timer" daemon prio=5 tid=7ff14cada800 nid=0x11cabd000 runnable [11cabc000] java.lang.Thread.State: RUNNABLE at javax.jmdns.impl.DNSCache.allValues(DNSCache.java:349) - locked <77e5be668> (a javax.jmdns.impl.DNSCache) at javax.jmdns.impl.JmDNSImpl.cleanCache(JmDNSImpl.java:1771) at javax.jmdns.impl.tasks.RecordReaper.run(RecordReaper.java:58) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) "pool-16-thread-1" daemon prio=5 tid=7ff14ce6e800 nid=0x120177000 waiting on condition [120176000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <77e686eb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.DelayQueue.take(DelayQueue.java:160) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917) at java.lang.Thread.run(Thread.java:680) "Executor #1 for master" prio=5 tid=7ff14ce6d800 nid=0x120074000 in Object.wait() [120073000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <77da4b940> (a hudson.model.Queue) at java.lang.Object.wait(Object.java:485) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) - locked <77da4b940> (a hudson.model.Queue) at hudson.model.Queue.pop(Queue.java:882) - locked <77da4b940> (a hudson.model.Queue) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) - locked <77da4b940> (a hudson.model.Queue) "Jenkins UDP 33848 monitoring thread" prio=5 tid=7ff14ce69800 nid=0x11fd6b000 runnable [11fd6a000] java.lang.Thread.State: RUNNABLE at java.net.PlainDatagramSocketImpl.receive0(Native Method) - locked <77e466810> (a java.net.PlainDatagramSocketImpl) at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145) - locked <77e466810> (a java.net.PlainDatagramSocketImpl) at java.net.DatagramSocket.receive(DatagramSocket.java:725) - locked <77e4ee470> (a java.net.DatagramPacket) - locked <77e6128e8> (a java.net.MulticastSocket) at hudson.UDPBroadcastThread.run(UDPBroadcastThread.java:82) "TCP slave agent listener port=0" prio=5 tid=7ff14ca42800 nid=0x11f478000 runnable [11f477000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408) - locked <77e64d808> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:462) at java.net.ServerSocket.accept(ServerSocket.java:430) at hudson.TcpSlaveAgentListener.run(TcpSlaveAgentListener.java:91) "NioSocketAcceptor-1" prio=5 tid=7ff14d0d8800 nid=0x11f2db000 runnable [11f2da000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:136) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:69) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <77e50f7d8> (a sun.nio.ch.Util$2) - locked <77e50f7c0> (a java.util.Collections$UnmodifiableSet) - locked <77e5f2d88> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) at org.apache.mina.transport.socket.nio.NioSocketAcceptor.select(NioSocketAcceptor.java:238) at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:432) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:680) "RequestHandlerThread[#5]" daemon prio=5 tid=7ff14d028000 nid=0x11f720000 waiting on condition [11f71f000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <77d8c4f40> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917) at java.lang.Thread.run(Thread.java:680) "jrobin " daemon prio=5 tid=7ff14d889800 nid=0x119f93000 in Object.wait() [119f92000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <77dcfd2b0> (a java.util.TaskQueue) at java.lang.Object.wait(Object.java:485) at java.util.TimerThread.mainLoop(Timer.java:483) - locked <77dcfd2b0> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "DestroyJavaVM" prio=5 tid=7ff14bdb8800 nid=0x1103ce000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE "LauncherControlThread[ControlPort=-1]" prio=5 tid=7ff14bdb7800 nid=0x11c6d3000 waiting on condition [11c6d2000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at winstone.Launcher.run(Launcher.java:265) at java.lang.Thread.run(Thread.java:680) "ConnectorThread:[http-8080]" daemon prio=5 tid=7ff14cea9000 nid=0x11c5d0000 runnable [11c5cf000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408) - locked <77d8b0530> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:462) at java.net.ServerSocket.accept(ServerSocket.java:430) at winstone.HttpListener.run(HttpListener.java:139) at java.lang.Thread.run(Thread.java:680) "WinstoneHostConfigurationMgmt:default" daemon prio=5 tid=7ff14cf30000 nid=0x11c4cd000 waiting on condition [11c4cc000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at winstone.HostConfiguration.run(HostConfiguration.java:174) at java.lang.Thread.run(Thread.java:680) "Java2D Disposer" daemon prio=10 tid=7ff14bd19800 nid=0x11ac46000 in Object.wait() [11ac45000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <77db5c678> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) - locked <77db5c678> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at sun.java2d.Disposer.run(Disposer.java:127) at java.lang.Thread.run(Thread.java:680) "AWT-AppKit" daemon prio=5 tid=7ff14ce3d000 nid=0x7fff74186180 runnable [00000000] java.lang.Thread.State: RUNNABLE "Low Memory Detector" daemon prio=5 tid=7ff14c87f000 nid=0x119b64000 runnable [00000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread1" daemon prio=9 tid=7ff14c87e000 nid=0x119a61000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread0" daemon prio=9 tid=7ff14c87d800 nid=0x11995e000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" daemon prio=9 tid=7ff14c87c800 nid=0x11985b000 runnable [00000000] java.lang.Thread.State: RUNNABLE "Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7ff14c87b800 nid=0x119758000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE "Finalizer" daemon prio=8 tid=7ff14b840800 nid=0x1194ca000 in Object.wait() [1194c9000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <77d875d58> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) - locked <77d875d58> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:171) "Reference Handler" daemon prio=10 tid=7ff14b83f800 nid=0x1193c7000 in Object.wait() [1193c6000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <77d86b5a8> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) - locked <77d86b5a8> (a java.lang.ref.Reference$Lock) "VM Thread" prio=9 tid=7ff14b83d000 nid=0x1192c4000 runnable "Gang worker#0 (Parallel GC Threads)" prio=9 tid=7ff14c801800 nid=0x1137d2000 runnable "Gang worker#1 (Parallel GC Threads)" prio=9 tid=7ff14c802000 nid=0x1138d5000 runnable "Concurrent Mark-Sweep GC Thread" prio=9 tid=7ff14c84d000 nid=0x118f3e000 runnable "VM Periodic Task Thread" prio=10 tid=7ff14c888800 nid=0x119c67000 waiting on condition "Exception Catcher Thread" prio=10 tid=7ff14c801000 nid=0x1105f9000 runnable JNI global references: 1556
@joncrooke your issue looks unrelated and should be filed separately. Use your browser’s debug view to see if /adjuncts/cb752a56/lib/layout/menu_right_arrow.png is being served with an appropriate cache header; ought to be loaded once as a 200 and then get 304 NOT MODIFIED thereafter.
Integrated in jenkins_main_trunk #2629
[FIXED JENKINS-17713 JENKINS-14362] (Revision bda95c67a6780ff0e6350f80d6f69cf0f61e737c)
Result = SUCCESS
kohsuke : bda95c67a6780ff0e6350f80d6f69cf0f61e737c
Files :
- core/pom.xml
- changelog.html
Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
changelog.html
core/pom.xml
http://jenkins-ci.org/commit/jenkins/bda95c67a6780ff0e6350f80d6f69cf0f61e737c
Log:
[FIXED JENKINS-17713 JENKINS-14362]
Integrated the new version of Stapler that fixes them.
Compare: https://github.com/jenkinsci/jenkins/compare/59d4c6a4c25a...bda95c67a678
Code changed in jenkins
User: Jesse Glick
Path:
cli/pom.xml
core/pom.xml
core/src/main/java/hudson/FilePath.java
core/src/main/java/hudson/console/AnnotatedLargeText.java
core/src/main/java/hudson/console/ConsoleNote.java
core/src/main/java/hudson/model/Run.java
core/src/main/java/hudson/model/UsageStatistics.java
core/src/main/java/hudson/security/HudsonAuthenticationEntryPoint.java
core/src/main/java/hudson/util/CompressedFile.java
maven-plugin/pom.xml
plugins/pom.xml
pom.xml
test/pom.xml
ui-samples-plugin/pom.xml
war/pom.xml
http://jenkins-ci.org/commit/jenkins/3701a327a75d35e5d0e5795341cc55bb123a2007
Log:
JENKINS-14362 Switch to jzlib for GZIP streaming.
Code changed in jenkins
User: Jesse Glick
Path:
changelog.html
cli/pom.xml
cli/src/main/java/hudson/cli/CLI.java
core/pom.xml
core/src/main/java/hudson/AboutJenkins.java
core/src/main/java/hudson/ClassicPluginStrategy.java
core/src/main/java/hudson/ExtensionFinder.java
core/src/main/java/hudson/FilePath.java
core/src/main/java/hudson/Functions.java
core/src/main/java/hudson/PluginManager.java
core/src/main/java/hudson/Util.java
core/src/main/java/hudson/WebAppMain.java
core/src/main/java/hudson/cli/ClientAuthenticationCache.java
core/src/main/java/hudson/cli/CommandDuringBuild.java
core/src/main/java/hudson/cli/SetBuildParameterCommand.java
core/src/main/java/hudson/init/impl/GroovyInitScript.java
core/src/main/java/hudson/logging/LogRecorder.java
core/src/main/java/hudson/markup/MyspacePolicy.java
core/src/main/java/hudson/matrix/Combination.java
core/src/main/java/hudson/matrix/DefaultMatrixExecutionStrategyImpl.java
core/src/main/java/hudson/matrix/MatrixBuild.java
core/src/main/java/hudson/matrix/MatrixConfiguration.java
core/src/main/java/hudson/matrix/MatrixProject.java
core/src/main/java/hudson/model/AbstractBuild.java
core/src/main/java/hudson/model/AbstractItem.java
core/src/main/java/hudson/model/AbstractProject.java
core/src/main/java/hudson/model/Cause.java
core/src/main/java/hudson/model/CauseAction.java
core/src/main/java/hudson/model/Computer.java
core/src/main/java/hudson/model/ComputerSet.java
core/src/main/java/hudson/model/Descriptor.java
core/src/main/java/hudson/model/DisplayNameListener.java
core/src/main/java/hudson/model/Executor.java
core/src/main/java/hudson/model/FileParameterDefinition.java
core/src/main/java/hudson/model/FileParameterValue.java
core/src/main/java/hudson/model/Fingerprint.java
core/src/main/java/hudson/model/ItemGroup.java
core/src/main/java/hudson/model/JDK.java
core/src/main/java/hudson/model/Job.java
core/src/main/java/hudson/model/ListView.java
core/src/main/java/hudson/model/Node.java
core/src/main/java/hudson/model/ParametersAction.java
core/src/main/java/hudson/model/ParametersDefinitionProperty.java
core/src/main/java/hudson/model/Queue.java
core/src/main/java/hudson/model/Run.java
core/src/main/java/hudson/model/RunAction.java
core/src/main/java/hudson/model/RunMap.java
core/src/main/java/hudson/model/RunParameterDefinition.java
core/src/main/java/hudson/model/RunParameterValue.java
core/src/main/java/hudson/model/StreamBuildListener.java
core/src/main/java/hudson/model/UpdateCenter.java
core/src/main/java/hudson/model/View.java
core/src/main/java/hudson/model/queue/CauseOfBlockage.java
core/src/main/java/hudson/model/queue/WorkUnit.java
core/src/main/java/hudson/os/PosixAPI.java
core/src/main/java/hudson/os/PosixException.java
core/src/main/java/hudson/scheduler/CronTab.java
core/src/main/java/hudson/search/Search.java
core/src/main/java/hudson/search/SearchItem.java
core/src/main/java/hudson/security/BasicAuthenticationFilter.java
core/src/main/java/hudson/security/FederatedLoginService.java
core/src/main/java/hudson/security/HudsonPrivateSecurityRealm.java
core/src/main/java/hudson/security/Permission.java
core/src/main/java/hudson/slaves/SlaveComputer.java
core/src/main/java/hudson/tasks/Fingerprinter.java
core/src/main/java/hudson/tasks/Maven.java
core/src/main/java/hudson/tasks/UserAvatarResolver.java
core/src/main/java/hudson/tasks/junit/CaseResult.java
core/src/main/java/hudson/tasks/junit/ClassResult.java
core/src/main/java/hudson/tasks/junit/JUnitParser.java
core/src/main/java/hudson/tasks/junit/PackageResult.java
core/src/main/java/hudson/tasks/junit/TestNameTransformer.java
core/src/main/java/hudson/tasks/test/TestResult.java
core/src/main/java/hudson/tools/CommandInstaller.java
core/src/main/java/hudson/tools/ZipExtractionInstaller.java
core/src/main/java/hudson/util/CopyOnWriteMap.java
core/src/main/java/hudson/util/FormFieldValidator.java
core/src/main/java/hudson/util/FormValidation.java
core/src/main/java/hudson/util/IOUtils.java
core/src/main/java/hudson/util/PersistedList.java
core/src/main/java/hudson/util/RunList.java
core/src/main/java/hudson/util/XStream2.java
core/src/main/java/hudson/util/jna/GNUCLibrary.java
core/src/main/java/hudson/widgets/Widget.java
core/src/main/java/jenkins/model/Jenkins.java
core/src/main/java/jenkins/model/PeepholePermalink.java
core/src/main/java/jenkins/model/RunAction2.java
core/src/main/java/jenkins/model/lazy/AbstractLazyLoadRunMap.java
core/src/main/java/jenkins/model/lazy/Boundary.java
core/src/main/java/jenkins/model/lazy/BuildReferenceMapAdapter.java
core/src/main/java/jenkins/mvn/GlobalMavenConfig.java
core/src/main/java/jenkins/security/ConfidentialStore.java
core/src/main/java/jenkins/security/RekeySecretAdminMonitor.java
core/src/main/java/jenkins/util/JSONSignatureValidator.java
core/src/main/java/jenkins/util/ProgressiveRendering.java
core/src/main/java/jenkins/widgets/BuildQueueWidget.java
core/src/main/java/jenkins/widgets/ExecutorsWidget.java
core/src/main/resources/dsld/IntelliJ.gdsl
core/src/main/resources/hudson/AboutJenkins/index.jelly
core/src/main/resources/hudson/AboutJenkins/index.properties
core/src/main/resources/hudson/AboutJenkins/index_fr.properties
core/src/main/resources/hudson/Messages.properties
core/src/main/resources/hudson/Messages_de.properties
core/src/main/resources/hudson/Messages_fr.properties
core/src/main/resources/hudson/PluginManager/advanced.jelly
core/src/main/resources/hudson/PluginManager/installed.jelly
core/src/main/resources/hudson/cli/Messages.properties
core/src/main/resources/hudson/logging/LogRecorder/configure.jelly
core/src/main/resources/hudson/logging/LogRecorder/index.jelly
core/src/main/resources/hudson/matrix/MatrixBuild/ajaxMatrix_fr.properties
core/src/main/resources/hudson/matrix/MatrixProject/ajaxMatrix_fr.properties
core/src/main/resources/hudson/matrix/MatrixProject/index.jelly
core/src/main/resources/hudson/matrix/Messages.properties
core/src/main/resources/hudson/model/Cause/UpstreamCause/description_pl.properties
core/src/main/resources/hudson/model/Cause/UserCause/description_pl.properties
core/src/main/resources/hudson/model/Cause/UserIdCause/description_pl.properties
core/src/main/resources/hudson/model/FileParameterDefinition/config_fr.properties
core/src/main/resources/hudson/model/FileParameterValue/value_fr.properties
core/src/main/resources/hudson/model/FreeStyleProject/newJobDetail_fr.properties
core/src/main/resources/hudson/model/Job/_api.jelly
core/src/main/resources/hudson/model/Job/buildTimeTrend_fr.properties
core/src/main/resources/hudson/model/Job/configure_fr.properties
core/src/main/resources/hudson/model/Job/index.jelly
core/src/main/resources/hudson/model/Job/index_fr.properties
core/src/main/resources/hudson/model/Messages.properties
core/src/main/resources/hudson/model/Messages_de.properties
core/src/main/resources/hudson/model/Messages_fr.properties
core/src/main/resources/hudson/model/ParametersAction/index_fr.properties
core/src/main/resources/hudson/model/ParametersDefinitionProperty/index_fr.properties
core/src/main/resources/hudson/model/PasswordParameterDefinition/config_fr.properties
core/src/main/resources/hudson/model/RunParameterDefinition/config.jelly
core/src/main/resources/hudson/model/RunParameterDefinition/index.jelly
core/src/main/resources/hudson/model/View/AsynchPeople/index.jelly
core/src/main/resources/hudson/model/View/AsynchPeople/index.properties
core/src/main/resources/hudson/model/View/sidepanel.jelly
core/src/main/resources/hudson/model/labels/LabelAtom/configure_fr.properties
core/src/main/resources/hudson/slaves/JNLPLauncher/main.jelly
core/src/main/resources/hudson/slaves/Messages_fr.properties
core/src/main/resources/hudson/slaves/OfflineCause/LaunchFailed/cause_fr.properties
core/src/main/resources/hudson/slaves/SlaveComputer/log.jelly
core/src/main/resources/hudson/slaves/SlaveComputer/statistics.groovy
core/src/main/resources/hudson/tasks/BuildTrigger/config.jelly
core/src/main/resources/hudson/tasks/Fingerprinter/FingerprintAction/index.jelly
core/src/main/resources/hudson/tasks/Maven/config.jelly
core/src/main/resources/hudson/tasks/junit/CaseResult/index.jelly
core/src/main/resources/hudson/tasks/junit/ClassResult/body.jelly
core/src/main/resources/hudson/tasks/test/AbstractTestResultAction/summary.jelly
core/src/main/resources/hudson/tasks/test/MetaTabulatedResult/body.jelly
core/src/main/resources/hudson/tools/JDKInstaller/config.jelly
core/src/main/resources/hudson/tools/label.jelly
core/src/main/resources/hudson/views/JobColumn/column.jelly
core/src/main/resources/jenkins/diagnostics/SecurityIsOffMonitor/message_fr.properties
core/src/main/resources/jenkins/management/Messages_pl.properties
core/src/main/resources/jenkins/management/PluginsLink/info_pl.properties
core/src/main/resources/jenkins/model/CauseOfInterruption/UserInterruption/summary_pl.properties
core/src/main/resources/jenkins/model/Jenkins/legend_fr.properties
core/src/main/resources/jenkins/model/Jenkins/manage.jelly
core/src/main/resources/jenkins/mvn/GlobalMavenConfig/config.groovy
core/src/main/resources/jenkins/widgets/BuildQueueWidget/index.groovy
core/src/main/resources/jenkins/widgets/ExecutorsWidget/index.groovy
core/src/main/resources/lib/form/apply_fr.properties
core/src/main/resources/lib/form/booleanRadio_fr.properties
core/src/main/resources/lib/form/breadcrumb-config-outline_fr.properties
core/src/main/resources/lib/form/dropdownDescriptorSelector.jelly
core/src/main/resources/lib/form/hetero-list_fr.properties
core/src/main/resources/lib/form/number.jelly
core/src/main/resources/lib/form/textarea.jelly
core/src/main/resources/lib/hudson/buildListTable.jelly
core/src/main/resources/lib/hudson/executors.jelly
core/src/main/resources/lib/hudson/jobLink.jelly
core/src/main/resources/lib/hudson/queue.jelly
core/src/main/resources/lib/layout/breadcrumbBar_fr.properties
core/src/main/resources/lib/layout/breadcrumbs.js
core/src/main/resources/lib/layout/breakable.jelly
core/src/main/resources/lib/layout/copyButton/copyButton.js
core/src/main/resources/lib/layout/layout.jelly
core/src/main/resources/lib/layout/layout_fr.properties
core/src/main/resources/lib/layout/progressiveRendering.jelly
core/src/main/resources/lib/layout/progressiveRendering/progressiveRendering.js
core/src/main/resources/lib/layout/progressiveRendering_fr.properties
core/src/main/resources/lib/layout/task_fr.properties
core/src/main/resources/windows-service/jenkins-slave.xml
core/src/main/resources/windows-service/jenkins.xml
core/src/test/java/hudson/FilePathTest.java
core/src/test/java/hudson/FunctionsTest.java
core/src/test/java/hudson/logging/LogRecorderTest.java
core/src/test/java/hudson/markup/MyspacePolicyTest.java
core/src/test/java/hudson/matrix/CombinationFilterUsingBuildParamsTest.java
core/src/test/java/hudson/model/FingerprintTest.java
core/src/test/java/hudson/model/ListViewTest.java
core/src/test/java/hudson/model/StubJob.java
core/src/test/java/hudson/tasks/junit/SuiteResultTest.java
core/src/test/java/hudson/util/CopyOnWriteMapTest.java
core/src/test/java/jenkins/model/PeepholePermalinkTest.java
core/src/test/java/jenkins/model/lazy/AbstractLazyLoadRunMapTest.java
core/src/test/resources/hudson/model/fingerprint.xml
debian/debian/changelog
maven-plugin/pom.xml
maven-plugin/src/main/java/hudson/maven/AbstractMavenProcessFactory.java
maven-plugin/src/main/java/hudson/maven/AggregatingClassLoader.java
maven-plugin/src/main/java/hudson/maven/Maven3Builder.java
maven-plugin/src/main/java/hudson/maven/Maven3ProcessFactory.java
maven-plugin/src/main/java/hudson/maven/MavenBuildInformation.java
maven-plugin/src/main/java/hudson/maven/MavenEmbedderRequest.java
maven-plugin/src/main/java/hudson/maven/MavenModuleSet.java
maven-plugin/src/main/java/hudson/maven/MavenModuleSetBuild.java
maven-plugin/src/main/java/hudson/maven/MavenProbeAction.java
maven-plugin/src/main/java/hudson/maven/MavenProcessFactory.java
maven-plugin/src/main/java/hudson/maven/MavenUtil.java
maven-plugin/src/main/java/hudson/maven/PlexusModuleContributor.java
maven-plugin/src/main/java/hudson/maven/reporters/MavenAbstractArtifactRecord.java
maven-plugin/src/main/java/hudson/maven/reporters/MavenFingerprinter.java
maven-plugin/src/main/java/hudson/maven/reporters/TestMojo.java
maven-plugin/src/main/resources/hudson/maven/MavenBuild/executedMojos_fr.properties
maven-plugin/src/main/resources/hudson/maven/MavenModuleSetBuild/main_pl.properties
maven-plugin/src/main/resources/hudson/maven/Messages.properties
maven-plugin/src/main/resources/hudson/maven/Messages_es.properties
maven-plugin/src/main/resources/hudson/maven/Messages_zh_TW.properties
maven-plugin/src/main/resources/hudson/maven/reporters/MavenAbstractArtifactRecord/index_fr.properties
maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java
maven-plugin/src/test/java/hudson/maven/reporters/TestMojoTest.java
plugins/pom.xml
pom.xml
test/pom.xml
test/src/main/java/org/jvnet/hudson/test/HudsonTestCase.java
test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java
test/src/main/java/org/jvnet/hudson/test/MemoryAssert.java
test/src/main/java/org/jvnet/hudson/test/MockFolder.java
test/src/main/java/org/jvnet/hudson/test/RunLoadCounter.java
test/src/main/resources/org/jvnet/hudson/test/ComputerConnectorTester/configure_fr.properties
test/src/test/groovy/hudson/cli/BuildCommandTest.groovy
test/src/test/groovy/hudson/cli/SetBuildParameterCommandTest.groovy
test/src/test/java/hudson/PluginManagerTest.java
test/src/test/java/hudson/cli/CopyJobCommandTest.java
test/src/test/java/hudson/init/impl/GroovyInitScriptTest.java
test/src/test/java/hudson/logging/LogRecorderManagerTest.java
test/src/test/java/hudson/matrix/MatrixTest.java
test/src/test/java/hudson/maven/MavenModuleSetTest.java
test/src/test/java/hudson/maven/MavenMultiModuleLogRotatorCleanArtifactsTest.java
test/src/test/java/hudson/maven/MavenMultiModuleTest.java
test/src/test/java/hudson/maven/MavenProjectTest.java
test/src/test/java/hudson/maven/PlexusModuleContributorTest.java
test/src/test/java/hudson/model/AbstractProjectTest.java
test/src/test/java/hudson/model/DisplayNameTest.java
test/src/test/java/hudson/model/JobTest.java
test/src/test/java/hudson/model/ListViewTest.java
test/src/test/java/hudson/model/ParametersTest.java
test/src/test/java/hudson/model/RunParameterDefinitionTest.java
test/src/test/java/hudson/model/RunTest.java
test/src/test/java/hudson/tasks/FingerprinterTest.java
test/src/test/java/hudson/tasks/MavenTest.java
test/src/test/java/hudson/tasks/junit/JUnitResultArchiverTest.java
test/src/test/java/hudson/tasks/junit/SuiteResultTest.java
test/src/test/java/hudson/tasks/junit/TestNameTransformerTest.java
test/src/test/java/hudson/tasks/junit/TestResultPublishingTest.java
test/src/test/java/jenkins/model/JenkinsTest.java
test/src/test/java/jenkins/security/RekeySecretAdminMonitorTest.java
test/src/test/java/org/jvnet/hudson/main/UseRecipesWithJenkinsRuleTest.java
test/src/test/resources/hudson/init/impl/GroovyInitScriptTest/errorsHandled.zip
test/src/test/resources/hudson/maven/custom-plexus-component.pom
test/src/test/resources/hudson/model/JobTest/configDotXmlPermission.zip
test/src/test/resources/hudson/model/JobTest/getArtifactsUpTo.zip
test/src/test/resources/hudson/model/JobTest/readPermission.zip
test/src/test/resources/hudson/model/JobTest/testConfigDotXmlPermission.zip
test/src/test/resources/hudson/model/JobTest/testGetArtifactsUpTo.zip
test/src/test/resources/hudson/model/JobTest/testReadPermission.zip
test/src/test/resources/hudson/model/ListViewTest/nullJobNames/config.xml
test/src/test/resources/hudson/tasks/FingerprinterTest/actionSerialization.zip
ui-samples-plugin/pom.xml
ui-samples-plugin/src/main/resources/index.jelly
war/pom.xml
war/src/main/webapp/css/style.css
war/src/main/webapp/help/parameter/file.html
war/src/main/webapp/help/parameter/run-filter.html
war/src/main/webapp/help/parameter/run-project.html
war/src/main/webapp/help/project-config/description.html
war/src/main/webapp/help/run-config/description.html
war/src/main/webapp/help/system-config/systemMessage.html
war/src/main/webapp/help/user/description.html
war/src/main/webapp/help/view-config/description.html
war/src/main/webapp/scripts/hudson-behavior.js
http://jenkins-ci.org/commit/jenkins/dedb24ef034a04fea288eaa50089e8e774714e99
Log:
Merge branch 'master' into JENKINS-14362-jzlib
Code changed in jenkins
User: Jesse Glick
Path:
changelog.html
core/pom.xml
core/src/main/java/hudson/ExtensionListView.java
core/src/main/java/hudson/FilePath.java
core/src/main/java/hudson/Functions.java
core/src/main/java/hudson/Launcher.java
core/src/main/java/hudson/TcpSlaveAgentListener.java
core/src/main/java/hudson/Util.java
core/src/main/java/hudson/cli/CliProtocol.java
core/src/main/java/hudson/cli/ListJobsCommand.java
core/src/main/java/hudson/model/AbstractBuild.java
core/src/main/java/hudson/model/AbstractProject.java
core/src/main/java/hudson/model/Computer.java
core/src/main/java/hudson/model/ComputerSet.java
core/src/main/java/hudson/model/Executor.java
core/src/main/java/hudson/model/Fingerprint.java
core/src/main/java/hudson/model/Label.java
core/src/main/java/hudson/model/Node.java
core/src/main/java/hudson/model/ParametersDefinitionProperty.java
core/src/main/java/hudson/model/Queue.java
core/src/main/java/hudson/model/RestartListener.java
core/src/main/java/hudson/model/Run.java
core/src/main/java/hudson/model/Slave.java
core/src/main/java/hudson/model/View.java
core/src/main/java/hudson/model/listeners/RunListener.java
core/src/main/java/hudson/model/queue/AbstractQueueTask.java
core/src/main/java/hudson/model/queue/CauseOfBlockage.java
core/src/main/java/hudson/model/queue/MappingWorksheet.java
core/src/main/java/hudson/model/queue/QueueListener.java
core/src/main/java/hudson/model/queue/SubTask.java
core/src/main/java/hudson/model/queue/SubTaskContributor.java
core/src/main/java/hudson/model/queue/Tasks.java
core/src/main/java/hudson/node_monitors/AbstractDiskSpaceMonitor.java
core/src/main/java/hudson/node_monitors/AbstractNodeMonitorDescriptor.java
core/src/main/java/hudson/node_monitors/DiskSpaceMonitor.java
core/src/main/java/hudson/node_monitors/DiskSpaceMonitorDescriptor.java
core/src/main/java/hudson/node_monitors/NodeMonitor.java
core/src/main/java/hudson/node_monitors/TemporarySpaceMonitor.java
core/src/main/java/hudson/security/GlobalSecurityConfiguration.java
core/src/main/java/hudson/slaves/CloudProvisioningListener.java
core/src/main/java/hudson/slaves/NodePropertyDescriptor.java
core/src/main/java/hudson/slaves/NodeProvisioner.java
core/src/main/java/hudson/slaves/SlaveComputer.java
core/src/main/java/hudson/tasks/junit/CaseResult.java
core/src/main/java/hudson/tasks/junit/ClassResult.java
core/src/main/java/hudson/tasks/junit/PackageResult.java
core/src/main/java/hudson/util/CopyOnWriteList.java
core/src/main/java/hudson/util/PersistedList.java
core/src/main/java/jenkins/model/Jenkins.java
core/src/main/java/jenkins/security/QueueItemAuthenticator.java
core/src/main/java/jenkins/security/QueueItemAuthenticatorConfiguration.java
core/src/main/java/jenkins/security/QueueItemAuthenticatorDescriptor.java
core/src/main/java/jenkins/slaves/JnlpSlaveAgentProtocol.java
core/src/main/java/jenkins/util/TimeDuration.java
core/src/main/resources/hudson/matrix/Messages_de.properties
core/src/main/resources/hudson/model/AbstractProject/sidepanel.jelly
core/src/main/resources/hudson/model/Messages.properties
core/src/main/resources/hudson/model/Messages_de.properties
core/src/main/resources/hudson/security/Messages_de.properties
core/src/main/resources/hudson/views/BuildButtonColumn/column.jelly
core/src/main/resources/jenkins/model/GlobalNodePropertiesConfiguration/config.groovy
core/src/main/resources/jenkins/security/QueueItemAuthenticator/config.groovy
core/src/main/resources/jenkins/security/QueueItemAuthenticatorConfiguration/config.groovy
core/src/main/resources/lib/hudson/projectView.jelly
core/src/main/resources/lib/layout/confirmationLink.jelly
core/src/main/resources/lib/layout/task.jelly
core/src/test/java/hudson/LauncherTest.java
core/src/test/java/hudson/cli/ListJobsCommandTest.java
core/src/test/java/hudson/model/ViewTest.java
core/src/test/java/hudson/tasks/junit/ClassResultTest.java
debian/debian/changelog
maven-plugin/src/main/resources/hudson/maven/Messages.properties
maven-plugin/src/main/resources/hudson/maven/Messages_de.properties
maven-plugin/src/main/resources/hudson/maven/RedeployPublisher/config_de.properties
maven-plugin/src/main/resources/hudson/maven/reporters/MavenAbstractArtifactRecord/index_de.properties
pom.xml
test/src/main/java/org/jvnet/hudson/test/HudsonTestCase.java
test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java
test/src/main/java/org/jvnet/hudson/test/TestPluginManager.java
test/src/test/java/hudson/maven/MavenMultiModuleTest.java
test/src/test/java/hudson/model/QueueTest.java
test/src/test/java/hudson/model/ViewTest.java
http://jenkins-ci.org/commit/jenkins/b7e2d6127f77711681113a020681508d57622ea5
Log:
Merge branch 'master' into JENKINS-14362-jzlib
Conflicts:
core/pom.xml
Code changed in jenkins
User: Jesse Glick
Path:
core/src/main/java/hudson/FilePath.java
core/src/main/java/hudson/console/AnnotatedLargeText.java
core/src/main/java/hudson/console/ConsoleNote.java
core/src/main/java/hudson/model/Run.java
core/src/main/java/hudson/model/UsageStatistics.java
core/src/main/java/hudson/security/HudsonAuthenticationEntryPoint.java
core/src/main/java/hudson/util/CompressedFile.java
http://jenkins-ci.org/commit/jenkins/59ff3514532f9c7faef2bc6d841aba53a8107f0f
Log:
Merge pull request #804 from jglick/JENKINS-14362-jzlib
Use jzlib
Compare: https://github.com/jenkinsci/jenkins/compare/c1c6475bbb1a...59ff3514532f
Code changed in jenkins
User: Jesse Glick
Path:
changelog.html
http://jenkins-ci.org/commit/jenkins/fbba0cc01eeb1c9d7c412a6e7b76c588cd36c332
Log:
JENKINS-17713 [FIXED JENKINS-14362] These fixes were (mistakenly?) recorded for 1.518 rather than for trunk.
Integrated in jenkins_main_trunk #2633
JENKINS-14362 Switch to jzlib for GZIP streaming. (Revision 3701a327a75d35e5d0e5795341cc55bb123a2007)
JENKINS-17713 [FIXED JENKINS-14362] These fixes were (mistakenly?) recorded for 1.518 rather than for trunk. (Revision fbba0cc01eeb1c9d7c412a6e7b76c588cd36c332)
Result = SUCCESS
Jesse Glick : 3701a327a75d35e5d0e5795341cc55bb123a2007
Files :
- core/src/main/java/hudson/security/HudsonAuthenticationEntryPoint.java
- core/src/main/java/hudson/model/Run.java
- war/pom.xml
- core/src/main/java/hudson/util/CompressedFile.java
- ui-samples-plugin/pom.xml
- pom.xml
- plugins/pom.xml
- core/pom.xml
- cli/pom.xml
- core/src/main/java/hudson/FilePath.java
- core/src/main/java/hudson/console/AnnotatedLargeText.java
- core/src/main/java/hudson/model/UsageStatistics.java
- core/src/main/java/hudson/console/ConsoleNote.java
- maven-plugin/pom.xml
- test/pom.xml
Jesse Glick : fbba0cc01eeb1c9d7c412a6e7b76c588cd36c332
Files :
- changelog.html
http://repo.jenkins-ci.org/releases/org/jenkins-ci/main/jenkins-war/1.509.2.JENKINS-14362-jzlib/jenkins-war-1.509.2.JENKINS-14362-jzlib.war now also available if anyone using LTS thinks they are running into this.
Is there any chance that this might get backported besides just providing the WAR? I noticed that you already marked it as an lts-candidate but it didn't make it into 1.509.2 and wasn't rejected either. Our infra guys prefer installing via RPM.
@justinharringa: not much made it into 1.509.2, but yes this is a candidate for 1.509.3. The most helpful thing you can do to push it forward is to verify that the fix actually works. Since I have never personally observed the original bug I cannot do that, and we are loath to backport unverified fixes, though I have gotten anecdotal reports that it works.
Verifying the fix would mean confirming in a comment in this issue that (a) you consistently observed this problem (meaning heavy CPU load with thread dumps showing one or more threads stuck in Deflater.deflateBytes) in an earlier build, such as 1.519 or 1.509.2, but (b) you have not seen it recur since updating to a build with the purported fix (1.520 or 1.509.2.JENKINS-14362-jzlib).
That sounds good. I'll see what I can do. I thought I had reproduced the issue by opening the Wall Display plugin report in a few tabs. After that, I hit refresh rapidly on one of the tabs. Then, the CPU was pegged. Unfortunately, the CPU usage calmed down after a while and I didn't see a thread dump indicating the usage of Deflater.deflateBytes. This seemed to cause issues in one of our more heavily used instances but it appears it corrects itself in the sandbox environment so it could be a red herring. This doesn't appear to be an effective way to reproduce the issue.
I'm currently on 1.480.3. Once I find a way to reproduce, I'll try to reproduce on 1.509.2 as well and then move to your patched WAR in this sandbox environment.
the CPU was pegged
Remember that this by itself does not necessarily, or even probably, indicate that you are seeing JENKINS-14362; there are numerous other bugs that could cause inappropriate heavy CPU usage. You have to check the master thread dump for threads using Deflater.deflateBytes. In reports I have heard, the typical symptom is that Jenkins starts off running fine, but over the course of a week or so more and more threads get “stuck” in this method (seemingly at random), each consuming 100% of one CPU, so that eventually there are no free processors left to do anything.
Sorry, I should have clarified that I didn't see any threads using Deflater.deflateBytes in the my attempt to reproduce using the Wall Display plugin. I have seen that, in conjunction with consistently pegged CPUs, in my production instance on 1.480.3 (see thread-dump-prod.txt). I was hoping to properly indicate that the Wall Display method I was using wouldn't adequately reproduce the situation since the CPU usage fell.
Just to make sure I've got it clear, my impression is that the combination of pegged CPUs, which stay pegged, as well as a threadDump with threads using Deflater.deflateBytes is what we're looking for in order to call it a valid reproduction of the problem. Is that correct?
It seems like the closest I've gotten to reproducing this is by using dave catalan's suggested steps. Unfortunately, I was only able to get the thread dump with threads using Deflater.deflateBytes but the CPU usage only spiked to 100% for a few minutes. It seems that this would indicate that I haven't reproduced the issue that could be fixed. I attached 3 separate dumps for the reproduced items (thread-dump-reproduce-1.txt appears to be before or after the deflateBytes call).
It would seem that we don't have a deterministic way to reproduce this yet.
the combination of pegged CPUs, which stay pegged, as well as a threadDump with threads using Deflater.deflateBytes is what we're looking for in order to call it a valid reproduction of the problem. Is that correct?
Correct. Or at least the other reports I have heard indicate that the threads suffering the problem do not fix themselves spontaneously; but perhaps sometimes they do. Merely having some threads running deflateBytes does not definitely indicate this bug, because you might genuinely be trying to compress a lot of data at that moment.
Does this version is recommand for production use?
http://repo.jenkins-ci.org/releases/org/jenkins-ci/main/jenkins-war/1.518.JENKINS-14362-jzlib/jenkins-war-1.518.JENKINS-14362-jzlib.war
We are still use 1.512 now, and we want to upgrade, but I check the community rating, since 1.519, lots of people rollback to previous version.
If the patch for 1.518 is ready for production use? We want to use it first to fix the 100% CPU problem for now and wait a stable version in the future.
You are recommended to just use 1.520 or later. 1.518.JENKINS-14362-jzlib was offered for experimental purposes only and should no longer be used.
@Jesse: On Monday, we're going to upgrade from 1.480.3 to the 1.509.2.JENKINS-14362-jzlib.war on an instance that seems to hit this every few days or so. I'll be sure to report by the end of next week if we see this issue crop up again. I'm hopeful that this will help and I suspect reporting the result will be helpful to you.
We haven't seen this issue pop up on our large instance since we deployed 1.509.2.JENKINS-14362-jzlib.war on August 5th. We generally saw this problem once a week and we haven't seen it occur in over 2 weeks. It's certainly not a scientific measurement but hopefully it helps.
For the record, CompressionFilter was introduced as the fix for JENKINS-13625. Unclear to me whether using org.kohsuke.stapler.jelly.DefaultScriptInvoker.compress=false would revert that fix, or merely disable a performance optimization which relies on that fix.
No, this is not in 1.509.3. If there is interest I could produce an experimental build of 1.509.3 with this patch (plus the attempted fix of JENKINS-19473) applied.
I'd be interested in the proposed patch of 1.509.3. We've had good luck with your patched version of 1.509.2 for this issue.
http://repo.jenkins-ci.org/releases/org/jenkins-ci/main/jenkins-war/1.509.3.JENKINS-14362-jzlib/jenkins-war-1.509.3.JENKINS-14362-jzlib.war now available (also includes purported fix of JENKINS-19473). As usual beware that this build has received no testing whatsoever.
Code changed in jenkins
User: Jesse Glick
Path:
core/src/main/java/hudson/FilePath.java
core/src/main/java/hudson/console/AnnotatedLargeText.java
core/src/main/java/hudson/console/ConsoleNote.java
core/src/main/java/hudson/model/Run.java
core/src/main/java/hudson/model/UsageStatistics.java
core/src/main/java/hudson/security/HudsonAuthenticationEntryPoint.java
core/src/main/java/hudson/util/CompressedFile.java
http://jenkins-ci.org/commit/jenkins/a021fe5c2422c47d977d7937f4e01483f121b777
Log:
JENKINS-14362 Switch to jzlib for GZIP streaming.
(cherry picked from commit 3701a327a75d35e5d0e5795341cc55bb123a2007)
Conflicts:
cli/pom.xml
core/pom.xml
maven-plugin/pom.xml
plugins/pom.xml
pom.xml
test/pom.xml
ui-samples-plugin/pom.xml
war/pom.xml
Code changed in jenkins
User: Jesse Glick
Path:
pom.xml
http://jenkins-ci.org/commit/maven-plugin/fba17cd8497d9ddac9c55fead9c9a8da61059b23
Log:
JENKINS-14362 Switch to jzlib for GZIP streaming.
Originally-Committed-As: 3701a327a75d35e5d0e5795341cc55bb123a2007
Code changed in jenkins
User: Jesse Glick
Path:
pom.xml
src/main/java/hudson/maven/AbstractMavenProcessFactory.java
src/main/java/hudson/maven/AggregatingClassLoader.java
src/main/java/hudson/maven/Maven3Builder.java
src/main/java/hudson/maven/Maven3ProcessFactory.java
src/main/java/hudson/maven/MavenBuildInformation.java
src/main/java/hudson/maven/MavenEmbedderRequest.java
src/main/java/hudson/maven/MavenModuleSet.java
src/main/java/hudson/maven/MavenModuleSetBuild.java
src/main/java/hudson/maven/MavenProbeAction.java
src/main/java/hudson/maven/MavenProcessFactory.java
src/main/java/hudson/maven/MavenUtil.java
src/main/java/hudson/maven/PlexusModuleContributor.java
src/main/java/hudson/maven/reporters/MavenAbstractArtifactRecord.java
src/main/java/hudson/maven/reporters/MavenFingerprinter.java
src/main/java/hudson/maven/reporters/TestMojo.java
src/main/resources/hudson/maven/MavenBuild/executedMojos_fr.properties
src/main/resources/hudson/maven/MavenModuleSetBuild/main_pl.properties
src/main/resources/hudson/maven/Messages.properties
src/main/resources/hudson/maven/Messages_es.properties
src/main/resources/hudson/maven/Messages_zh_TW.properties
src/main/resources/hudson/maven/reporters/MavenAbstractArtifactRecord/index_fr.properties
src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java
src/test/java/hudson/maven/reporters/TestMojoTest.java
http://jenkins-ci.org/commit/maven-plugin/14863a0498e0e6c389f2779b9150e9280b363d3e
Log:
Merge branch 'master' into JENKINS-14362-jzlib
Originally-Committed-As: dedb24ef034a04fea288eaa50089e8e774714e99
Code changed in jenkins
User: Jesse Glick
Path:
src/main/resources/hudson/maven/Messages.properties
src/main/resources/hudson/maven/Messages_de.properties
src/main/resources/hudson/maven/RedeployPublisher/config_de.properties
src/main/resources/hudson/maven/reporters/MavenAbstractArtifactRecord/index_de.properties
http://jenkins-ci.org/commit/maven-plugin/d75a8c218b6ffd24fe2ddea8953ffaf22da63bc6
Log:
Merge branch 'master' into JENKINS-14362-jzlib
Conflicts:
core/pom.xml
Originally-Committed-As: b7e2d6127f77711681113a020681508d57622ea5
Hi There,
Just verifying. I see this bug has a 1.509.4 fixed label, but on Jenkins-CI.org, it is not listed in the change log until 1.520. Can someone please validate where the fix is? Thanks a lot.
It was first fixed in trunk in 1.520, then backported to 1.509.4. http://jenkins-ci.org/changelog-stable neglects to mention it, but that changelog is generally unreliable anyway.
FWIW, this appears to be the same bug as this JDK bug https://bugs.openjdk.java.net/browse/JDK-8193682
We are using jenkins 1.484 and are experiencing the same issue. The "-Dorg.kohsuke.stapler.compression.CompressionFilter.disabled=true" option did result in the threads no longer taking 100% CPU. However, this broke loading the CSS. Perhaps due to what Walter Kacynski noticed.