Jenkins ThreadDump from one of our hung executors:
Executor #0 for GJ-WTX64-S05V13 : executing jenkins_admin_component_1 #25147 / waiting for hudson.remoting.Channel@45b513d2:GJ-WTX64-S05V13
"Executor #0 for GJ-WTX64-S05V13 : executing jenkins_admin_component_1 #25147 / waiting for hudson.remoting.Channel@45b513d2:GJ-WTX64-S05V13" Id=204 Group=main TIMED_WAITING on hudson.remoting.UserRequest@33f1c9bd
at java.lang.Object.wait(Native Method)
- waiting on hudson.remoting.UserRequest@33f1c9bd
at hudson.remoting.Request.call(Request.java:147)
at hudson.remoting.Channel.call(Channel.java:796)
at hudson.FilePath.act(FilePath.java:1007)
at hudson.FilePath.act(FilePath.java:996)
at hudson.FilePath.deleteRecursive(FilePath.java:1198)
at hudson.plugins.perforce.PerforceSCM.checkout(PerforceSCM.java:934)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1278)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1720)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:404)
ThreadDump from that server:
Channel reader thread: channel
"Channel reader thread: channel" Id=2301 Group=main RUNNABLE (in native)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@40bda018
at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:86)
at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72)
at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103)
at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
main
"main" Id=1 Group=main WAITING on hudson.remoting.Engine@a3d1956
at java.lang.Object.wait(Native Method)
- waiting on hudson.remoting.Engine@a3d1956
at java.lang.Thread.join(Unknown Source)
at java.lang.Thread.join(Unknown Source)
at hudson.remoting.jnlp.Main.main(Main.java:150)
at hudson.remoting.jnlp.Main._main(Main.java:143)
at hudson.remoting.Launcher.run(Launcher.java:231)
at hudson.remoting.Launcher.main(Launcher.java:195)
Ping thread for channel hudson.remoting.Channel@662535d4:channel
"Ping thread for channel hudson.remoting.Channel@662535d4:channel" Id=2302 Group=main TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at hudson.remoting.PingThread.run(PingThread.java:91)
pool-1-thread-2253 for channel
"pool-1-thread-2253 for channel" Id=2298 Group=main RUNNABLE
at com.sun.jna.Native.initIDs(Native Method)
at com.sun.jna.Native.<clinit>(Native.java:148)
at hudson.util.jna.Kernel32Utils.load(Kernel32Utils.java:112)
at hudson.util.jna.Kernel32.<clinit>(Kernel32.java:37)
at hudson.util.jna.Kernel32Utils.getWin32FileAttributes(Kernel32Utils.java:77)
at hudson.util.jna.Kernel32Utils.isJunctionOrSymlink(Kernel32Utils.java:98)
at hudson.Util.isSymlink(Util.java:510)
at hudson.FilePath.deleteRecursive(FilePath.java:1221)
at hudson.FilePath.access$1000(FilePath.java:195)
at hudson.FilePath$14.invoke(FilePath.java:1201)
at hudson.FilePath$14.invoke(FilePath.java:1198)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2772)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:332)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:85)
at java.lang.Thread.run(Unknown Source)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@75d956c4
pool-1-thread-2256 for channel
"pool-1-thread-2256 for channel" Id=2309 Group=main RUNNABLE
at com.sun.jna.Pointer.<clinit>(Pointer.java:41)
at com.sun.jna.Structure.<clinit>(Structure.java:2078)
at org.jvnet.hudson.Windows.monitor(Windows.java:42)
at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:124)
at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:114)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:332)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:85)
at java.lang.Thread.run(Unknown Source)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@756a3221
pool-1-thread-2261 for channel
"pool-1-thread-2261 for channel" Id=2314 Group=main RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
at hudson.Functions.getThreadInfos(Functions.java:1220)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:98)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:95)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:332)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:85)
at java.lang.Thread.run(Unknown Source)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@50474da7
RemoteInvocationHandler 19
"RemoteInvocationHandler 19" Id=2300 Group=main TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@fec0bff
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.ReferenceQueue$Lock@fec0bff
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:564)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
at java.lang.Thread.run(Unknown Source)
Thread-1
"Thread-1" Id=13 Group=main TIMED_WAITING on hudson.remoting.Channel@662535d4
at java.lang.Object.wait(Native Method)
- waiting on hudson.remoting.Channel@662535d4
at hudson.remoting.Channel.join(Channel.java:948)
at hudson.remoting.Engine.run(Engine.java:316)
Attach Listener
"Attach Listener" Id=5 Group=system RUNNABLE
Finalizer
"Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@6641224e
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.ReferenceQueue$Lock@6641224e
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
Reference Handler
"Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@272dd0cb
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.Reference$Lock@272dd0cb
at java.lang.Object.wait(Unknown Source)
at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
Signal Dispatcher
"Signal Dispatcher" Id=4 Group=system RUNNABLE
I've got 12 systems hung like this, all Windows 10 X64.
Jenkins ThreadDump from one of our hung executors:
ThreadDump from that server:
I've got 12 systems hung like this, all Windows 10 X64.