-
Bug
-
Resolution: Unresolved
-
Critical
-
WinXP32/WinXP64
-
Powered by SuggestiMate
Hi folks,
Last week, I started to experience a weird issue with freezing/endless waiting of building jobs on slave machines. It first started with WinXP 32 bit slave, but I'm now experiencing the same problem also with WinXP 64 bit slave.
The symptoms are like this...
The job is started by SCM change or predefined time.
Both slave machines are correctly started (the last line, in their respective logs, says "Slave successfully connected and online").
Now the problem is, that the last line in console says "Building remotely on slave-name..." and there is just scrolling the progress circle. Nothing else happens. No GIT or SVN activity, no building, no testing, simply nothing and it stays like that until I kill the job manually.
Sometimes, the problem happens with just one machine, and the other one runs OK. But today it struck both machines.
Last week, when the problem started, I tried to update the Jenkins and both slaves with the most recent versions and latest plugins (except the "Git Client Plugin", because of a known issue with building history). Few hours all worked great, then the problem reappeared.
Please see the attached files with logs and thread dumps from both slave machines.
What struck me is the high number of threads started on both machines, which looks really weird? But I don't understand Jenkins that deep so I can't say for sure if it's normal or not?
Any idea what to try or how to solve this behavior?
Have a nice day,
Pavel
[JENKINS-19445] Jobs randomly stuck with "building remotely on slave-name" message
Hi again. One more piece to the puzzle.
This is what I saw in log while attempting to run a Jenkins job...
Sep 05, 2013 7:55:24 AM INFO hudson.slaves.RetentionStrategy$Demand check
Launching computer LB3D-Wxpp32sp3 as it has been in demand for 1 min 34 sec
Sep 05, 2013 7:55:24 AM INFO hudson.slaves.RetentionStrategy$Demand check
Launching computer LB3D-Wxpp64sp2 as it has been in demand for 1 min 34 sec
Sep 05, 2013 7:55:24 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog
[LB3D-Wxpp32sp3] Starting Virtual Machine...
Sep 05, 2013 7:55:24 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog
[LB3D-Wxpp64sp2] Starting Virtual Machine...
Sep 05, 2013 7:55:24 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog
[LB3D-Wxpp32sp3] Powering on VM
Sep 05, 2013 7:55:27 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog
[LB3D-Wxpp64sp2] Powering on VM
Sep 05, 2013 7:55:55 AM INFO hudson.model.AsyncPeriodicWork$1 run
Started PeriodicBackup
Sep 05, 2013 7:55:55 AM INFO hudson.model.AsyncPeriodicWork$1 run
Finished PeriodicBackup. 0 ms
Sep 05, 2013 7:56:35 AM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
Accepted connection #1 from /172.28.63.75:1034
Sep 05, 2013 7:56:54 AM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
Accepted connection #2 from /172.28.63.95:1047
Sep 05, 2013 7:56:55 AM INFO hudson.model.AsyncPeriodicWork$1 run
Started PeriodicBackup
Sep 05, 2013 7:56:55 AM INFO hudson.model.AsyncPeriodicWork$1 run
Finished PeriodicBackup. 0 ms
Sep 05, 2013 7:57:30 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor
Failed to monitor LB3D-Wxpp64sp2 for Free Swap Space
java.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:275)
at hudson.remoting.Request$1.get(Request.java:210)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282)
As you can see, two slaves are powered ON and successfully connected. But after few seconds, the second VM LB3D-Wxpp64sp2 experiences a problem with Free Swap Space monitoring and from now on, nothing else happens on that machine. The last line in slave console shows Building remotely on LB3D-Wxpp64sp2 in workspace etc... and the spinning wheel. The other one successfully updates SVN and GIT and continues with building.
Could this Free Swap Space monitoring error really be a root of the problem? Is there a way to disable this monitoring for good? And yes, as I mentioned in my previous comment, I tried to disable the monitoring in Jenkins Node configuration section but apparently, the monitoring is still active on a problematic VM?
OK, I'm now even more convinced that the free swap space monitoring has something to do with my problem. During today's night build, the error message appeared on both slaves and sure enough, building on both machines got stuck on "Building remotely on slave-name..." and spinning wheel.
Sep 06, 2013 1:28:15 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor
Failed to monitor LB3D-Wxpp64sp2 for Free Swap Space
java.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:275)
at hudson.remoting.Request$1.get(Request.java:210)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282)
Sep 06, 2013 2:08:39 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor
Failed to monitor LB3D-Wxpp32sp3 for Free Disk Space
java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
at hudson.remoting.Request$1.get(Request.java:278)
at hudson.remoting.Request$1.get(Request.java:210)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282)
Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
at hudson.remoting.Request.abort(Request.java:299)
at hudson.remoting.Channel.terminate(Channel.java:774)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.Command.readFrom(Command.java:92)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
Any idea how to kill this free swap space monitoring for good?
OK. One more addition to the problem. I reconfigured both slaves to keep them online as much as possible and so far (5 days), no problem with job start.
Unfortuantely, from the long perspective, this is not a very good solution. Slaves which are still online eat the server memory even if there is no need for them to stay online. Therefore, we would like to have them online only on demand and then shut them off after the build and test is finished. But this is currently impossible because of problem with randomly stuck slaves during their start (most probably caused by "free swap space monitoring). Anyone please?
Anyone please? The workaround with always enabled slaves works quite OK, but I hate to have the slaves permanently running. It's relative OK in case of just two slaves, but I'm planning to add few more slaves and then it would become a real problem.
One month later, once again updated Jenkins and all plugins, but nothing has changed. Jobs still randomly stuck with "building remotely on slave-name" message. Anyone please? Am I alone with such kind of issue?
It happened again today Any idea? Anyone please...
This is what I see in jenkins-slave.err.log:
Oct 31, 2013 7:41:16 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Oct 31, 2013 7:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among http://tcskjira:8095/
Oct 31, 2013 7:41:17 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to tcskjira:1695
Oct 31, 2013 7:41:17 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Oct 31, 2013 7:41:17 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
This is what I see in jenkins-slave.wrapper.log:
2013-10-31 07:41:09 - Starting C:\Program Files\Java\jre7\bin\java.exe -Xrs -jar "C:\Jenkins\slave.jar" -jnlpUrl http://tcskjira:8095/computer/LB3D-Wxpp64sp2/slave-agent.jnlp -secret 1ba0fb4096bc76c5ef1332c6368cd286a76074134a7091cd7603e93ec938b060
2013-10-31 07:41:09 - Started 1576
This is what I see in Jenkins Log:
Oct 31, 2013 7:40:34 AM INFO hudson.slaves.RetentionStrategy$Demand check
Launching computer LB3D-Wxpp64sp2 as it has been in demand for 39 sec
Oct 31, 2013 7:40:34 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog
[LB3D-Wxpp64sp2] Starting Virtual Machine...
Oct 31, 2013 7:40:35 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog
[LB3D-Wxpp64sp2] Powering on VM
Oct 31, 2013 7:40:51 AM INFO hudson.model.AsyncPeriodicWork$1 run
Started PeriodicBackup
Oct 31, 2013 7:40:51 AM INFO hudson.model.AsyncPeriodicWork$1 run
Finished PeriodicBackup. 0 ms
Oct 31, 2013 7:41:16 AM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
Accepted connection #40 from /172.28.63.95:1034
Oct 31, 2013 7:41:23 AM WARNING hudson.node_monitors.ResponseTimeMonitor$1 monitor
Making CAAV5R19x64 offline because it’s not responding
Oct 31, 2013 7:41:23 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog
[CAAV5R19x64] Already disconnecting on a separate thread
Oct 31, 2013 7:41:51 AM INFO hudson.model.AsyncPeriodicWork$1 run
Started PeriodicBackup
Oct 31, 2013 7:41:51 AM INFO hudson.model.AsyncPeriodicWork$1 run
Finished PeriodicBackup. 0 ms
Oct 31, 2013 7:41:52 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor
Failed to monitor LB3D-Wxpp64sp2 for Free Swap Space
java.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:275)
at hudson.remoting.Request$1.get(Request.java:210)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282)
Slave Log says this:
JNLP agent connected from /172.28.63.95
<===[JENKINS REMOTING CAPACITY]===>Slave.jar version: 2.32
This is a Windows slave
Slave successfully connected and online
followed by spinning wheel...
Slave Thread Dump:
Channel reader thread: channel
"Channel reader thread: channel" Id=11 Group=main RUNNABLE (in native)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@1c2ba649
at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.Command.readFrom(Command.java:92)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
main
"main" Id=1 Group=main WAITING on hudson.remoting.Engine@7a0136b0
at java.lang.Object.wait(Native Method)
- waiting on hudson.remoting.Engine@7a0136b0
at java.lang.Thread.join(Unknown Source)
at java.lang.Thread.join(Unknown Source)
at hudson.remoting.jnlp.Main.main(Main.java:137)
at hudson.remoting.jnlp.Main._main(Main.java:130)
at hudson.remoting.Launcher.run(Launcher.java:223)
at hudson.remoting.Launcher.main(Launcher.java:192)
Ping thread for channel hudson.remoting.Channel@28509953:channel
"Ping thread for channel hudson.remoting.Channel@28509953:channel" Id=13 Group=main TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at hudson.remoting.PingThread.run(PingThread.java:86)
pool-1-thread-17
"pool-1-thread-17" Id=29 Group=main RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
at hudson.Functions.getThreadInfos(Functions.java:1131)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Unknown Source)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@493df066
pool-1-thread-2
"pool-1-thread-2" Id=14 Group=main RUNNABLE
at com.sun.jna.Native.initIDs(Native Method)
at com.sun.jna.Native.<clinit>(Native.java:122)
at org.tmatesoft.svn.core.internal.util.jna.JNALibraryLoader.<clinit>(JNALibraryLoader.java:50)
at org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:93)
at org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196)
at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1869)
at org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57)
at hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:975)
at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:952)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:885)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:872)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2420)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Unknown Source)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@5133066c
pool-1-thread-5
"pool-1-thread-5" Id=17 Group=main RUNNABLE
at com.sun.jna.Structure.<clinit>(Structure.java:134)
at org.jvnet.hudson.Windows.monitor(Windows.java:40)
at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:110)
at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:100)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Unknown Source)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@786dee4c
Thread-1
"Thread-1" Id=10 Group=main WAITING on hudson.remoting.Channel@28509953
at java.lang.Object.wait(Native Method)
- waiting on hudson.remoting.Channel@28509953
at java.lang.Object.wait(Object.java:503)
at hudson.remoting.Channel.join(Channel.java:842)
at hudson.remoting.Engine.run(Engine.java:271)
Attach Listener
"Attach Listener" Id=4 Group=system RUNNABLE
Finalizer
"Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@75b21111
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.ReferenceQueue$Lock@75b21111
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
Reference Handler
"Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@7d7eaf74
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.Reference$Lock@7d7eaf74
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
System Thread Dump:
Thread Dump
AWT-Windows
"AWT-Windows" Id=614 Group=main RUNNABLE (in native)
at sun.awt.windows.WToolkit.eventLoop(Native Method)
at sun.awt.windows.WToolkit.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Channel reader thread: LB3D-Wxpp32sp3
"Channel reader thread: LB3D-Wxpp32sp3" Id=12151 Group=main RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@7a7ec776
at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.Command.readFrom(Command.java:92)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
Channel reader thread: LB3D-Wxpp64sp2
"Channel reader thread: LB3D-Wxpp64sp2" Id=12178 Group=main RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@764058b0
at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.Command.readFrom(Command.java:92)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
Channel reader thread: Slave-TestProcessBuild
"Channel reader thread: Slave-TestProcessBuild" Id=6006 Group=main RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@3563374a
at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.Command.readFrom(Command.java:92)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
Channel reader thread: Slave-TestProcessTest
"Channel reader thread: Slave-TestProcessTest" Id=6121 Group=main RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@374b5f62
at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.Command.readFrom(Command.java:92)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
DestroyJavaVM
"DestroyJavaVM" Id=19 Group=main RUNNABLE
Executor #0 for LB3D-Wxpp32sp3 : executing ipprotection_32_64 #171
"Executor #0 for LB3D-Wxpp32sp3 : executing ipprotection_32_64 #171" Id=10955 Group=main TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at hudson.matrix.DefaultMatrixExecutionStrategyImpl.waitForCompletion(DefaultMatrixExecutionStrategyImpl.java:290)
at hudson.matrix.DefaultMatrixExecutionStrategyImpl.run(DefaultMatrixExecutionStrategyImpl.java:141)
at hudson.matrix.MatrixBuild$MatrixBuildExecution.doRun(MatrixBuild.java:355)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:562)
at hudson.model.Run.execute(Run.java:1665)
at hudson.matrix.MatrixBuild.run(MatrixBuild.java:304)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:230)
Executor #0 for LB3D-Wxpp64sp2 : executing ipprotection_32_64 » LB3D-Wxpp64sp2 #171 / waiting for hudson.remoting.Channel@59f73f53:LB3D-Wxpp64sp2
"Executor #0 for LB3D-Wxpp64sp2 : executing ipprotection_32_64 » LB3D-Wxpp64sp2 #171 / waiting for hudson.remoting.Channel@59f73f53:LB3D-Wxpp64sp2" Id=11106 Group=main TIMED_WAITING on hudson.remoting.UserRequest@3690a4a8
at java.lang.Object.wait(Native Method)
- waiting on hudson.remoting.UserRequest@3690a4a8
at hudson.remoting.Request.call(Request.java:146)
at hudson.remoting.Channel.call(Channel.java:714)
at hudson.FilePath.act(FilePath.java:901)
at hudson.FilePath.act(FilePath.java:885)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:848)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:786)
at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:117)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1411)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:652)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:557)
at hudson.model.Run.execute(Run.java:1665)
at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:230)
Handling GET /threadDump : RequestHandlerThread595
"Handling GET /threadDump : RequestHandlerThread595" Id=12350 Group=main TIMED_WAITING on java.util.concurrent.FutureTask$Sync@4deadcb6
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.FutureTask$Sync@4deadcb6
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at hudson.remoting.LocalChannel$2.get(LocalChannel.java:81)
at jenkins.model.Jenkins.getAllThreadDumps(Jenkins.java:2919)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258)
at org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104)
at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
at org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61)
at org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsIterator(ExpressionSupport.java:94)
at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:89)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:727)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:841)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:630)
at org.kohsuke.stapler.Stapler.service(Stapler.java:224)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:208)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:181)
at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86)
at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:64)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@620c96ba
HashSessionScavenger-0
"HashSessionScavenger-0" Id=11 Group=main TIMED_WAITING on java.util.TaskQueue@4a7cb042
at java.lang.Object.wait(Native Method)
- waiting on java.util.TaskQueue@4a7cb042
at java.util.TimerThread.mainLoop(Unknown Source)
at java.util.TimerThread.run(Unknown Source)
javamelody
"javamelody" Id=27 Group=main TIMED_WAITING on java.util.TaskQueue@1b9a5f49
at java.lang.Object.wait(Native Method)
- waiting on java.util.TaskQueue@1b9a5f49
at java.util.TimerThread.mainLoop(Unknown Source)
at java.util.TimerThread.run(Unknown Source)
Jenkins cron thread
"Jenkins cron thread" Id=14 Group=main TIMED_WAITING on java.util.TaskQueue@258e37c9
at java.lang.Object.wait(Native Method)
- waiting on java.util.TaskQueue@258e37c9
at java.util.TimerThread.mainLoop(Unknown Source)
at java.util.TimerThread.run(Unknown Source)
Jenkins UDP 33848 monitoring thread
"Jenkins UDP 33848 monitoring thread" Id=31 Group=main RUNNABLE (in native)
at java.net.TwoStacksPlainDatagramSocketImpl.receive0(Native Method)
- locked java.net.TwoStacksPlainDatagramSocketImpl@1f036f93
at java.net.TwoStacksPlainDatagramSocketImpl.receive(Unknown Source) - locked java.net.TwoStacksPlainDatagramSocketImpl@1f036f93
at java.net.DatagramSocket.receive(Unknown Source) - locked java.net.DatagramPacket@7279ad5b
- locked java.net.MulticastSocket@12646c14
at hudson.UDPBroadcastThread.run(UDPBroadcastThread.java:82)
Jenkins-Remoting-Thread-1218
"Jenkins-Remoting-Thread-1218" Id=12433 Group=main RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
at hudson.Functions.getThreadInfos(Functions.java:1131)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
at hudson.remoting.LocalChannel$1.call(LocalChannel.java:52)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@1904332a
Jenkins-Remoting-Thread-1219
"Jenkins-Remoting-Thread-1219" Id=12434 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@2baacf98
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.SynchronousQueue$TransferStack@2baacf98
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Jenkins-Remoting-Thread-1220
"Jenkins-Remoting-Thread-1220" Id=12435 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@2baacf98
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.SynchronousQueue$TransferStack@2baacf98
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
JmDNS(tcskjira.local.).State.Timer
"JmDNS(tcskjira.local.).State.Timer" Id=48 Group=main TIMED_WAITING on java.util.TaskQueue@587c9b22
at java.lang.Object.wait(Native Method)
- waiting on java.util.TaskQueue@587c9b22
at java.util.TimerThread.mainLoop(Unknown Source)
at java.util.TimerThread.run(Unknown Source)
JmDNS(tcskjira.local.).Timer
"JmDNS(tcskjira.local.).Timer" Id=47 Group=main TIMED_WAITING on java.util.TaskQueue@5cad0290
at java.lang.Object.wait(Native Method)
- waiting on java.util.TaskQueue@5cad0290
at java.util.TimerThread.mainLoop(Unknown Source)
at java.util.TimerThread.run(Unknown Source)
jnr.ffi.util.ref.internal.Finalizer
"jnr.ffi.util.ref.internal.Finalizer" Id=5472 Group=main WAITING on java.lang.ref.ReferenceQueue$Lock@4c2f6a8
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.ReferenceQueue$Lock@4c2f6a8
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at jnr.ffi.util.ref.internal.Finalizer.run(Finalizer.java:128)
LauncherControlThread[ControlPort=-1]
"LauncherControlThread[ControlPort=-1]" Id=17 Group=main TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at winstone.Launcher.run(Launcher.java:245)
at java.lang.Thread.run(Unknown Source)
NioSocketAcceptor-1
"NioSocketAcceptor-1" Id=29 Group=main RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked sun.nio.ch.Util$2@5feec26b
- locked java.util.Collections$UnmodifiableSet@29e1ac62
- locked sun.nio.ch.WindowsSelectorImpl@4defca63
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at org.apache.mina.transport.socket.nio.NioSocketAcceptor.select(NioSocketAcceptor.java:238)
at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:432)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@29090dda
Ping thread for channel hudson.remoting.Channel@1675652c:Slave-TestProcessTest
"Ping thread for channel hudson.remoting.Channel@1675652c:Slave-TestProcessTest" Id=6122 Group=main TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at hudson.remoting.PingThread.run(PingThread.java:86)
Ping thread for channel hudson.remoting.Channel@50923d0b:Slave-TestProcessBuild
"Ping thread for channel hudson.remoting.Channel@50923d0b:Slave-TestProcessBuild" Id=6007 Group=main TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at hudson.remoting.PingThread.run(PingThread.java:86)
Ping thread for channel hudson.remoting.Channel@59f73f53:LB3D-Wxpp64sp2
"Ping thread for channel hudson.remoting.Channel@59f73f53:LB3D-Wxpp64sp2" Id=12179 Group=main TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at hudson.remoting.PingThread.run(PingThread.java:86)
Ping thread for channel hudson.remoting.Channel@5ade8f7a:LB3D-Wxpp32sp3
"Ping thread for channel hudson.remoting.Channel@5ade8f7a:LB3D-Wxpp32sp3" Id=12152 Group=main TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at hudson.remoting.PingThread.run(PingThread.java:86)
pool-1-svnkit-thread-1
"pool-1-svnkit-thread-1" Id=411 Group=main WAITING on java.util.concurrent.SynchronousQueue$TransferStack@1bcf9209
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.SynchronousQueue$TransferStack@1bcf9209
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
at java.util.concurrent.SynchronousQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
pool-1-svnkit-thread-2
"pool-1-svnkit-thread-2" Id=2322 Group=main WAITING on java.util.concurrent.SynchronousQueue$TransferStack@1bcf9209
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.SynchronousQueue$TransferStack@1bcf9209
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
at java.util.concurrent.SynchronousQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
pool-4-thread-1272
"pool-4-thread-1272" Id=12398 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
pool-4-thread-1273
"pool-4-thread-1273" Id=12399 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
pool-4-thread-1274
"pool-4-thread-1274" Id=12417 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
pool-4-thread-1275
"pool-4-thread-1275" Id=12418 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
pool-9-thread-1
"pool-9-thread-1" Id=44 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2ba02595
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2ba02595
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
RequestHandlerThread1 Selector0
"RequestHandlerThread1 Selector0" Id=15 Group=main RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked sun.nio.ch.Util$2@2fea3452
- locked java.util.Collections$UnmodifiableSet@e2f4d5c
- locked sun.nio.ch.WindowsSelectorImpl@7ed452e4
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@4da2d209
RequestHandlerThread2 Acceptor0 SelectChannelConnector@0.0.0.0:8095
"RequestHandlerThread2 Acceptor0 SelectChannelConnector@0.0.0.0:8095" Id=16 Group=main RUNNABLE (in native)
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)
- locked java.lang.Object@33e5bfc7
at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@5db00667
RequestHandlerThread603
"RequestHandlerThread603" Id=12388 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
RequestHandlerThread605
"RequestHandlerThread605" Id=12391 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
RequestHandlerThread607
"RequestHandlerThread607" Id=12427 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
RequestHandlerThread608
"RequestHandlerThread608" Id=12431 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
SocketListener(tcskjira.local.)
"SocketListener(tcskjira.local.)" Id=46 Group=main RUNNABLE (in native)
at java.net.TwoStacksPlainDatagramSocketImpl.receive0(Native Method)
- locked java.net.TwoStacksPlainDatagramSocketImpl@62836b78
at java.net.TwoStacksPlainDatagramSocketImpl.receive(Unknown Source) - locked java.net.TwoStacksPlainDatagramSocketImpl@62836b78
at java.net.DatagramSocket.receive(Unknown Source) - locked java.net.DatagramPacket@6756aa7b
- locked java.net.MulticastSocket@6ad2a63d
at javax.jmdns.impl.SocketListener.run(SocketListener.java:41)
TCP slave agent listener port=0
"TCP slave agent listener port=0" Id=30 Group=main RUNNABLE (in native)
at java.net.TwoStacksPlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(Unknown Source)
at java.net.PlainSocketImpl.accept(Unknown Source)
- locked java.net.SocksSocketImpl@7ef39e7a
at java.net.ServerSocket.implAccept(Unknown Source)
at java.net.ServerSocket.accept(Unknown Source)
at hudson.TcpSlaveAgentListener.run(TcpSlaveAgentListener.java:93)
ViewJob reload thread
"ViewJob reload thread" Id=28 Group=main TIMED_WAITING on java.util.LinkedHashSet@39d0518e
at java.lang.Object.wait(Native Method)
- waiting on java.util.LinkedHashSet@39d0518e
at hudson.model.ViewJob$ReloadThread.getNext(ViewJob.java:176)
at hudson.model.ViewJob$ReloadThread.run(ViewJob.java:193)
Waiting to acquire c:\Jenkins\workspace\ipprotection_32_64 : SCM polling for hudson.matrix.MatrixProject@3b277576[ipprotection_32_64]
"Waiting to acquire c:\Jenkins\workspace\ipprotection_32_64 : SCM polling for hudson.matrix.MatrixProject@3b277576[ipprotection_32_64]" Id=12065 Group=main WAITING on hudson.slaves.WorkspaceList@195952ce
at java.lang.Object.wait(Native Method)
- waiting on hudson.slaves.WorkspaceList@195952ce
at java.lang.Object.wait(Object.java:503)
at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:232)
at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:211)
at hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1580)
at hudson.model.AbstractProject._poll(AbstractProject.java:1558)
at hudson.model.AbstractProject.poll(AbstractProject.java:1490)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:439)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:468)
at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@3435c217
Attach Listener
"Attach Listener" Id=5 Group=system RUNNABLE
Finalizer
"Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@4017e342
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.ReferenceQueue$Lock@4017e342
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
Java2D Disposer
"Java2D Disposer" Id=12 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@1bf08062
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.ReferenceQueue$Lock@1bf08062
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at sun.java2d.Disposer.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Reference Handler
"Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@604922ed
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.Reference$Lock@604922ed
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
Signal Dispatcher
"Signal Dispatcher" Id=4 Group=system RUNNABLE
CAAV5R19x64
N/A
offline
LB3D-Wxpp64sp2
Channel reader thread: channel
"Channel reader thread: channel" Id=11 Group=main RUNNABLE (in native)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@1c2ba649
at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.Command.readFrom(Command.java:92)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
main
"main" Id=1 Group=main WAITING on hudson.remoting.Engine@7a0136b0
at java.lang.Object.wait(Native Method)
- waiting on hudson.remoting.Engine@7a0136b0
at java.lang.Thread.join(Unknown Source)
at java.lang.Thread.join(Unknown Source)
at hudson.remoting.jnlp.Main.main(Main.java:137)
at hudson.remoting.jnlp.Main._main(Main.java:130)
at hudson.remoting.Launcher.run(Launcher.java:223)
at hudson.remoting.Launcher.main(Launcher.java:192)
Ping thread for channel hudson.remoting.Channel@28509953:channel
"Ping thread for channel hudson.remoting.Channel@28509953:channel" Id=13 Group=main TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at hudson.remoting.PingThread.run(PingThread.java:86)
pool-1-thread-18
"pool-1-thread-18" Id=30 Group=main RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
at hudson.Functions.getThreadInfos(Functions.java:1131)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Unknown Source)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@59d9c141
pool-1-thread-2
"pool-1-thread-2" Id=14 Group=main RUNNABLE
at com.sun.jna.Native.initIDs(Native Method)
at com.sun.jna.Native.<clinit>(Native.java:122)
at org.tmatesoft.svn.core.internal.util.jna.JNALibraryLoader.<clinit>(JNALibraryLoader.java:50)
at org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:93)
at org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196)
at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1869)
at org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57)
at hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:975)
at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:952)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:885)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:872)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2420)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Unknown Source)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@5133066c
pool-1-thread-5
"pool-1-thread-5" Id=17 Group=main RUNNABLE
at com.sun.jna.Structure.<clinit>(Structure.java:134)
at org.jvnet.hudson.Windows.monitor(Windows.java:40)
at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:110)
at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:100)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Unknown Source)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@786dee4c
Thread-1
"Thread-1" Id=10 Group=main WAITING on hudson.remoting.Channel@28509953
at java.lang.Object.wait(Native Method)
- waiting on hudson.remoting.Channel@28509953
at java.lang.Object.wait(Object.java:503)
at hudson.remoting.Channel.join(Channel.java:842)
at hudson.remoting.Engine.run(Engine.java:271)
Attach Listener
"Attach Listener" Id=4 Group=system RUNNABLE
Finalizer
"Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@75b21111
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.ReferenceQueue$Lock@75b21111
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
Reference Handler
"Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@7d7eaf74
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.Reference$Lock@7d7eaf74
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
Slave-TestProcessBuild
Channel reader thread: channel
"Channel reader thread: channel" Id=11 Group=main RUNNABLE (in native)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@74784829
at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.Command.readFrom(Command.java:92)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
main
"main" Id=1 Group=main WAITING on hudson.remoting.Engine@7f25ab41
at java.lang.Object.wait(Native Method)
- waiting on hudson.remoting.Engine@7f25ab41
at java.lang.Thread.join(Unknown Source)
at java.lang.Thread.join(Unknown Source)
at hudson.remoting.jnlp.Main.main(Main.java:137)
at hudson.remoting.jnlp.Main._main(Main.java:130)
at hudson.remoting.Launcher.run(Launcher.java:223)
at hudson.remoting.Launcher.main(Launcher.java:192)
Ping thread for channel hudson.remoting.Channel@1c06d198:channel
"Ping thread for channel hudson.remoting.Channel@1c06d198:channel" Id=13 Group=main TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at hudson.remoting.PingThread.run(PingThread.java:86)
pool-1-thread-802
"pool-1-thread-802" Id=916 Group=main RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
at hudson.Functions.getThreadInfos(Functions.java:1131)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Unknown Source)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@13c5810e
Thread-1
"Thread-1" Id=10 Group=main WAITING on hudson.remoting.Channel@1c06d198
at java.lang.Object.wait(Native Method)
- waiting on hudson.remoting.Channel@1c06d198
at java.lang.Object.wait(Object.java:503)
at hudson.remoting.Channel.join(Channel.java:842)
at hudson.remoting.Engine.run(Engine.java:271)
Attach Listener
"Attach Listener" Id=4 Group=system RUNNABLE
Finalizer
"Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@4c9b77c
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.ReferenceQueue$Lock@4c9b77c
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
Reference Handler
"Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@572d3058
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.Reference$Lock@572d3058
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
LB3D-Wxpp32sp3
Channel reader thread: channel
"Channel reader thread: channel" Id=10 Group=main RUNNABLE (in native)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@b9293d
at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.Command.readFrom(Command.java:92)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
main
"main" Id=1 Group=main WAITING on hudson.remoting.Engine@1059a9a
at java.lang.Object.wait(Native Method)
- waiting on hudson.remoting.Engine@1059a9a
at java.lang.Thread.join(Unknown Source)
at java.lang.Thread.join(Unknown Source)
at hudson.remoting.jnlp.Main.main(Main.java:137)
at hudson.remoting.jnlp.Main._main(Main.java:130)
at hudson.remoting.Launcher.run(Launcher.java:223)
at hudson.remoting.Launcher.main(Launcher.java:192)
Ping thread for channel hudson.remoting.Channel@41f871:channel
"Ping thread for channel hudson.remoting.Channel@41f871:channel" Id=12 Group=main TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at hudson.remoting.PingThread.run(PingThread.java:86)
pool-1-thread-8
"pool-1-thread-8" Id=42 Group=main RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
at hudson.Functions.getThreadInfos(Functions.java:1131)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Unknown Source)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@5322ec
Thread-1
"Thread-1" Id=9 Group=main WAITING on hudson.remoting.Channel@41f871
at java.lang.Object.wait(Native Method)
- waiting on hudson.remoting.Channel@41f871
at java.lang.Object.wait(Object.java:503)
at hudson.remoting.Channel.join(Channel.java:842)
at hudson.remoting.Engine.run(Engine.java:271)
Attach Listener
"Attach Listener" Id=4 Group=system RUNNABLE
Finalizer
"Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@429b8e
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.ReferenceQueue$Lock@429b8e
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
Reference Handler
"Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@f2c136
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.Reference$Lock@f2c136
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
Slave-TestProcessTest
Channel reader thread: channel
"Channel reader thread: channel" Id=10 Group=main RUNNABLE (in native)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@18e02f3
at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.Command.readFrom(Command.java:92)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
main
"main" Id=1 Group=main WAITING on hudson.remoting.Engine@13ac592
at java.lang.Object.wait(Native Method)
- waiting on hudson.remoting.Engine@13ac592
at java.lang.Thread.join(Unknown Source)
at java.lang.Thread.join(Unknown Source)
at hudson.remoting.jnlp.Main.main(Main.java:137)
at hudson.remoting.jnlp.Main._main(Main.java:130)
at hudson.remoting.Launcher.run(Launcher.java:223)
at hudson.remoting.Launcher.main(Launcher.java:192)
Ping thread for channel hudson.remoting.Channel@1a32902:channel
"Ping thread for channel hudson.remoting.Channel@1a32902:channel" Id=12 Group=main TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at hudson.remoting.PingThread.run(PingThread.java:86)
pool-1-svnkit-thread-1
"pool-1-svnkit-thread-1" Id=60 Group=main WAITING on java.util.concurrent.SynchronousQueue$TransferStack@a0f65b
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.SynchronousQueue$TransferStack@a0f65b
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
at java.util.concurrent.SynchronousQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
pool-1-thread-79
"pool-1-thread-79" Id=135 Group=main RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
at hudson.Functions.getThreadInfos(Functions.java:1131)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Unknown Source)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@1232e8d
Thread-1
"Thread-1" Id=9 Group=main WAITING on hudson.remoting.Channel@1a32902
at java.lang.Object.wait(Native Method)
- waiting on hudson.remoting.Channel@1a32902
at java.lang.Object.wait(Object.java:503)
at hudson.remoting.Channel.join(Channel.java:842)
at hudson.remoting.Engine.run(Engine.java:271)
Attach Listener
"Attach Listener" Id=4 Group=system RUNNABLE
Finalizer
"Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@63bafc
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.ReferenceQueue$Lock@63bafc
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
Reference Handler
"Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@1893737
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.Reference$Lock@1893737
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
I've also ran into a similar issue in a sense that it also involved stuck builds and "Failed to monitor <slave-name> for Free Swap Space" exception.
However, the exception details are a bit different:
WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor
Failed to monitor <slave-name> for Free Swap Space
java.util.concurrent.ExecutionException: java.lang.UnsatisfiedLinkError: com.sun.jna.Native.malloc(J)J
at hudson.remoting.Channel$3.adapt(Channel.java:747)
at hudson.remoting.Channel$3.adapt(Channel.java:742)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282)
Caused by: java.lang.UnsatisfiedLinkError: com.sun.jna.Native.malloc(J)J
at com.sun.jna.Native.malloc(Native Method)
at com.sun.jna.Memory.malloc(Memory.java:690)
at com.sun.jna.Memory.(Memory.java:79)
at com.sun.jna.Structure$AutoAllocated.(Structure.java:1366)
at com.sun.jna.Structure.autoAllocate(Structure.java:231)
at com.sun.jna.Structure.allocateMemory(Structure.java:302)
at com.sun.jna.Structure.(Structure.java:179)
at com.sun.jna.Structure.(Structure.java:169)
at com.sun.jna.Structure.(Structure.java:165)
at com.sun.jna.Structure.(Structure.java:156)
at org.jvnet.hudson.Windows$MEMORYSTATUSEX.(Windows.java:55)
at org.jvnet.hudson.Windows.monitor(Windows.java:40)
at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:110)
at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:100)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
What I also noticed a couple of times is that java.exe process on master starts hobbing all of the available RAM plus it's thread count rises.
I'm running Jenkins ver. 1.530 on a 2008 R2 server, nodes (all of them are windows servers 2003 x86) are managed via vSphere Cloud plug-in and are being controlled as a Windows service using administrator account.
I'm also encountering this problem. Same symptoms. Jenkins 1.538, Windows 2008 R2 server, vSphere Cloud plugin. All the slaves that have these symptoms are showing this only after a slave restart - if they are always on, they don't have this problem. So far, only seen on Windows slaves; they use the JNLP agent starting mechanism. Several versions of Windows; Windows 7, Windows 8, Windows XP.
Hi guys,
Good to hear I'm not alone in this trouble! Any idea who we should contact to look at this issue? I think my company would no mind to pay a "service fee" to get this issue fixed.
Yesterday, I changed the way of starting the slave service. Instead of automatically started service, I'm now using a batch file executed during the windows start. Sadly, this did not help. Night build got stuck again with one slave and sure enough, the log contained several "Failed to monitor SlaveName for Free Swap Space" errors.
Sadly, I have no idea where to start looking for a solution. Since this only happens on our large production system and not our smaller test system, I can't effectively try solutions. I was going to try a fresh Jenkins with just the vSphere plugin, then slowly add plugins to match production.
Guys, are you running Jenkins on built-in Winstone or Apache Tomcat?
Same here. I've got a hunch though, that the exception is somehow related to it, so I'm going to set up Tomcat and see if it makes any difference.
So I've set Jenkins up to run on Tomcat 7 up and upgraded 'archive artifacts' plug-in to latest version for a good measure (since all of my builds are using that and it is known to be greedy with RAM) and what do you know, 3 days without stalled builds and counting. Will let you guys know if the issue indeed went away.
After running the setup for a while, I came to the conclusion that the issue is still there, although with reduced amount of fails.
Thought i'd throw in my errors. We are seeing HTTP stop responding and visualVM can't take a dump once it fails. Memory is fine, but threads just keep increasing, once they hit 550-600 active it just stops and says memory issue won't respond till the master is restarted.
Jenkins 1.484, win2k3 (master) win2k8/win2k3 (slaves)
Feb 25, 2014 9:21:06 AM hudson.model.DownloadService$Downloadable doPostBack
INFO: Obtained the updated data file for hudson.tools.JDKInstaller
Feb 25, 2014 9:21:06 AM hudson.model.DownloadService$Downloadable doPostBack
INFO: Obtained the updated data file for hudson.tasks.Ant.AntInstaller
Feb 25, 2014 9:21:06 AM org.kohsuke.stapler.compression.CompressionFilter reportException
WARNING: Untrapped servlet exception
java.io.IOException: Unable to delete E:\jenkins\updates\hudson.tasks.Maven.MavenInstaller
at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:112)
at hudson.util.TextFile.write(TextFile.java:81)
at hudson.model.DownloadService$Downloadable.doPostBack(DownloadService.java:260)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:288)
at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:151)
at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:90)
at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659)
at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659)
at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:384)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659)
at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:203)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:488)
at org.kohsuke.stapler.Stapler.service(Stapler.java:162)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
at winstone.ServletConfiguration.execute(ServletConfiguration.java:248)
at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)
at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
FREE DISK SPACE MONITORING STILL IN PROGRESS, FAILED, THREADS LEFT, MAXING
Feb 21, 2014 4:55:20 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted connection #12 from /172.16.69.140:64901
Feb 21, 2014 4:55:21 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted connection #13 from /172.16.69.82:61763
Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record <init>
WARNING: Previous Free Swap Space monitoring activity still in progress. Interrupting
Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record <init>
WARNING: Previous Free Disk Space monitoring activity still in progress. Interrupting
Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record <init>
WARNING: Previous Free Temp Space monitoring activity still in progress. Interrupting
Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run
WARNING: Node monitoring FFX-MOM-BLD10 for Free Swap Space aborted.
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at hudson.remoting.Request.call(Request.java:146)
at hudson.remoting.Channel.call(Channel.java:664)
at hudson.node_monitors.SwapSpaceMonitor$1.monitor(SwapSpaceMonitor.java:83)
at hudson.node_monitors.SwapSpaceMonitor$1.monitor(SwapSpaceMonitor.java:81)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219)
Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run
WARNING: Node monitoring FFX-MOM-BLD5 for Free Temp Space aborted.
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at hudson.remoting.Request.call(Request.java:146)
at hudson.remoting.Channel.call(Channel.java:664)
at hudson.FilePath.act(FilePath.java:840)
at hudson.FilePath.act(FilePath.java:824)
at hudson.node_monitors.TemporarySpaceMonitor$1.getFreeSpace(TemporarySpaceMonitor.java:73)
at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:169)
at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:49)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219)
Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run
WARNING: Node monitoring FFX-MOM-BLD5 for Free Disk Space aborted.
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at hudson.remoting.Request.call(Request.java:146)
at hudson.remoting.Channel.call(Channel.java:664)
at hudson.FilePath.act(FilePath.java:840)
at hudson.FilePath.act(FilePath.java:824)
at hudson.node_monitors.DiskSpaceMonitor$1.getFreeSpace(DiskSpaceMonitor.java:71)
at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:169)
at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:49)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219)
BELOW LEAVING OPEN THREADS AND MAXING????
Feb 20, 2014 6:32:49 AM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run
WARNING: Failed to monitor FFX-MOM-BLD10 for Free Temp Space
hudson.util.IOException2: remote file operation failed: E:\jenkins at hudson.remoting.Channel@18b622:FFX-MOM-BLD10
at hudson.FilePath.act(FilePath.java:847)
at hudson.FilePath.act(FilePath.java:824)
at hudson.node_monitors.TemporarySpaceMonitor$1.getFreeSpace(TemporarySpaceMonitor.java:73)
at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:169)
at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:49)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219)
Caused by: java.io.IOException: Remote call on FFX-MOM-BLD10 failed
at hudson.remoting.Channel.call(Channel.java:673)
at hudson.FilePath.act(FilePath.java:840)
... 5 more
Caused by: java.lang.Error: Failed to deserialize the Callable object.
at hudson.remoting.UserRequest.perform(UserRequest.java:104)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:60)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.reflect.UndeclaredThrowableException
at $Proxy6.fetch2(Unknown Source)
at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:122)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClassInternal(Unknown Source)
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Unknown Source)
at java.lang.Class.getDeclaredMethod(Unknown Source)
at java.io.ObjectStreamClass.getPrivateMethod(Unknown Source)
at java.io.ObjectStreamClass.access$1700(Unknown Source)
at java.io.ObjectStreamClass$2.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.ObjectStreamClass.<init>(Unknown Source)
at java.io.ObjectStreamClass.lookup(Unknown Source)
at java.io.ObjectStreamClass.initNonProxy(Unknown Source)
at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)
at java.io.ObjectInputStream.readClassDesc(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.UserRequest.deserialize(UserRequest.java:182)
at hudson.remoting.UserRequest.perform(UserRequest.java:98)
... 9 more
Caused by: java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at hudson.remoting.Request.call(Request.java:146)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:160)
... 35 more
Feb 20, 2014 6:32:49 AM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run
WARNING: Failed to monitor FFX-MOM-BLD12 for Free Swap Space
java.io.IOException: Remote call on FFX-MOM-BLD12 failed
at hudson.remoting.Channel.call(Channel.java:673)
at hudson.node_monitors.SwapSpaceMonitor$1.monitor(SwapSpaceMonitor.java:83)
at hudson.node_monitors.SwapSpaceMonitor$1.monitor(SwapSpaceMonitor.java:81)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.jvnet.hudson.Windows$Kernel32
at org.jvnet.hudson.Windows.monitor(Windows.java:41)
at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:113)
at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:99)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at hudson.remoting.Engine$1$1.run(Engine.java:60)
at java.lang.Thread.run(Thread.java:619)
Feb 20, 2014 7:32:49 AM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run
WARNING: Failed to monitor FFX-MOM-BLD10 for Free Temp Space
hudson.util.IOException2: remote file operation failed: E:\jenkins at hudson.remoting.Channel@18b622:FFX-MOM-BLD10
at hudson.FilePath.act(FilePath.java:847)
at hudson.FilePath.act(FilePath.java:824)
at hudson.node_monitors.TemporarySpaceMonitor$1.getFreeSpace(TemporarySpaceMonitor.java:73)
at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:169)
at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:49)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219)
Caused by: java.io.IOException: Remote call on FFX-MOM-BLD10 failed
at hudson.remoting.Channel.call(Channel.java:673)
at hudson.FilePath.act(FilePath.java:840)
... 5 more
Caused by: java.lang.Error: Failed to deserialize the Callable object.
at hudson.remoting.UserRequest.perform(UserRequest.java:104)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:60)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.reflect.UndeclaredThrowableException
at $Proxy6.fetch2(Unknown Source)
at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:122)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClassInternal(Unknown Source)
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Unknown Source)
at java.lang.Class.getDeclaredMethod(Unknown Source)
at java.io.ObjectStreamClass.getPrivateMethod(Unknown Source)
at java.io.ObjectStreamClass.access$1700(Unknown Source)
at java.io.ObjectStreamClass$2.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.ObjectStreamClass.<init>(Unknown Source)
at java.io.ObjectStreamClass.lookup(Unknown Source)
at java.io.ObjectStreamClass.initNonProxy(Unknown Source)
at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)
at java.io.ObjectInputStream.readClassDesc(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.UserRequest.deserialize(UserRequest.java:182)
at hudson.remoting.UserRequest.perform(UserRequest.java:98)
... 9 more
Caused by: java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at hudson.remoting.Request.call(Request.java:146)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:160)
... 35 more
Feb 20, 2014 7:50:42 AM winstone.Logger logInternal
SEVERE: Error during HTTP listener init or shutdown
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
at winstone.BoundedExecutorService.scheduleNext(BoundedExecutorService.java:74)
at winstone.BoundedExecutorService.execute(BoundedExecutorService.java:64)
at java.util.concurrent.AbstractExecutorService.submit(Unknown Source)
at winstone.ObjectPool.handleRequest(ObjectPool.java:102)
at winstone.HttpListener.run(HttpListener.java:144)
at java.lang.Thread.run(Unknown Source)
Feb 20, 2014 7:50:45 AM org.kohsuke.stapler.compression.CompressionFilter reportException
WARNING: Untrapped servlet exception
winstone.ClientSocketException: Failed to write to client
at winstone.ClientOutputStream.write(ClientOutputStream.java:41)
at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
at java.io.BufferedOutputStream.flush(Unknown Source)
at winstone.WinstoneOutputStream.commit(WinstoneOutputStream.java:167)
at winstone.WinstoneOutputStream.commit(WinstoneOutputStream.java:123)
at winstone.WinstoneOutputStream.write(WinstoneOutputStream.java:112)
at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:421)
at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:249)
at org.kohsuke.stapler.Stapler.service(Stapler.java:152)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
at winstone.ServletConfiguration.execute(ServletConfiguration.java:248)
at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)
at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at winstone.ClientOutputStream.write(ClientOutputStream.java:39)
... 57 more
REACHED LIMIT AND FAIL
Feb 23, 2014 5:07:53 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started Workspace clean-up
Feb 23, 2014 5:07:56 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished Workspace clean-up. 3156 ms
Feb 23, 2014 7:08:07 AM hudson.triggers.SafeTimerTask run
SEVERE: Timer task hudson.node_monitors.AbstractNodeMonitorDescriptor$1@142971b failed
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
at hudson.node_monitors.AbstractNodeMonitorDescriptor.triggerUpdate(AbstractNodeMonitorDescriptor.java:181)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$1.doRun(AbstractNodeMonitorDescriptor.java:78)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
at java.util.TimerThread.mainLoop(Unknown Source)
at java.util.TimerThread.run(Unknown Source)
Feb 23, 2014 7:15:08 AM winstone.Logger logInternal
WARNING: Untrapped Error in Servlet
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
at com4j.ComThread.<init>(ComThread.java:63)
at com4j.ComThread.<init>(ComThread.java:21)
at com4j.ComThread$1.initialValue(ComThread.java:31)
at com4j.ComThread$1.initialValue(ComThread.java:26)
at java.lang.ThreadLocal.setInitialValue(Unknown Source)
at java.lang.ThreadLocal.get(Unknown Source)
at com4j.ComThread.get(ComThread.java:39)
at com4j.Task.execute(Task.java:25)
at com4j.COM4J.createInstance(COM4J.java:97)
at com4j.COM4J.createInstance(COM4J.java:72)
at com4j.typelibs.ado20.ClassFactory.createCommand(ClassFactory.java:17)
at hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider.getDnOfUserOrGroup(ActiveDirectoryAuthenticationProvider.java:162)
at hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider.retrieveUser(ActiveDirectoryAuthenticationProvider.java:61)
at org.acegisecurity.providers.dao.AbstractUserDetailsAuthenticationProvider.authenticate(AbstractUserDetailsAuthenticationProvider.java:119)
at org.acegisecurity.providers.ProviderManager.doAuthentication(ProviderManager.java:195)
at org.acegisecurity.AbstractAuthenticationManager.authenticate(AbstractAuthenticationManager.java:45)
at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:138)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)
at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Feb 23, 2014 7:15:08 AM winstone.Logger logInternal
WARNING: Untrapped Error in Servlet
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
at com4j.ComThread.<init>(ComThread.java:63)
at com4j.ComThread.<init>(ComThread.java:21)
at com4j.ComThread$1.initialValue(ComThread.java:31)
at com4j.ComThread$1.initialValue(ComThread.java:26)
at java.lang.ThreadLocal.setInitialValue(Unknown Source)
at java.lang.ThreadLocal.get(Unknown Source)
at com4j.ComThread.get(ComThread.java:39)
at com4j.Task.execute(Task.java:25)
at com4j.COM4J.createInstance(COM4J.java:97)
at com4j.COM4J.createInstance(COM4J.java:72)
at com4j.typelibs.ado20.ClassFactory.createCommand(ClassFactory.java:17)
at hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider.getDnOfUserOrGroup(ActiveDirectoryAuthenticationProvider.java:162)
at hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider.retrieveUser(ActiveDirectoryAuthenticationProvider.java:61)
at org.acegisecurity.providers.dao.AbstractUserDetailsAuthenticationProvider.authenticate(AbstractUserDetailsAuthenticationProvider.java:119)
at org.acegisecurity.providers.ProviderManager.doAuthentication(ProviderManager.java:195)
at org.acegisecurity.AbstractAuthenticationManager.authenticate(AbstractAuthenticationManager.java:45)
at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:138)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)
at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Feb 23, 2014 7:15:08 AM winstone.Logger logInternal
SEVERE: Error during HTTP listener init or shutdown
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
at winstone.BoundedExecutorService.scheduleNext(BoundedExecutorService.java:74)
at winstone.BoundedExecutorService.execute(BoundedExecutorService.java:64)
at java.util.concurrent.AbstractExecutorService.submit(Unknown Source)
at winstone.ObjectPool.handleRequest(ObjectPool.java:102)
at winstone.HttpListener.run(HttpListener.java:144)
at java.lang.Thread.run(Unknown Source)
Feb 23, 2014 8:08:07 AM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record <init>
WARNING: Previous Free Swap Space monitoring activity still in progress. Interrupting
Feb 23, 2014 2:53:11 PM hudson.model.AsyncPeriodicWork$1 run
INFO: Started Fingerprint cleanup
Feb 23, 2014 2:53:11 PM hudson.model.FingerprintCleanupThread execute
INFO: Cleaned up 0 records
I am facing the same or similar problem.
I have Jenkins linux master and slaves on the Proxmox Windows XP machines.
Time to time, a slave stuck.
The cosole output ends with line - building remotely on ...
No SVN checkout is executed after, the build is totally stuck.
The slave says: Slave successfully connected and online
I have to restart the slave machine and abort the job. If i only abort the job, the next build is stuck too.
This is caused by a deadlock with the subversion plugin and the "Free swap space" monitor option. To prevent this issue you can go to Manage Jenkins > Manage Nodes > Configure and disable this option.
To prevent this issue you can go to Manage Jenkins > Manage Nodes > Configure and disable this option.
Aren't node monitors still run, just won't take nodes offline that fail the check?
We are seeing this intermittently on Jenkins 1.656 with a windows 7 remote slave. We have disabled the Free swap space monitoring option but still see the issue.
We are also experiencing this problem. We have disabled all monitoring on the slaves.
We updated to the latest SVN plugin 2.5.7, including SVNKit 1.8.11
Seems like the problem is a deadlock in the startup of the windows slave. We are using cygwin/ssh connections, if that matters. Looking at the stack traces with SVN involved*:
The java method getApplicationDataPath() is loading the JNA, which is synchronized in all of it's static methods. So as this is the first time the class is refenced is also doing the static initialization <clinit>. Looks like the Native.loadLibrary is stuck and producing the blockage, to all the threads using SVN.
*) The actual thread stack dump. Involving SVN.
"pool-1-thread-3 for channel" prio=5 RUNNABLE
com.sun.jna.Native.initIDs(Native Method)
com.sun.jna.Native.<clinit>(Native.java:122)
org.tmatesoft.svn.core.internal.util.jna.JNALibraryLoader.<clinit>(JNALibraryLoader.java:50)
org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:113)
org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196)
org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1948)
org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57)
hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:1055)
hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1032)
hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:965)
hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:945)
hudson.FilePath$FileCallableWrapper.call(FilePath.java:2577)
hudson.remoting.UserRequest.perform(UserRequest.java:121)
hudson.remoting.UserRequest.perform(UserRequest.java:49)
hudson.remoting.Request$2.run(Request.java:324)
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
java.util.concurrent.FutureTask.run(FutureTask.java:262)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
"pool-1-thread-52 for channel" prio=5 RUNNABLE
org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:113)
org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196)
org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1948)
org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57)
hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:1055)
hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1032)
hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:965)
hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:945)
hudson.FilePath$FileCallableWrapper.call(FilePath.java:2577)
hudson.remoting.UserRequest.perform(UserRequest.java:121)
hudson.remoting.UserRequest.perform(UserRequest.java:49)
hudson.remoting.Request$2.run(Request.java:324)
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
java.util.concurrent.FutureTask.run(FutureTask.java:262)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
@See https://issues.tmatesoft.com/issue/SGT-451
Slaves need to force load of jna classes at startup.
FYI, we have a patch for jenkins/remoting-2.47 that allows to by-pass this issue by defining property hudson.remoting.RemoteClassLoader.force to com.sun.jna.Native. This forces the load of the offending class early on at any UserRequest with a new ClassLoader.
We have tested this patch timed job every 5 minutes, with a dormant slave that needs to be waken. We will be requesting merge in github master shortly.
Deactivating the (all) monitorings on the Jenkins master did not help us. Before reconnecting the swarm client, we see this:
Curiously, we also still see the monitoring in the Jenkins log:
{{
May 09, 2016 9:38:39 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor
Failed to monitor ****-ed055395 for Free Swap Space
java.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:273)
at hudson.remoting.Request$1.get(Request.java:207)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
}}
On the build node, amidst tons of threads, we see two that are referencing org.tmatesoft.svn.core.internal.util.jna:
{{pool-1-thread-2 for channel
"pool-1-thread-2 for channel" Id=16 Group=main RUNNABLE
at com.sun.jna.Native.initIDs(Native Method)
at com.sun.jna.Native.<clinit>(Native.java:148)
at org.tmatesoft.svn.core.internal.util.jna.JNALibraryLoader.<clinit>(JNALibraryLoader.java:50)
at org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:113)
at org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196)
at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1948)
at org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57)
at hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:1055)
at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1032)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:965)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:945)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2719)
at hudson.remoting.UserRequest.perform(UserRequest.java:121)
at hudson.remoting.UserRequest.perform(UserRequest.java:49)
at hudson.remoting.Request$2.run(Request.java:324)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Unknown Source)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@5f599d56
}}
{{
pool-1-thread-916 for channel
"pool-1-thread-916 for channel" Id=930 Group=main RUNNABLE
at org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:113)
at org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196)
at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1948)
at org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57)
at hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:1055)
at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1032)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:965)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:945)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2719)
at hudson.remoting.UserRequest.perform(UserRequest.java:121)
at hudson.remoting.UserRequest.perform(UserRequest.java:49)
at hudson.remoting.Request$2.run(Request.java:324)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Unknown Source)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@772c6f4f
}}
After reconnecting two swarm clients (but not restarting Jenkins), one builds, one hangs, showing similar traces.
From the hanging client:
{{pool-1-thread-6 for channel
"pool-1-thread-6 for channel" Id=20 Group=main RUNNABLE
at com.sun.jna.Pointer.<clinit>(Pointer.java:41)
at com.sun.jna.Structure.<clinit>(Structure.java:2078)
at org.jvnet.hudson.Windows.monitor(Windows.java:42)
at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:113)
at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:103)
at hudson.remoting.UserRequest.perform(UserRequest.java:121)
at hudson.remoting.UserRequest.perform(UserRequest.java:49)
at hudson.remoting.Request$2.run(Request.java:324)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Unknown Source)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@2e201c5c}}
It seems the FreeSwapMonitoring starts a new thread on the node every time it is started (we can start it manually using the "Refresh Status" Button in the BuildExecutor Status view).
We see that some threads are actually closed, so it's not that all threads hang, but for every time we hit refresh status, we see a new instance of at ```org.jvnet.hudson.Windows.monitor``` (if we wait some minute between hitting that button)
Code changed in jenkins
User: Jenkins BBVA
Path:
src/main/java/hudson/remoting/UserRequest.java
http://jenkins-ci.org/commit/remoting/8e0a8667a04dfd8d3509a08dc693de31cd5d2e77
Log:
JENKINS-19445 Force class load on UserRequest (#82)
- Force class load on UserRequest, see JENKINS-19445
This is a workaround to force class load of any class. This helps prevent deadlock on windows nodes when using JNA and Subversion.
Use property hudson.remoting.RemoteClassLoader.force to name the class to load. Ideally this forced load should happen earlier on the startup, however the classloader isn't available.
- Update UserRequest as per code review comments
Fixes based on code review of pull-request: https://github.com/jenkinsci/remoting/pull/82
- Fix Logger.log() invocation
Fix Logger.log() invocation: needs Object[], not vargs.
Code changed in jenkins
User: Oleg Nenashev
Path:
pom.xml
http://jenkins-ci.org/commit/jenkins/409438f36dc80f20964fb16f8d88041e11ba4ed4
Log:
[JENKINS-19445, JENKINS-34213, JENKINS-34808, JENKINS-34121] Bump remoting to 2.59. (#2344)
- [JENKINS-19445, JENKINS-34213, JENKINS-34808] Bump remoting to 2.58.
Changes:
JENKINS-34213(https://issues.jenkins-ci.org/browse/JENKINS-34213) - Ensure that the unexporter cleans up whatever it can each sweep (https://github.com/jenkinsci/remoting/pull/81)- JENKINS-19445(https://issues.jenkins-ci.org/browse/JENKINS-19445) Force class load on UserRequest in order to prevent deadlock on windows nodes when using JNA and Subversion (https://github.com/jenkinsci/remoting/pull/81)
JENKINS-34808(https://issues.jenkins-ci.org/browse/JENKINS-34808) - Allow user to adjust socket timeout (https://github.com/jenkinsci/remoting/pull/68)
JENKINS-34121- Upgrade remoting to 2.59
ddaumiller
Jenkins 2 LTS - yes
1.651.3 - under review: https://github.com/jenkinsci/jenkins/pull/2357
Code changed in jenkins
User: Oleg Nenashev
Path:
pom.xml
http://jenkins-ci.org/commit/jenkins/12e79963cca5122351943ee107f65c3ad91a2e25
Log:
[JENKINS-19445, JENKINS-34213, JENKINS-34808, JENKINS-34121] Bump remoting to 2.59. (#2344)
- [JENKINS-19445, JENKINS-34213, JENKINS-34808] Bump remoting to 2.58.
Changes:
JENKINS-34213(https://issues.jenkins-ci.org/browse/JENKINS-34213) - Ensure that the unexporter cleans up whatever it can each sweep (https://github.com/jenkinsci/remoting/pull/81)- JENKINS-19445(https://issues.jenkins-ci.org/browse/JENKINS-19445) Force class load on UserRequest in order to prevent deadlock on windows nodes when using JNA and Subversion (https://github.com/jenkinsci/remoting/pull/81)
JENKINS-34808(https://issues.jenkins-ci.org/browse/JENKINS-34808) - Allow user to adjust socket timeout (https://github.com/jenkinsci/remoting/pull/68)
JENKINS-34121- Upgrade remoting to 2.59
(cherry picked from commit 409438f36dc80f20964fb16f8d88041e11ba4ed4)
We are showing this happening on the latest jenkins build (2.17) and latest slave jar (2.60) on a Windows 7 64-bit slave. Appears to be an intermittent fault but almost always occurs when an on-demand build is taking place.
Dave Taddei, can you confirm you are using the property hudson.remoting.RemoteClassLoader.force to force the load of com.sun.jna.Native. Also that you can see that the class is being loaded via the log of the slave.
Hi BBVA, it does not appear so but I am unsure where to look. Should this be set in the JAVA_ARGS for the master, slave or both? I was also under the impression that had been added to release 2.47 so didn't set anything.
drtaddei set JVM Options of the slave with: -Dhudson.remoting.RemoteClassLoader.force=com.sun.jna.Native
We are using dynamic slaves, with the vcenter plugin. The vcenter plugin does not allow the specifications of any JVM options.
At least, I do not believe it does. Could the above change be made permanent, or set as the default for all slaves some how?
gregcovertsmith et al.—see the last comment I left in JENKINS-39179: I wrote a little plugin which might work around JNA deadlocks without needing individual configuration of static agents.
I've added also master thread dump.
And one more weird thing I just noticed on 64bit machine, which just stuck again.
At some point, Jenkins console says this this...
←[33mSep 04, 2013 12:52:34 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescr
iptor monitor
WARNING: Failed to monitor LB3D-Wxpp64sp2 for Free Swap Space
java.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:275)
at hudson.remoting.Request$1.get(Request.java:210)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(Abstr
actAsyncNodeMonitorDescriptor.java:97)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(Abstrac
tNodeMonitorDescriptor.java:282)
The weird thing is, that I've disabled the "Free Swap Space" monitoring in Jenkins Node configuration section. So how the "Free Swap Space" monitoring could fail on that slave machine? Could it be a cause of broken master-slave connection or maybe corrupted threads?