Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19445

Jobs randomly stuck with "building remotely on slave-name" message

      Hi folks,

      Last week, I started to experience a weird issue with freezing/endless waiting of building jobs on slave machines. It first started with WinXP 32 bit slave, but I'm now experiencing the same problem also with WinXP 64 bit slave.

      The symptoms are like this...
      The job is started by SCM change or predefined time.
      Both slave machines are correctly started (the last line, in their respective logs, says "Slave successfully connected and online").
      Now the problem is, that the last line in console says "Building remotely on slave-name..." and there is just scrolling the progress circle. Nothing else happens. No GIT or SVN activity, no building, no testing, simply nothing and it stays like that until I kill the job manually.

      Sometimes, the problem happens with just one machine, and the other one runs OK. But today it struck both machines.

      Last week, when the problem started, I tried to update the Jenkins and both slaves with the most recent versions and latest plugins (except the "Git Client Plugin", because of a known issue with building history). Few hours all worked great, then the problem reappeared.

      Please see the attached files with logs and thread dumps from both slave machines.

      What struck me is the high number of threads started on both machines, which looks really weird? But I don't understand Jenkins that deep so I can't say for sure if it's normal or not?

      Any idea what to try or how to solve this behavior?

      Have a nice day,
      Pavel

        1. threadump.txt
          9 kB
        2. remoting-2.47.patch
          2 kB
        3. LB3D-MASTER_05_09_13.txt
          53 kB
        4. LB3D-Wxpp32sp3_05_09_13.txt
          6 kB
        5. LB3D-Wxpp64sp2_05_09_13.txt
          7 kB
        6. LB3D-MASTER.txt
          41 kB
        7. LB3D-Wxpp32sp3.txt
          89 kB
        8. LB3D-Wxpp64sp2.txt
          92 kB

          [JENKINS-19445] Jobs randomly stuck with "building remotely on slave-name" message

          Pavel Kudrys added a comment - - edited

          I've added also master thread dump.

          And one more weird thing I just noticed on 64bit machine, which just stuck again.
          At some point, Jenkins console says this this...
          ←[33mSep 04, 2013 12:52:34 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescr
          iptor monitor
          WARNING: Failed to monitor LB3D-Wxpp64sp2 for Free Swap Space
          java.util.concurrent.TimeoutException
          at hudson.remoting.Request$1.get(Request.java:275)
          at hudson.remoting.Request$1.get(Request.java:210)
          at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(Abstr
          actAsyncNodeMonitorDescriptor.java:97)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(Abstrac
          tNodeMonitorDescriptor.java:282)

          The weird thing is, that I've disabled the "Free Swap Space" monitoring in Jenkins Node configuration section. So how the "Free Swap Space" monitoring could fail on that slave machine? Could it be a cause of broken master-slave connection or maybe corrupted threads?

          Pavel Kudrys added a comment - - edited I've added also master thread dump. And one more weird thing I just noticed on 64bit machine, which just stuck again. At some point, Jenkins console says this this... ←[33mSep 04, 2013 12:52:34 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescr iptor monitor WARNING: Failed to monitor LB3D-Wxpp64sp2 for Free Swap Space java.util.concurrent.TimeoutException at hudson.remoting.Request$1.get(Request.java:275) at hudson.remoting.Request$1.get(Request.java:210) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(Abstr actAsyncNodeMonitorDescriptor.java:97) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(Abstrac tNodeMonitorDescriptor.java:282) The weird thing is, that I've disabled the "Free Swap Space" monitoring in Jenkins Node configuration section. So how the "Free Swap Space" monitoring could fail on that slave machine? Could it be a cause of broken master-slave connection or maybe corrupted threads?

          Pavel Kudrys added a comment -

          Hi again. One more piece to the puzzle.

          This is what I saw in log while attempting to run a Jenkins job...

          Sep 05, 2013 7:55:24 AM INFO hudson.slaves.RetentionStrategy$Demand check
          Launching computer LB3D-Wxpp32sp3 as it has been in demand for 1 min 34 sec
          Sep 05, 2013 7:55:24 AM INFO hudson.slaves.RetentionStrategy$Demand check
          Launching computer LB3D-Wxpp64sp2 as it has been in demand for 1 min 34 sec
          Sep 05, 2013 7:55:24 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog
          [LB3D-Wxpp32sp3] Starting Virtual Machine...
          Sep 05, 2013 7:55:24 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog
          [LB3D-Wxpp64sp2] Starting Virtual Machine...
          Sep 05, 2013 7:55:24 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog
          [LB3D-Wxpp32sp3] Powering on VM
          Sep 05, 2013 7:55:27 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog
          [LB3D-Wxpp64sp2] Powering on VM
          Sep 05, 2013 7:55:55 AM INFO hudson.model.AsyncPeriodicWork$1 run
          Started PeriodicBackup
          Sep 05, 2013 7:55:55 AM INFO hudson.model.AsyncPeriodicWork$1 run
          Finished PeriodicBackup. 0 ms
          Sep 05, 2013 7:56:35 AM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
          Accepted connection #1 from /172.28.63.75:1034
          Sep 05, 2013 7:56:54 AM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
          Accepted connection #2 from /172.28.63.95:1047
          Sep 05, 2013 7:56:55 AM INFO hudson.model.AsyncPeriodicWork$1 run
          Started PeriodicBackup
          Sep 05, 2013 7:56:55 AM INFO hudson.model.AsyncPeriodicWork$1 run
          Finished PeriodicBackup. 0 ms
          Sep 05, 2013 7:57:30 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor
          Failed to monitor LB3D-Wxpp64sp2 for Free Swap Space
          java.util.concurrent.TimeoutException
          at hudson.remoting.Request$1.get(Request.java:275)
          at hudson.remoting.Request$1.get(Request.java:210)
          at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282)

          As you can see, two slaves are powered ON and successfully connected. But after few seconds, the second VM LB3D-Wxpp64sp2 experiences a problem with Free Swap Space monitoring and from now on, nothing else happens on that machine. The last line in slave console shows Building remotely on LB3D-Wxpp64sp2 in workspace etc... and the spinning wheel. The other one successfully updates SVN and GIT and continues with building.

          Could this Free Swap Space monitoring error really be a root of the problem? Is there a way to disable this monitoring for good? And yes, as I mentioned in my previous comment, I tried to disable the monitoring in Jenkins Node configuration section but apparently, the monitoring is still active on a problematic VM?

          Pavel Kudrys added a comment - Hi again. One more piece to the puzzle. This is what I saw in log while attempting to run a Jenkins job... Sep 05, 2013 7:55:24 AM INFO hudson.slaves.RetentionStrategy$Demand check Launching computer LB3D-Wxpp32sp3 as it has been in demand for 1 min 34 sec Sep 05, 2013 7:55:24 AM INFO hudson.slaves.RetentionStrategy$Demand check Launching computer LB3D-Wxpp64sp2 as it has been in demand for 1 min 34 sec Sep 05, 2013 7:55:24 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog [LB3D-Wxpp32sp3] Starting Virtual Machine... Sep 05, 2013 7:55:24 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog [LB3D-Wxpp64sp2] Starting Virtual Machine... Sep 05, 2013 7:55:24 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog [LB3D-Wxpp32sp3] Powering on VM Sep 05, 2013 7:55:27 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog [LB3D-Wxpp64sp2] Powering on VM Sep 05, 2013 7:55:55 AM INFO hudson.model.AsyncPeriodicWork$1 run Started PeriodicBackup Sep 05, 2013 7:55:55 AM INFO hudson.model.AsyncPeriodicWork$1 run Finished PeriodicBackup. 0 ms Sep 05, 2013 7:56:35 AM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run Accepted connection #1 from /172.28.63.75:1034 Sep 05, 2013 7:56:54 AM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run Accepted connection #2 from /172.28.63.95:1047 Sep 05, 2013 7:56:55 AM INFO hudson.model.AsyncPeriodicWork$1 run Started PeriodicBackup Sep 05, 2013 7:56:55 AM INFO hudson.model.AsyncPeriodicWork$1 run Finished PeriodicBackup. 0 ms Sep 05, 2013 7:57:30 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor Failed to monitor LB3D-Wxpp64sp2 for Free Swap Space java.util.concurrent.TimeoutException at hudson.remoting.Request$1.get(Request.java:275) at hudson.remoting.Request$1.get(Request.java:210) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282) As you can see, two slaves are powered ON and successfully connected. But after few seconds, the second VM LB3D-Wxpp64sp2 experiences a problem with Free Swap Space monitoring and from now on, nothing else happens on that machine. The last line in slave console shows Building remotely on LB3D-Wxpp64sp2 in workspace etc... and the spinning wheel. The other one successfully updates SVN and GIT and continues with building. Could this Free Swap Space monitoring error really be a root of the problem? Is there a way to disable this monitoring for good? And yes, as I mentioned in my previous comment, I tried to disable the monitoring in Jenkins Node configuration section but apparently, the monitoring is still active on a problematic VM?

          Pavel Kudrys added a comment - - edited

          OK, I'm now even more convinced that the free swap space monitoring has something to do with my problem. During today's night build, the error message appeared on both slaves and sure enough, building on both machines got stuck on "Building remotely on slave-name..." and spinning wheel.

          Sep 06, 2013 1:28:15 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor

          Failed to monitor LB3D-Wxpp64sp2 for Free Swap Space
          java.util.concurrent.TimeoutException
          at hudson.remoting.Request$1.get(Request.java:275)
          at hudson.remoting.Request$1.get(Request.java:210)
          at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282)

          Sep 06, 2013 2:08:39 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor

          Failed to monitor LB3D-Wxpp32sp3 for Free Disk Space
          java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
          at hudson.remoting.Request$1.get(Request.java:278)
          at hudson.remoting.Request$1.get(Request.java:210)
          at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282)
          Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
          at hudson.remoting.Request.abort(Request.java:299)
          at hudson.remoting.Channel.terminate(Channel.java:774)
          at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
          Caused by: java.net.SocketException: Connection reset
          at java.net.SocketInputStream.read(Unknown Source)
          at java.net.SocketInputStream.read(Unknown Source)
          at java.io.BufferedInputStream.fill(Unknown Source)
          at java.io.BufferedInputStream.read(Unknown Source)
          at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
          at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
          at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
          at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
          at java.io.ObjectInputStream.readObject0(Unknown Source)
          at java.io.ObjectInputStream.readObject(Unknown Source)
          at hudson.remoting.Command.readFrom(Command.java:92)
          at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
          at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

          Any idea how to kill this free swap space monitoring for good?

          Pavel Kudrys added a comment - - edited OK, I'm now even more convinced that the free swap space monitoring has something to do with my problem. During today's night build, the error message appeared on both slaves and sure enough, building on both machines got stuck on "Building remotely on slave-name..." and spinning wheel. Sep 06, 2013 1:28:15 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor Failed to monitor LB3D-Wxpp64sp2 for Free Swap Space java.util.concurrent.TimeoutException at hudson.remoting.Request$1.get(Request.java:275) at hudson.remoting.Request$1.get(Request.java:210) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282) Sep 06, 2013 2:08:39 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor Failed to monitor LB3D-Wxpp32sp3 for Free Disk Space java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset at hudson.remoting.Request$1.get(Request.java:278) at hudson.remoting.Request$1.get(Request.java:210) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282) Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset at hudson.remoting.Request.abort(Request.java:299) at hudson.remoting.Channel.terminate(Channel.java:774) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69) Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77) at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) Any idea how to kill this free swap space monitoring for good?

          Pavel Kudrys added a comment -

          OK. One more addition to the problem. I reconfigured both slaves to keep them online as much as possible and so far (5 days), no problem with job start.

          Unfortuantely, from the long perspective, this is not a very good solution. Slaves which are still online eat the server memory even if there is no need for them to stay online. Therefore, we would like to have them online only on demand and then shut them off after the build and test is finished. But this is currently impossible because of problem with randomly stuck slaves during their start (most probably caused by "free swap space monitoring). Anyone please?

          Pavel Kudrys added a comment - OK. One more addition to the problem. I reconfigured both slaves to keep them online as much as possible and so far (5 days), no problem with job start. Unfortuantely, from the long perspective, this is not a very good solution. Slaves which are still online eat the server memory even if there is no need for them to stay online. Therefore, we would like to have them online only on demand and then shut them off after the build and test is finished. But this is currently impossible because of problem with randomly stuck slaves during their start (most probably caused by "free swap space monitoring). Anyone please?

          Pavel Kudrys added a comment -

          Anyone please? The workaround with always enabled slaves works quite OK, but I hate to have the slaves permanently running. It's relative OK in case of just two slaves, but I'm planning to add few more slaves and then it would become a real problem.

          Pavel Kudrys added a comment - Anyone please? The workaround with always enabled slaves works quite OK, but I hate to have the slaves permanently running. It's relative OK in case of just two slaves, but I'm planning to add few more slaves and then it would become a real problem.

          Pavel Kudrys added a comment -

          One month later, once again updated Jenkins and all plugins, but nothing has changed. Jobs still randomly stuck with "building remotely on slave-name" message. Anyone please? Am I alone with such kind of issue?

          Pavel Kudrys added a comment - One month later, once again updated Jenkins and all plugins, but nothing has changed. Jobs still randomly stuck with "building remotely on slave-name" message. Anyone please? Am I alone with such kind of issue?

          Pavel Kudrys added a comment -

          It happened again today Any idea? Anyone please...

          This is what I see in jenkins-slave.err.log:

          Oct 31, 2013 7:41:16 AM hudson.remoting.jnlp.Main$CuiListener <init>
          INFO: Jenkins agent is running in headless mode.
          Oct 31, 2013 7:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Locating server among http://tcskjira:8095/
          Oct 31, 2013 7:41:17 AM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Connecting to tcskjira:1695
          Oct 31, 2013 7:41:17 AM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Handshaking
          Oct 31, 2013 7:41:17 AM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Connected

          This is what I see in jenkins-slave.wrapper.log:

          2013-10-31 07:41:09 - Starting C:\Program Files\Java\jre7\bin\java.exe -Xrs -jar "C:\Jenkins\slave.jar" -jnlpUrl http://tcskjira:8095/computer/LB3D-Wxpp64sp2/slave-agent.jnlp -secret 1ba0fb4096bc76c5ef1332c6368cd286a76074134a7091cd7603e93ec938b060
          2013-10-31 07:41:09 - Started 1576

          This is what I see in Jenkins Log:

          Oct 31, 2013 7:40:34 AM INFO hudson.slaves.RetentionStrategy$Demand check

          Launching computer LB3D-Wxpp64sp2 as it has been in demand for 39 sec

          Oct 31, 2013 7:40:34 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog

          [LB3D-Wxpp64sp2] Starting Virtual Machine...

          Oct 31, 2013 7:40:35 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog

          [LB3D-Wxpp64sp2] Powering on VM

          Oct 31, 2013 7:40:51 AM INFO hudson.model.AsyncPeriodicWork$1 run

          Started PeriodicBackup

          Oct 31, 2013 7:40:51 AM INFO hudson.model.AsyncPeriodicWork$1 run

          Finished PeriodicBackup. 0 ms

          Oct 31, 2013 7:41:16 AM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run

          Accepted connection #40 from /172.28.63.95:1034

          Oct 31, 2013 7:41:23 AM WARNING hudson.node_monitors.ResponseTimeMonitor$1 monitor

          Making CAAV5R19x64 offline because it’s not responding

          Oct 31, 2013 7:41:23 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog

          [CAAV5R19x64] Already disconnecting on a separate thread

          Oct 31, 2013 7:41:51 AM INFO hudson.model.AsyncPeriodicWork$1 run

          Started PeriodicBackup

          Oct 31, 2013 7:41:51 AM INFO hudson.model.AsyncPeriodicWork$1 run

          Finished PeriodicBackup. 0 ms

          Oct 31, 2013 7:41:52 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor

          Failed to monitor LB3D-Wxpp64sp2 for Free Swap Space
          java.util.concurrent.TimeoutException
          at hudson.remoting.Request$1.get(Request.java:275)
          at hudson.remoting.Request$1.get(Request.java:210)
          at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282)

          Slave Log says this:

          JNLP agent connected from /172.28.63.95
          <===[JENKINS REMOTING CAPACITY]===>Slave.jar version: 2.32
          This is a Windows slave
          Slave successfully connected and online
          followed by spinning wheel...

          Slave Thread Dump:

          Channel reader thread: channel

          "Channel reader thread: channel" Id=11 Group=main RUNNABLE (in native)
          at java.net.SocketInputStream.socketRead0(Native Method)
          at java.net.SocketInputStream.read(Unknown Source)
          at java.net.SocketInputStream.read(Unknown Source)
          at java.io.BufferedInputStream.fill(Unknown Source)
          at java.io.BufferedInputStream.read(Unknown Source)

          • locked java.io.BufferedInputStream@1c2ba649
            at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
            at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
            at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
            at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
            at java.io.ObjectInputStream.readObject0(Unknown Source)
            at java.io.ObjectInputStream.readObject(Unknown Source)
            at hudson.remoting.Command.readFrom(Command.java:92)
            at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

          main

          "main" Id=1 Group=main WAITING on hudson.remoting.Engine@7a0136b0
          at java.lang.Object.wait(Native Method)

          • waiting on hudson.remoting.Engine@7a0136b0
            at java.lang.Thread.join(Unknown Source)
            at java.lang.Thread.join(Unknown Source)
            at hudson.remoting.jnlp.Main.main(Main.java:137)
            at hudson.remoting.jnlp.Main._main(Main.java:130)
            at hudson.remoting.Launcher.run(Launcher.java:223)
            at hudson.remoting.Launcher.main(Launcher.java:192)

          Ping thread for channel hudson.remoting.Channel@28509953:channel

          "Ping thread for channel hudson.remoting.Channel@28509953:channel" Id=13 Group=main TIMED_WAITING
          at java.lang.Thread.sleep(Native Method)
          at hudson.remoting.PingThread.run(PingThread.java:86)

          pool-1-thread-17

          "pool-1-thread-17" Id=29 Group=main RUNNABLE
          at sun.management.ThreadImpl.dumpThreads0(Native Method)
          at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
          at hudson.Functions.getThreadInfos(Functions.java:1131)
          at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96)
          at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
          at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:326)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at hudson.remoting.Engine$1$1.run(Engine.java:63)
          at java.lang.Thread.run(Unknown Source)

          Number of locked synchronizers = 1

          • java.util.concurrent.ThreadPoolExecutor$Worker@493df066

          pool-1-thread-2

          "pool-1-thread-2" Id=14 Group=main RUNNABLE
          at com.sun.jna.Native.initIDs(Native Method)
          at com.sun.jna.Native.<clinit>(Native.java:122)
          at org.tmatesoft.svn.core.internal.util.jna.JNALibraryLoader.<clinit>(JNALibraryLoader.java:50)
          at org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:93)
          at org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196)
          at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1869)
          at org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57)
          at hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:975)
          at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:952)
          at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:885)
          at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:872)
          at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2420)
          at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:326)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at hudson.remoting.Engine$1$1.run(Engine.java:63)
          at java.lang.Thread.run(Unknown Source)

          Number of locked synchronizers = 1

          • java.util.concurrent.ThreadPoolExecutor$Worker@5133066c

          pool-1-thread-5

          "pool-1-thread-5" Id=17 Group=main RUNNABLE
          at com.sun.jna.Structure.<clinit>(Structure.java:134)
          at org.jvnet.hudson.Windows.monitor(Windows.java:40)
          at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:110)
          at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:100)
          at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:326)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at hudson.remoting.Engine$1$1.run(Engine.java:63)
          at java.lang.Thread.run(Unknown Source)

          Number of locked synchronizers = 1

          • java.util.concurrent.ThreadPoolExecutor$Worker@786dee4c

          Thread-1

          "Thread-1" Id=10 Group=main WAITING on hudson.remoting.Channel@28509953
          at java.lang.Object.wait(Native Method)

          • waiting on hudson.remoting.Channel@28509953
            at java.lang.Object.wait(Object.java:503)
            at hudson.remoting.Channel.join(Channel.java:842)
            at hudson.remoting.Engine.run(Engine.java:271)

          Attach Listener

          "Attach Listener" Id=4 Group=system RUNNABLE

          Finalizer

          "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@75b21111
          at java.lang.Object.wait(Native Method)

          • waiting on java.lang.ref.ReferenceQueue$Lock@75b21111
            at java.lang.ref.ReferenceQueue.remove(Unknown Source)
            at java.lang.ref.ReferenceQueue.remove(Unknown Source)
            at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

          Reference Handler

          "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@7d7eaf74
          at java.lang.Object.wait(Native Method)

          • waiting on java.lang.ref.Reference$Lock@7d7eaf74
            at java.lang.Object.wait(Object.java:503)
            at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)

          System Thread Dump:

          Thread Dump
          AWT-Windows

          "AWT-Windows" Id=614 Group=main RUNNABLE (in native)
          at sun.awt.windows.WToolkit.eventLoop(Native Method)
          at sun.awt.windows.WToolkit.run(Unknown Source)
          at java.lang.Thread.run(Unknown Source)

          Channel reader thread: LB3D-Wxpp32sp3

          "Channel reader thread: LB3D-Wxpp32sp3" Id=12151 Group=main RUNNABLE
          at java.net.SocketInputStream.socketRead0(Native Method)
          at java.net.SocketInputStream.read(Unknown Source)
          at java.net.SocketInputStream.read(Unknown Source)
          at java.io.BufferedInputStream.fill(Unknown Source)
          at java.io.BufferedInputStream.read(Unknown Source)

          • locked java.io.BufferedInputStream@7a7ec776
            at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
            at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
            at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
            at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
            at java.io.ObjectInputStream.readObject0(Unknown Source)
            at java.io.ObjectInputStream.readObject(Unknown Source)
            at hudson.remoting.Command.readFrom(Command.java:92)
            at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

          Channel reader thread: LB3D-Wxpp64sp2

          "Channel reader thread: LB3D-Wxpp64sp2" Id=12178 Group=main RUNNABLE
          at java.net.SocketInputStream.socketRead0(Native Method)
          at java.net.SocketInputStream.read(Unknown Source)
          at java.net.SocketInputStream.read(Unknown Source)
          at java.io.BufferedInputStream.fill(Unknown Source)
          at java.io.BufferedInputStream.read(Unknown Source)

          • locked java.io.BufferedInputStream@764058b0
            at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
            at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
            at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
            at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
            at java.io.ObjectInputStream.readObject0(Unknown Source)
            at java.io.ObjectInputStream.readObject(Unknown Source)
            at hudson.remoting.Command.readFrom(Command.java:92)
            at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

          Channel reader thread: Slave-TestProcessBuild

          "Channel reader thread: Slave-TestProcessBuild" Id=6006 Group=main RUNNABLE
          at java.net.SocketInputStream.socketRead0(Native Method)
          at java.net.SocketInputStream.read(Unknown Source)
          at java.net.SocketInputStream.read(Unknown Source)
          at java.io.BufferedInputStream.fill(Unknown Source)
          at java.io.BufferedInputStream.read(Unknown Source)

          • locked java.io.BufferedInputStream@3563374a
            at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
            at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
            at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
            at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
            at java.io.ObjectInputStream.readObject0(Unknown Source)
            at java.io.ObjectInputStream.readObject(Unknown Source)
            at hudson.remoting.Command.readFrom(Command.java:92)
            at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

          Channel reader thread: Slave-TestProcessTest

          "Channel reader thread: Slave-TestProcessTest" Id=6121 Group=main RUNNABLE
          at java.net.SocketInputStream.socketRead0(Native Method)
          at java.net.SocketInputStream.read(Unknown Source)
          at java.net.SocketInputStream.read(Unknown Source)
          at java.io.BufferedInputStream.fill(Unknown Source)
          at java.io.BufferedInputStream.read(Unknown Source)

          • locked java.io.BufferedInputStream@374b5f62
            at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
            at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
            at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
            at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
            at java.io.ObjectInputStream.readObject0(Unknown Source)
            at java.io.ObjectInputStream.readObject(Unknown Source)
            at hudson.remoting.Command.readFrom(Command.java:92)
            at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

          DestroyJavaVM

          "DestroyJavaVM" Id=19 Group=main RUNNABLE

          Executor #0 for LB3D-Wxpp32sp3 : executing ipprotection_32_64 #171

          "Executor #0 for LB3D-Wxpp32sp3 : executing ipprotection_32_64 #171" Id=10955 Group=main TIMED_WAITING
          at java.lang.Thread.sleep(Native Method)
          at hudson.matrix.DefaultMatrixExecutionStrategyImpl.waitForCompletion(DefaultMatrixExecutionStrategyImpl.java:290)
          at hudson.matrix.DefaultMatrixExecutionStrategyImpl.run(DefaultMatrixExecutionStrategyImpl.java:141)
          at hudson.matrix.MatrixBuild$MatrixBuildExecution.doRun(MatrixBuild.java:355)
          at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:562)
          at hudson.model.Run.execute(Run.java:1665)
          at hudson.matrix.MatrixBuild.run(MatrixBuild.java:304)
          at hudson.model.ResourceController.execute(ResourceController.java:88)
          at hudson.model.Executor.run(Executor.java:230)

          Executor #0 for LB3D-Wxpp64sp2 : executing ipprotection_32_64 » LB3D-Wxpp64sp2 #171 / waiting for hudson.remoting.Channel@59f73f53:LB3D-Wxpp64sp2

          "Executor #0 for LB3D-Wxpp64sp2 : executing ipprotection_32_64 » LB3D-Wxpp64sp2 #171 / waiting for hudson.remoting.Channel@59f73f53:LB3D-Wxpp64sp2" Id=11106 Group=main TIMED_WAITING on hudson.remoting.UserRequest@3690a4a8
          at java.lang.Object.wait(Native Method)

          • waiting on hudson.remoting.UserRequest@3690a4a8
            at hudson.remoting.Request.call(Request.java:146)
            at hudson.remoting.Channel.call(Channel.java:714)
            at hudson.FilePath.act(FilePath.java:901)
            at hudson.FilePath.act(FilePath.java:885)
            at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:848)
            at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:786)
            at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:117)
            at hudson.model.AbstractProject.checkout(AbstractProject.java:1411)
            at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:652)
            at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
            at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:557)
            at hudson.model.Run.execute(Run.java:1665)
            at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
            at hudson.model.ResourceController.execute(ResourceController.java:88)
            at hudson.model.Executor.run(Executor.java:230)

          Handling GET /threadDump : RequestHandlerThread595

          "Handling GET /threadDump : RequestHandlerThread595" Id=12350 Group=main TIMED_WAITING on java.util.concurrent.FutureTask$Sync@4deadcb6
          at sun.misc.Unsafe.park(Native Method)

          • waiting on java.util.concurrent.FutureTask$Sync@4deadcb6
            at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
            at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(Unknown Source)
            at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(Unknown Source)
            at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
            at java.util.concurrent.FutureTask.get(Unknown Source)
            at hudson.remoting.LocalChannel$2.get(LocalChannel.java:81)
            at jenkins.model.Jenkins.getAllThreadDumps(Jenkins.java:2919)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
            at java.lang.reflect.Method.invoke(Unknown Source)
            at org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258)
            at org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104)
            at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
            at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
            at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
            at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
            at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
            at org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61)
            at org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsIterator(ExpressionSupport.java:94)
            at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:89)
            at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
            at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
            at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
            at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
            at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
            at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
            at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
            at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
            at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
            at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
            at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
            at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
            at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
            at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
            at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95)
            at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:727)
            at org.kohsuke.stapler.Stapler.invoke(Stapler.java:841)
            at org.kohsuke.stapler.Stapler.invoke(Stapler.java:630)
            at org.kohsuke.stapler.Stapler.service(Stapler.java:224)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
            at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
            at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
            at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
            at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:208)
            at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:181)
            at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86)
            at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84)
            at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
            at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
            at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
            at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
            at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
            at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:64)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
            at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
            at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
            at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
            at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
            at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
            at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
            at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
            at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
            at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
            at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
            at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
            at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
            at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
            at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
            at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
            at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
            at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
            at org.eclipse.jetty.server.Server.handle(Server.java:370)
            at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
            at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
            at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
            at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
            at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
            at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
            at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
            at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
            at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

          Number of locked synchronizers = 1

          • java.util.concurrent.ThreadPoolExecutor$Worker@620c96ba

          HashSessionScavenger-0

          "HashSessionScavenger-0" Id=11 Group=main TIMED_WAITING on java.util.TaskQueue@4a7cb042
          at java.lang.Object.wait(Native Method)

          • waiting on java.util.TaskQueue@4a7cb042
            at java.util.TimerThread.mainLoop(Unknown Source)
            at java.util.TimerThread.run(Unknown Source)

          javamelody

          "javamelody" Id=27 Group=main TIMED_WAITING on java.util.TaskQueue@1b9a5f49
          at java.lang.Object.wait(Native Method)

          • waiting on java.util.TaskQueue@1b9a5f49
            at java.util.TimerThread.mainLoop(Unknown Source)
            at java.util.TimerThread.run(Unknown Source)

          Jenkins cron thread

          "Jenkins cron thread" Id=14 Group=main TIMED_WAITING on java.util.TaskQueue@258e37c9
          at java.lang.Object.wait(Native Method)

          • waiting on java.util.TaskQueue@258e37c9
            at java.util.TimerThread.mainLoop(Unknown Source)
            at java.util.TimerThread.run(Unknown Source)

          Jenkins UDP 33848 monitoring thread

          "Jenkins UDP 33848 monitoring thread" Id=31 Group=main RUNNABLE (in native)
          at java.net.TwoStacksPlainDatagramSocketImpl.receive0(Native Method)

          • locked java.net.TwoStacksPlainDatagramSocketImpl@1f036f93
            at java.net.TwoStacksPlainDatagramSocketImpl.receive(Unknown Source)
          • locked java.net.TwoStacksPlainDatagramSocketImpl@1f036f93
            at java.net.DatagramSocket.receive(Unknown Source)
          • locked java.net.DatagramPacket@7279ad5b
          • locked java.net.MulticastSocket@12646c14
            at hudson.UDPBroadcastThread.run(UDPBroadcastThread.java:82)

          Jenkins-Remoting-Thread-1218

          "Jenkins-Remoting-Thread-1218" Id=12433 Group=main RUNNABLE
          at sun.management.ThreadImpl.dumpThreads0(Native Method)
          at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
          at hudson.Functions.getThreadInfos(Functions.java:1131)
          at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96)
          at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
          at hudson.remoting.LocalChannel$1.call(LocalChannel.java:52)
          at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at java.lang.Thread.run(Unknown Source)

          Number of locked synchronizers = 1

          • java.util.concurrent.ThreadPoolExecutor$Worker@1904332a

          Jenkins-Remoting-Thread-1219

          "Jenkins-Remoting-Thread-1219" Id=12434 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@2baacf98
          at sun.misc.Unsafe.park(Native Method)

          • waiting on java.util.concurrent.SynchronousQueue$TransferStack@2baacf98
            at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
            at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

          Jenkins-Remoting-Thread-1220

          "Jenkins-Remoting-Thread-1220" Id=12435 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@2baacf98
          at sun.misc.Unsafe.park(Native Method)

          • waiting on java.util.concurrent.SynchronousQueue$TransferStack@2baacf98
            at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
            at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

          JmDNS(tcskjira.local.).State.Timer

          "JmDNS(tcskjira.local.).State.Timer" Id=48 Group=main TIMED_WAITING on java.util.TaskQueue@587c9b22
          at java.lang.Object.wait(Native Method)

          • waiting on java.util.TaskQueue@587c9b22
            at java.util.TimerThread.mainLoop(Unknown Source)
            at java.util.TimerThread.run(Unknown Source)

          JmDNS(tcskjira.local.).Timer

          "JmDNS(tcskjira.local.).Timer" Id=47 Group=main TIMED_WAITING on java.util.TaskQueue@5cad0290
          at java.lang.Object.wait(Native Method)

          • waiting on java.util.TaskQueue@5cad0290
            at java.util.TimerThread.mainLoop(Unknown Source)
            at java.util.TimerThread.run(Unknown Source)

          jnr.ffi.util.ref.internal.Finalizer

          "jnr.ffi.util.ref.internal.Finalizer" Id=5472 Group=main WAITING on java.lang.ref.ReferenceQueue$Lock@4c2f6a8
          at java.lang.Object.wait(Native Method)

          • waiting on java.lang.ref.ReferenceQueue$Lock@4c2f6a8
            at java.lang.ref.ReferenceQueue.remove(Unknown Source)
            at java.lang.ref.ReferenceQueue.remove(Unknown Source)
            at jnr.ffi.util.ref.internal.Finalizer.run(Finalizer.java:128)

          LauncherControlThread[ControlPort=-1]

          "LauncherControlThread[ControlPort=-1]" Id=17 Group=main TIMED_WAITING
          at java.lang.Thread.sleep(Native Method)
          at winstone.Launcher.run(Launcher.java:245)
          at java.lang.Thread.run(Unknown Source)

          NioSocketAcceptor-1

          "NioSocketAcceptor-1" Id=29 Group=main RUNNABLE (in native)
          at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
          at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
          at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
          at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
          at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)

          • locked sun.nio.ch.Util$2@5feec26b
          • locked java.util.Collections$UnmodifiableSet@29e1ac62
          • locked sun.nio.ch.WindowsSelectorImpl@4defca63
            at sun.nio.ch.SelectorImpl.select(Unknown Source)
            at sun.nio.ch.SelectorImpl.select(Unknown Source)
            at org.apache.mina.transport.socket.nio.NioSocketAcceptor.select(NioSocketAcceptor.java:238)
            at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:432)
            at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

          Number of locked synchronizers = 1

          • java.util.concurrent.ThreadPoolExecutor$Worker@29090dda

          Ping thread for channel hudson.remoting.Channel@1675652c:Slave-TestProcessTest

          "Ping thread for channel hudson.remoting.Channel@1675652c:Slave-TestProcessTest" Id=6122 Group=main TIMED_WAITING
          at java.lang.Thread.sleep(Native Method)
          at hudson.remoting.PingThread.run(PingThread.java:86)

          Ping thread for channel hudson.remoting.Channel@50923d0b:Slave-TestProcessBuild

          "Ping thread for channel hudson.remoting.Channel@50923d0b:Slave-TestProcessBuild" Id=6007 Group=main TIMED_WAITING
          at java.lang.Thread.sleep(Native Method)
          at hudson.remoting.PingThread.run(PingThread.java:86)

          Ping thread for channel hudson.remoting.Channel@59f73f53:LB3D-Wxpp64sp2

          "Ping thread for channel hudson.remoting.Channel@59f73f53:LB3D-Wxpp64sp2" Id=12179 Group=main TIMED_WAITING
          at java.lang.Thread.sleep(Native Method)
          at hudson.remoting.PingThread.run(PingThread.java:86)

          Ping thread for channel hudson.remoting.Channel@5ade8f7a:LB3D-Wxpp32sp3

          "Ping thread for channel hudson.remoting.Channel@5ade8f7a:LB3D-Wxpp32sp3" Id=12152 Group=main TIMED_WAITING
          at java.lang.Thread.sleep(Native Method)
          at hudson.remoting.PingThread.run(PingThread.java:86)

          pool-1-svnkit-thread-1

          "pool-1-svnkit-thread-1" Id=411 Group=main WAITING on java.util.concurrent.SynchronousQueue$TransferStack@1bcf9209
          at sun.misc.Unsafe.park(Native Method)

          • waiting on java.util.concurrent.SynchronousQueue$TransferStack@1bcf9209
            at java.util.concurrent.locks.LockSupport.park(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
            at java.util.concurrent.SynchronousQueue.take(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

          pool-1-svnkit-thread-2

          "pool-1-svnkit-thread-2" Id=2322 Group=main WAITING on java.util.concurrent.SynchronousQueue$TransferStack@1bcf9209
          at sun.misc.Unsafe.park(Native Method)

          • waiting on java.util.concurrent.SynchronousQueue$TransferStack@1bcf9209
            at java.util.concurrent.locks.LockSupport.park(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
            at java.util.concurrent.SynchronousQueue.take(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

          pool-4-thread-1272

          "pool-4-thread-1272" Id=12398 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9
          at sun.misc.Unsafe.park(Native Method)

          • waiting on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9
            at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
            at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

          pool-4-thread-1273

          "pool-4-thread-1273" Id=12399 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9
          at sun.misc.Unsafe.park(Native Method)

          • waiting on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9
            at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
            at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

          pool-4-thread-1274

          "pool-4-thread-1274" Id=12417 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9
          at sun.misc.Unsafe.park(Native Method)

          • waiting on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9
            at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
            at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

          pool-4-thread-1275

          "pool-4-thread-1275" Id=12418 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9
          at sun.misc.Unsafe.park(Native Method)

          • waiting on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9
            at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
            at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

          pool-9-thread-1

          "pool-9-thread-1" Id=44 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2ba02595
          at sun.misc.Unsafe.park(Native Method)

          • waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2ba02595
            at java.util.concurrent.locks.LockSupport.park(Unknown Source)
            at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
            at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

          RequestHandlerThread1 Selector0

          "RequestHandlerThread1 Selector0" Id=15 Group=main RUNNABLE (in native)
          at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
          at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
          at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
          at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
          at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)

          • locked sun.nio.ch.Util$2@2fea3452
          • locked java.util.Collections$UnmodifiableSet@e2f4d5c
          • locked sun.nio.ch.WindowsSelectorImpl@7ed452e4
            at sun.nio.ch.SelectorImpl.select(Unknown Source)
            at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
            at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
            at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

          Number of locked synchronizers = 1

          • java.util.concurrent.ThreadPoolExecutor$Worker@4da2d209

          RequestHandlerThread2 Acceptor0 SelectChannelConnector@0.0.0.0:8095

          "RequestHandlerThread2 Acceptor0 SelectChannelConnector@0.0.0.0:8095" Id=16 Group=main RUNNABLE (in native)
          at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
          at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)

          • locked java.lang.Object@33e5bfc7
            at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
            at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
            at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

          Number of locked synchronizers = 1

          • java.util.concurrent.ThreadPoolExecutor$Worker@5db00667

          RequestHandlerThread603

          "RequestHandlerThread603" Id=12388 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d
          at sun.misc.Unsafe.park(Native Method)

          • waiting on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d
            at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
            at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

          RequestHandlerThread605

          "RequestHandlerThread605" Id=12391 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d
          at sun.misc.Unsafe.park(Native Method)

          • waiting on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d
            at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
            at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

          RequestHandlerThread607

          "RequestHandlerThread607" Id=12427 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d
          at sun.misc.Unsafe.park(Native Method)

          • waiting on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d
            at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
            at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

          RequestHandlerThread608

          "RequestHandlerThread608" Id=12431 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d
          at sun.misc.Unsafe.park(Native Method)

          • waiting on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d
            at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
            at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

          SocketListener(tcskjira.local.)

          "SocketListener(tcskjira.local.)" Id=46 Group=main RUNNABLE (in native)
          at java.net.TwoStacksPlainDatagramSocketImpl.receive0(Native Method)

          • locked java.net.TwoStacksPlainDatagramSocketImpl@62836b78
            at java.net.TwoStacksPlainDatagramSocketImpl.receive(Unknown Source)
          • locked java.net.TwoStacksPlainDatagramSocketImpl@62836b78
            at java.net.DatagramSocket.receive(Unknown Source)
          • locked java.net.DatagramPacket@6756aa7b
          • locked java.net.MulticastSocket@6ad2a63d
            at javax.jmdns.impl.SocketListener.run(SocketListener.java:41)

          TCP slave agent listener port=0

          "TCP slave agent listener port=0" Id=30 Group=main RUNNABLE (in native)
          at java.net.TwoStacksPlainSocketImpl.socketAccept(Native Method)
          at java.net.AbstractPlainSocketImpl.accept(Unknown Source)
          at java.net.PlainSocketImpl.accept(Unknown Source)

          • locked java.net.SocksSocketImpl@7ef39e7a
            at java.net.ServerSocket.implAccept(Unknown Source)
            at java.net.ServerSocket.accept(Unknown Source)
            at hudson.TcpSlaveAgentListener.run(TcpSlaveAgentListener.java:93)

          ViewJob reload thread

          "ViewJob reload thread" Id=28 Group=main TIMED_WAITING on java.util.LinkedHashSet@39d0518e
          at java.lang.Object.wait(Native Method)

          • waiting on java.util.LinkedHashSet@39d0518e
            at hudson.model.ViewJob$ReloadThread.getNext(ViewJob.java:176)
            at hudson.model.ViewJob$ReloadThread.run(ViewJob.java:193)

          Waiting to acquire c:\Jenkins\workspace\ipprotection_32_64 : SCM polling for hudson.matrix.MatrixProject@3b277576[ipprotection_32_64]

          "Waiting to acquire c:\Jenkins\workspace\ipprotection_32_64 : SCM polling for hudson.matrix.MatrixProject@3b277576[ipprotection_32_64]" Id=12065 Group=main WAITING on hudson.slaves.WorkspaceList@195952ce
          at java.lang.Object.wait(Native Method)

          • waiting on hudson.slaves.WorkspaceList@195952ce
            at java.lang.Object.wait(Object.java:503)
            at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:232)
            at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:211)
            at hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1580)
            at hudson.model.AbstractProject._poll(AbstractProject.java:1558)
            at hudson.model.AbstractProject.poll(AbstractProject.java:1490)
            at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:439)
            at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:468)
            at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
            at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
            at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

          Number of locked synchronizers = 1

          • java.util.concurrent.ThreadPoolExecutor$Worker@3435c217

          Attach Listener

          "Attach Listener" Id=5 Group=system RUNNABLE

          Finalizer

          "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@4017e342
          at java.lang.Object.wait(Native Method)

          • waiting on java.lang.ref.ReferenceQueue$Lock@4017e342
            at java.lang.ref.ReferenceQueue.remove(Unknown Source)
            at java.lang.ref.ReferenceQueue.remove(Unknown Source)
            at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

          Java2D Disposer

          "Java2D Disposer" Id=12 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@1bf08062
          at java.lang.Object.wait(Native Method)

          • waiting on java.lang.ref.ReferenceQueue$Lock@1bf08062
            at java.lang.ref.ReferenceQueue.remove(Unknown Source)
            at java.lang.ref.ReferenceQueue.remove(Unknown Source)
            at sun.java2d.Disposer.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

          Reference Handler

          "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@604922ed
          at java.lang.Object.wait(Native Method)

          • waiting on java.lang.ref.Reference$Lock@604922ed
            at java.lang.Object.wait(Object.java:503)
            at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)

          Signal Dispatcher

          "Signal Dispatcher" Id=4 Group=system RUNNABLE

          CAAV5R19x64
          N/A

          offline

          LB3D-Wxpp64sp2
          Channel reader thread: channel

          "Channel reader thread: channel" Id=11 Group=main RUNNABLE (in native)
          at java.net.SocketInputStream.socketRead0(Native Method)
          at java.net.SocketInputStream.read(Unknown Source)
          at java.net.SocketInputStream.read(Unknown Source)
          at java.io.BufferedInputStream.fill(Unknown Source)
          at java.io.BufferedInputStream.read(Unknown Source)

          • locked java.io.BufferedInputStream@1c2ba649
            at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
            at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
            at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
            at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
            at java.io.ObjectInputStream.readObject0(Unknown Source)
            at java.io.ObjectInputStream.readObject(Unknown Source)
            at hudson.remoting.Command.readFrom(Command.java:92)
            at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

          main

          "main" Id=1 Group=main WAITING on hudson.remoting.Engine@7a0136b0
          at java.lang.Object.wait(Native Method)

          • waiting on hudson.remoting.Engine@7a0136b0
            at java.lang.Thread.join(Unknown Source)
            at java.lang.Thread.join(Unknown Source)
            at hudson.remoting.jnlp.Main.main(Main.java:137)
            at hudson.remoting.jnlp.Main._main(Main.java:130)
            at hudson.remoting.Launcher.run(Launcher.java:223)
            at hudson.remoting.Launcher.main(Launcher.java:192)

          Ping thread for channel hudson.remoting.Channel@28509953:channel

          "Ping thread for channel hudson.remoting.Channel@28509953:channel" Id=13 Group=main TIMED_WAITING
          at java.lang.Thread.sleep(Native Method)
          at hudson.remoting.PingThread.run(PingThread.java:86)

          pool-1-thread-18

          "pool-1-thread-18" Id=30 Group=main RUNNABLE
          at sun.management.ThreadImpl.dumpThreads0(Native Method)
          at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
          at hudson.Functions.getThreadInfos(Functions.java:1131)
          at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96)
          at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
          at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:326)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at hudson.remoting.Engine$1$1.run(Engine.java:63)
          at java.lang.Thread.run(Unknown Source)

          Number of locked synchronizers = 1

          • java.util.concurrent.ThreadPoolExecutor$Worker@59d9c141

          pool-1-thread-2

          "pool-1-thread-2" Id=14 Group=main RUNNABLE
          at com.sun.jna.Native.initIDs(Native Method)
          at com.sun.jna.Native.<clinit>(Native.java:122)
          at org.tmatesoft.svn.core.internal.util.jna.JNALibraryLoader.<clinit>(JNALibraryLoader.java:50)
          at org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:93)
          at org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196)
          at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1869)
          at org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57)
          at hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:975)
          at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:952)
          at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:885)
          at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:872)
          at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2420)
          at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:326)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at hudson.remoting.Engine$1$1.run(Engine.java:63)
          at java.lang.Thread.run(Unknown Source)

          Number of locked synchronizers = 1

          • java.util.concurrent.ThreadPoolExecutor$Worker@5133066c

          pool-1-thread-5

          "pool-1-thread-5" Id=17 Group=main RUNNABLE
          at com.sun.jna.Structure.<clinit>(Structure.java:134)
          at org.jvnet.hudson.Windows.monitor(Windows.java:40)
          at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:110)
          at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:100)
          at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:326)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at hudson.remoting.Engine$1$1.run(Engine.java:63)
          at java.lang.Thread.run(Unknown Source)

          Number of locked synchronizers = 1

          • java.util.concurrent.ThreadPoolExecutor$Worker@786dee4c

          Thread-1

          "Thread-1" Id=10 Group=main WAITING on hudson.remoting.Channel@28509953
          at java.lang.Object.wait(Native Method)

          • waiting on hudson.remoting.Channel@28509953
            at java.lang.Object.wait(Object.java:503)
            at hudson.remoting.Channel.join(Channel.java:842)
            at hudson.remoting.Engine.run(Engine.java:271)

          Attach Listener

          "Attach Listener" Id=4 Group=system RUNNABLE

          Finalizer

          "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@75b21111
          at java.lang.Object.wait(Native Method)

          • waiting on java.lang.ref.ReferenceQueue$Lock@75b21111
            at java.lang.ref.ReferenceQueue.remove(Unknown Source)
            at java.lang.ref.ReferenceQueue.remove(Unknown Source)
            at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

          Reference Handler

          "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@7d7eaf74
          at java.lang.Object.wait(Native Method)

          • waiting on java.lang.ref.Reference$Lock@7d7eaf74
            at java.lang.Object.wait(Object.java:503)
            at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)

          Slave-TestProcessBuild
          Channel reader thread: channel

          "Channel reader thread: channel" Id=11 Group=main RUNNABLE (in native)
          at java.net.SocketInputStream.socketRead0(Native Method)
          at java.net.SocketInputStream.read(Unknown Source)
          at java.net.SocketInputStream.read(Unknown Source)
          at java.io.BufferedInputStream.fill(Unknown Source)
          at java.io.BufferedInputStream.read(Unknown Source)

          • locked java.io.BufferedInputStream@74784829
            at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
            at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
            at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
            at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
            at java.io.ObjectInputStream.readObject0(Unknown Source)
            at java.io.ObjectInputStream.readObject(Unknown Source)
            at hudson.remoting.Command.readFrom(Command.java:92)
            at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

          main

          "main" Id=1 Group=main WAITING on hudson.remoting.Engine@7f25ab41
          at java.lang.Object.wait(Native Method)

          • waiting on hudson.remoting.Engine@7f25ab41
            at java.lang.Thread.join(Unknown Source)
            at java.lang.Thread.join(Unknown Source)
            at hudson.remoting.jnlp.Main.main(Main.java:137)
            at hudson.remoting.jnlp.Main._main(Main.java:130)
            at hudson.remoting.Launcher.run(Launcher.java:223)
            at hudson.remoting.Launcher.main(Launcher.java:192)

          Ping thread for channel hudson.remoting.Channel@1c06d198:channel

          "Ping thread for channel hudson.remoting.Channel@1c06d198:channel" Id=13 Group=main TIMED_WAITING
          at java.lang.Thread.sleep(Native Method)
          at hudson.remoting.PingThread.run(PingThread.java:86)

          pool-1-thread-802

          "pool-1-thread-802" Id=916 Group=main RUNNABLE
          at sun.management.ThreadImpl.dumpThreads0(Native Method)
          at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
          at hudson.Functions.getThreadInfos(Functions.java:1131)
          at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96)
          at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
          at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:326)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at hudson.remoting.Engine$1$1.run(Engine.java:63)
          at java.lang.Thread.run(Unknown Source)

          Number of locked synchronizers = 1

          • java.util.concurrent.ThreadPoolExecutor$Worker@13c5810e

          Thread-1

          "Thread-1" Id=10 Group=main WAITING on hudson.remoting.Channel@1c06d198
          at java.lang.Object.wait(Native Method)

          • waiting on hudson.remoting.Channel@1c06d198
            at java.lang.Object.wait(Object.java:503)
            at hudson.remoting.Channel.join(Channel.java:842)
            at hudson.remoting.Engine.run(Engine.java:271)

          Attach Listener

          "Attach Listener" Id=4 Group=system RUNNABLE

          Finalizer

          "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@4c9b77c
          at java.lang.Object.wait(Native Method)

          • waiting on java.lang.ref.ReferenceQueue$Lock@4c9b77c
            at java.lang.ref.ReferenceQueue.remove(Unknown Source)
            at java.lang.ref.ReferenceQueue.remove(Unknown Source)
            at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

          Reference Handler

          "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@572d3058
          at java.lang.Object.wait(Native Method)

          • waiting on java.lang.ref.Reference$Lock@572d3058
            at java.lang.Object.wait(Object.java:503)
            at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)

          LB3D-Wxpp32sp3
          Channel reader thread: channel

          "Channel reader thread: channel" Id=10 Group=main RUNNABLE (in native)
          at java.net.SocketInputStream.socketRead0(Native Method)
          at java.net.SocketInputStream.read(Unknown Source)
          at java.net.SocketInputStream.read(Unknown Source)
          at java.io.BufferedInputStream.fill(Unknown Source)
          at java.io.BufferedInputStream.read(Unknown Source)

          • locked java.io.BufferedInputStream@b9293d
            at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
            at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
            at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
            at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
            at java.io.ObjectInputStream.readObject0(Unknown Source)
            at java.io.ObjectInputStream.readObject(Unknown Source)
            at hudson.remoting.Command.readFrom(Command.java:92)
            at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

          main

          "main" Id=1 Group=main WAITING on hudson.remoting.Engine@1059a9a
          at java.lang.Object.wait(Native Method)

          • waiting on hudson.remoting.Engine@1059a9a
            at java.lang.Thread.join(Unknown Source)
            at java.lang.Thread.join(Unknown Source)
            at hudson.remoting.jnlp.Main.main(Main.java:137)
            at hudson.remoting.jnlp.Main._main(Main.java:130)
            at hudson.remoting.Launcher.run(Launcher.java:223)
            at hudson.remoting.Launcher.main(Launcher.java:192)

          Ping thread for channel hudson.remoting.Channel@41f871:channel

          "Ping thread for channel hudson.remoting.Channel@41f871:channel" Id=12 Group=main TIMED_WAITING
          at java.lang.Thread.sleep(Native Method)
          at hudson.remoting.PingThread.run(PingThread.java:86)

          pool-1-thread-8

          "pool-1-thread-8" Id=42 Group=main RUNNABLE
          at sun.management.ThreadImpl.dumpThreads0(Native Method)
          at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
          at hudson.Functions.getThreadInfos(Functions.java:1131)
          at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96)
          at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
          at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:326)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at hudson.remoting.Engine$1$1.run(Engine.java:63)
          at java.lang.Thread.run(Unknown Source)

          Number of locked synchronizers = 1

          • java.util.concurrent.ThreadPoolExecutor$Worker@5322ec

          Thread-1

          "Thread-1" Id=9 Group=main WAITING on hudson.remoting.Channel@41f871
          at java.lang.Object.wait(Native Method)

          • waiting on hudson.remoting.Channel@41f871
            at java.lang.Object.wait(Object.java:503)
            at hudson.remoting.Channel.join(Channel.java:842)
            at hudson.remoting.Engine.run(Engine.java:271)

          Attach Listener

          "Attach Listener" Id=4 Group=system RUNNABLE

          Finalizer

          "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@429b8e
          at java.lang.Object.wait(Native Method)

          • waiting on java.lang.ref.ReferenceQueue$Lock@429b8e
            at java.lang.ref.ReferenceQueue.remove(Unknown Source)
            at java.lang.ref.ReferenceQueue.remove(Unknown Source)
            at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

          Reference Handler

          "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@f2c136
          at java.lang.Object.wait(Native Method)

          • waiting on java.lang.ref.Reference$Lock@f2c136
            at java.lang.Object.wait(Object.java:503)
            at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)

          Slave-TestProcessTest
          Channel reader thread: channel

          "Channel reader thread: channel" Id=10 Group=main RUNNABLE (in native)
          at java.net.SocketInputStream.socketRead0(Native Method)
          at java.net.SocketInputStream.read(Unknown Source)
          at java.net.SocketInputStream.read(Unknown Source)
          at java.io.BufferedInputStream.fill(Unknown Source)
          at java.io.BufferedInputStream.read(Unknown Source)

          • locked java.io.BufferedInputStream@18e02f3
            at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
            at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
            at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
            at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
            at java.io.ObjectInputStream.readObject0(Unknown Source)
            at java.io.ObjectInputStream.readObject(Unknown Source)
            at hudson.remoting.Command.readFrom(Command.java:92)
            at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

          main

          "main" Id=1 Group=main WAITING on hudson.remoting.Engine@13ac592
          at java.lang.Object.wait(Native Method)

          • waiting on hudson.remoting.Engine@13ac592
            at java.lang.Thread.join(Unknown Source)
            at java.lang.Thread.join(Unknown Source)
            at hudson.remoting.jnlp.Main.main(Main.java:137)
            at hudson.remoting.jnlp.Main._main(Main.java:130)
            at hudson.remoting.Launcher.run(Launcher.java:223)
            at hudson.remoting.Launcher.main(Launcher.java:192)

          Ping thread for channel hudson.remoting.Channel@1a32902:channel

          "Ping thread for channel hudson.remoting.Channel@1a32902:channel" Id=12 Group=main TIMED_WAITING
          at java.lang.Thread.sleep(Native Method)
          at hudson.remoting.PingThread.run(PingThread.java:86)

          pool-1-svnkit-thread-1

          "pool-1-svnkit-thread-1" Id=60 Group=main WAITING on java.util.concurrent.SynchronousQueue$TransferStack@a0f65b
          at sun.misc.Unsafe.park(Native Method)

          • waiting on java.util.concurrent.SynchronousQueue$TransferStack@a0f65b
            at java.util.concurrent.locks.LockSupport.park(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
            at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
            at java.util.concurrent.SynchronousQueue.take(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

          pool-1-thread-79

          "pool-1-thread-79" Id=135 Group=main RUNNABLE
          at sun.management.ThreadImpl.dumpThreads0(Native Method)
          at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
          at hudson.Functions.getThreadInfos(Functions.java:1131)
          at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96)
          at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
          at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:326)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at hudson.remoting.Engine$1$1.run(Engine.java:63)
          at java.lang.Thread.run(Unknown Source)

          Number of locked synchronizers = 1

          • java.util.concurrent.ThreadPoolExecutor$Worker@1232e8d

          Thread-1

          "Thread-1" Id=9 Group=main WAITING on hudson.remoting.Channel@1a32902
          at java.lang.Object.wait(Native Method)

          • waiting on hudson.remoting.Channel@1a32902
            at java.lang.Object.wait(Object.java:503)
            at hudson.remoting.Channel.join(Channel.java:842)
            at hudson.remoting.Engine.run(Engine.java:271)

          Attach Listener

          "Attach Listener" Id=4 Group=system RUNNABLE

          Finalizer

          "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@63bafc
          at java.lang.Object.wait(Native Method)

          • waiting on java.lang.ref.ReferenceQueue$Lock@63bafc
            at java.lang.ref.ReferenceQueue.remove(Unknown Source)
            at java.lang.ref.ReferenceQueue.remove(Unknown Source)
            at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

          Reference Handler

          "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@1893737
          at java.lang.Object.wait(Native Method)

          • waiting on java.lang.ref.Reference$Lock@1893737
            at java.lang.Object.wait(Object.java:503)
            at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)

          Pavel Kudrys added a comment - It happened again today Any idea? Anyone please... This is what I see in jenkins-slave.err.log: Oct 31, 2013 7:41:16 AM hudson.remoting.jnlp.Main$CuiListener <init> INFO: Jenkins agent is running in headless mode. Oct 31, 2013 7:41:16 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among http://tcskjira:8095/ Oct 31, 2013 7:41:17 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to tcskjira:1695 Oct 31, 2013 7:41:17 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Oct 31, 2013 7:41:17 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected This is what I see in jenkins-slave.wrapper.log: 2013-10-31 07:41:09 - Starting C:\Program Files\Java\jre7\bin\java.exe -Xrs -jar "C:\Jenkins\slave.jar" -jnlpUrl http://tcskjira:8095/computer/LB3D-Wxpp64sp2/slave-agent.jnlp -secret 1ba0fb4096bc76c5ef1332c6368cd286a76074134a7091cd7603e93ec938b060 2013-10-31 07:41:09 - Started 1576 This is what I see in Jenkins Log: Oct 31, 2013 7:40:34 AM INFO hudson.slaves.RetentionStrategy$Demand check Launching computer LB3D-Wxpp64sp2 as it has been in demand for 39 sec Oct 31, 2013 7:40:34 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog [LB3D-Wxpp64sp2] Starting Virtual Machine... Oct 31, 2013 7:40:35 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog [LB3D-Wxpp64sp2] Powering on VM Oct 31, 2013 7:40:51 AM INFO hudson.model.AsyncPeriodicWork$1 run Started PeriodicBackup Oct 31, 2013 7:40:51 AM INFO hudson.model.AsyncPeriodicWork$1 run Finished PeriodicBackup. 0 ms Oct 31, 2013 7:41:16 AM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run Accepted connection #40 from /172.28.63.95:1034 Oct 31, 2013 7:41:23 AM WARNING hudson.node_monitors.ResponseTimeMonitor$1 monitor Making CAAV5R19x64 offline because it’s not responding Oct 31, 2013 7:41:23 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog [CAAV5R19x64] Already disconnecting on a separate thread Oct 31, 2013 7:41:51 AM INFO hudson.model.AsyncPeriodicWork$1 run Started PeriodicBackup Oct 31, 2013 7:41:51 AM INFO hudson.model.AsyncPeriodicWork$1 run Finished PeriodicBackup. 0 ms Oct 31, 2013 7:41:52 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor Failed to monitor LB3D-Wxpp64sp2 for Free Swap Space java.util.concurrent.TimeoutException at hudson.remoting.Request$1.get(Request.java:275) at hudson.remoting.Request$1.get(Request.java:210) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282) Slave Log says this: JNLP agent connected from /172.28.63.95 <=== [JENKINS REMOTING CAPACITY] ===>Slave.jar version: 2.32 This is a Windows slave Slave successfully connected and online followed by spinning wheel... Slave Thread Dump: Channel reader thread: channel "Channel reader thread: channel" Id=11 Group=main RUNNABLE (in native) at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) locked java.io.BufferedInputStream@1c2ba649 at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77) at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) main "main" Id=1 Group=main WAITING on hudson.remoting.Engine@7a0136b0 at java.lang.Object.wait(Native Method) waiting on hudson.remoting.Engine@7a0136b0 at java.lang.Thread.join(Unknown Source) at java.lang.Thread.join(Unknown Source) at hudson.remoting.jnlp.Main.main(Main.java:137) at hudson.remoting.jnlp.Main._main(Main.java:130) at hudson.remoting.Launcher.run(Launcher.java:223) at hudson.remoting.Launcher.main(Launcher.java:192) Ping thread for channel hudson.remoting.Channel@28509953:channel "Ping thread for channel hudson.remoting.Channel@28509953:channel" Id=13 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) pool-1-thread-17 "pool-1-thread-17" Id=29 Group=main RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(Unknown Source) at hudson.Functions.getThreadInfos(Functions.java:1131) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@493df066 pool-1-thread-2 "pool-1-thread-2" Id=14 Group=main RUNNABLE at com.sun.jna.Native.initIDs(Native Method) at com.sun.jna.Native.<clinit>(Native.java:122) at org.tmatesoft.svn.core.internal.util.jna.JNALibraryLoader.<clinit>(JNALibraryLoader.java:50) at org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:93) at org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196) at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1869) at org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57) at hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:975) at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:952) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:885) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:872) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2420) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@5133066c pool-1-thread-5 "pool-1-thread-5" Id=17 Group=main RUNNABLE at com.sun.jna.Structure.<clinit>(Structure.java:134) at org.jvnet.hudson.Windows.monitor(Windows.java:40) at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:110) at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:100) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@786dee4c Thread-1 "Thread-1" Id=10 Group=main WAITING on hudson.remoting.Channel@28509953 at java.lang.Object.wait(Native Method) waiting on hudson.remoting.Channel@28509953 at java.lang.Object.wait(Object.java:503) at hudson.remoting.Channel.join(Channel.java:842) at hudson.remoting.Engine.run(Engine.java:271) Attach Listener "Attach Listener" Id=4 Group=system RUNNABLE Finalizer "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@75b21111 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.ReferenceQueue$Lock@75b21111 at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) Reference Handler "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@7d7eaf74 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.Reference$Lock@7d7eaf74 at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) System Thread Dump: Thread Dump AWT-Windows "AWT-Windows" Id=614 Group=main RUNNABLE (in native) at sun.awt.windows.WToolkit.eventLoop(Native Method) at sun.awt.windows.WToolkit.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Channel reader thread: LB3D-Wxpp32sp3 "Channel reader thread: LB3D-Wxpp32sp3" Id=12151 Group=main RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) locked java.io.BufferedInputStream@7a7ec776 at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77) at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) Channel reader thread: LB3D-Wxpp64sp2 "Channel reader thread: LB3D-Wxpp64sp2" Id=12178 Group=main RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) locked java.io.BufferedInputStream@764058b0 at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77) at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) Channel reader thread: Slave-TestProcessBuild "Channel reader thread: Slave-TestProcessBuild" Id=6006 Group=main RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) locked java.io.BufferedInputStream@3563374a at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77) at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) Channel reader thread: Slave-TestProcessTest "Channel reader thread: Slave-TestProcessTest" Id=6121 Group=main RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) locked java.io.BufferedInputStream@374b5f62 at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77) at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) DestroyJavaVM "DestroyJavaVM" Id=19 Group=main RUNNABLE Executor #0 for LB3D-Wxpp32sp3 : executing ipprotection_32_64 #171 "Executor #0 for LB3D-Wxpp32sp3 : executing ipprotection_32_64 #171" Id=10955 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.matrix.DefaultMatrixExecutionStrategyImpl.waitForCompletion(DefaultMatrixExecutionStrategyImpl.java:290) at hudson.matrix.DefaultMatrixExecutionStrategyImpl.run(DefaultMatrixExecutionStrategyImpl.java:141) at hudson.matrix.MatrixBuild$MatrixBuildExecution.doRun(MatrixBuild.java:355) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:562) at hudson.model.Run.execute(Run.java:1665) at hudson.matrix.MatrixBuild.run(MatrixBuild.java:304) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:230) Executor #0 for LB3D-Wxpp64sp2 : executing ipprotection_32_64 » LB3D-Wxpp64sp2 #171 / waiting for hudson.remoting.Channel@59f73f53:LB3D-Wxpp64sp2 "Executor #0 for LB3D-Wxpp64sp2 : executing ipprotection_32_64 » LB3D-Wxpp64sp2 #171 / waiting for hudson.remoting.Channel@59f73f53:LB3D-Wxpp64sp2" Id=11106 Group=main TIMED_WAITING on hudson.remoting.UserRequest@3690a4a8 at java.lang.Object.wait(Native Method) waiting on hudson.remoting.UserRequest@3690a4a8 at hudson.remoting.Request.call(Request.java:146) at hudson.remoting.Channel.call(Channel.java:714) at hudson.FilePath.act(FilePath.java:901) at hudson.FilePath.act(FilePath.java:885) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:848) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:786) at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:117) at hudson.model.AbstractProject.checkout(AbstractProject.java:1411) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:652) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:557) at hudson.model.Run.execute(Run.java:1665) at hudson.matrix.MatrixRun.run(MatrixRun.java:146) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:230) Handling GET /threadDump : RequestHandlerThread 595 "Handling GET /threadDump : RequestHandlerThread 595 " Id=12350 Group=main TIMED_WAITING on java.util.concurrent.FutureTask$Sync@4deadcb6 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.FutureTask$Sync@4deadcb6 at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source) at java.util.concurrent.FutureTask.get(Unknown Source) at hudson.remoting.LocalChannel$2.get(LocalChannel.java:81) at jenkins.model.Jenkins.getAllThreadDumps(Jenkins.java:2919) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258) at org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104) at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83) at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57) at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51) at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80) at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74) at org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61) at org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsIterator(ExpressionSupport.java:94) at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:89) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53) at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:727) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:841) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:630) at org.kohsuke.stapler.Stapler.service(Stapler.java:224) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:208) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:181) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:64) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@620c96ba HashSessionScavenger-0 "HashSessionScavenger-0" Id=11 Group=main TIMED_WAITING on java.util.TaskQueue@4a7cb042 at java.lang.Object.wait(Native Method) waiting on java.util.TaskQueue@4a7cb042 at java.util.TimerThread.mainLoop(Unknown Source) at java.util.TimerThread.run(Unknown Source) javamelody "javamelody" Id=27 Group=main TIMED_WAITING on java.util.TaskQueue@1b9a5f49 at java.lang.Object.wait(Native Method) waiting on java.util.TaskQueue@1b9a5f49 at java.util.TimerThread.mainLoop(Unknown Source) at java.util.TimerThread.run(Unknown Source) Jenkins cron thread "Jenkins cron thread" Id=14 Group=main TIMED_WAITING on java.util.TaskQueue@258e37c9 at java.lang.Object.wait(Native Method) waiting on java.util.TaskQueue@258e37c9 at java.util.TimerThread.mainLoop(Unknown Source) at java.util.TimerThread.run(Unknown Source) Jenkins UDP 33848 monitoring thread "Jenkins UDP 33848 monitoring thread" Id=31 Group=main RUNNABLE (in native) at java.net.TwoStacksPlainDatagramSocketImpl.receive0(Native Method) locked java.net.TwoStacksPlainDatagramSocketImpl@1f036f93 at java.net.TwoStacksPlainDatagramSocketImpl.receive(Unknown Source) locked java.net.TwoStacksPlainDatagramSocketImpl@1f036f93 at java.net.DatagramSocket.receive(Unknown Source) locked java.net.DatagramPacket@7279ad5b locked java.net.MulticastSocket@12646c14 at hudson.UDPBroadcastThread.run(UDPBroadcastThread.java:82) Jenkins-Remoting-Thread-1218 "Jenkins-Remoting-Thread-1218" Id=12433 Group=main RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(Unknown Source) at hudson.Functions.getThreadInfos(Functions.java:1131) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92) at hudson.remoting.LocalChannel$1.call(LocalChannel.java:52) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@1904332a Jenkins-Remoting-Thread-1219 "Jenkins-Remoting-Thread-1219" Id=12434 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@2baacf98 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@2baacf98 at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Jenkins-Remoting-Thread-1220 "Jenkins-Remoting-Thread-1220" Id=12435 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@2baacf98 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@2baacf98 at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) JmDNS(tcskjira.local.).State.Timer "JmDNS(tcskjira.local.).State.Timer" Id=48 Group=main TIMED_WAITING on java.util.TaskQueue@587c9b22 at java.lang.Object.wait(Native Method) waiting on java.util.TaskQueue@587c9b22 at java.util.TimerThread.mainLoop(Unknown Source) at java.util.TimerThread.run(Unknown Source) JmDNS(tcskjira.local.).Timer "JmDNS(tcskjira.local.).Timer" Id=47 Group=main TIMED_WAITING on java.util.TaskQueue@5cad0290 at java.lang.Object.wait(Native Method) waiting on java.util.TaskQueue@5cad0290 at java.util.TimerThread.mainLoop(Unknown Source) at java.util.TimerThread.run(Unknown Source) jnr.ffi.util.ref.internal.Finalizer "jnr.ffi.util.ref.internal.Finalizer" Id=5472 Group=main WAITING on java.lang.ref.ReferenceQueue$Lock@4c2f6a8 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.ReferenceQueue$Lock@4c2f6a8 at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at jnr.ffi.util.ref.internal.Finalizer.run(Finalizer.java:128) LauncherControlThread [ControlPort=-1] "LauncherControlThread [ControlPort=-1] " Id=17 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at winstone.Launcher.run(Launcher.java:245) at java.lang.Thread.run(Unknown Source) NioSocketAcceptor-1 "NioSocketAcceptor-1" Id=29 Group=main RUNNABLE (in native) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source) at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source) at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source) locked sun.nio.ch.Util$2@5feec26b locked java.util.Collections$UnmodifiableSet@29e1ac62 locked sun.nio.ch.WindowsSelectorImpl@4defca63 at sun.nio.ch.SelectorImpl.select(Unknown Source) at sun.nio.ch.SelectorImpl.select(Unknown Source) at org.apache.mina.transport.socket.nio.NioSocketAcceptor.select(NioSocketAcceptor.java:238) at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:432) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@29090dda Ping thread for channel hudson.remoting.Channel@1675652c:Slave-TestProcessTest "Ping thread for channel hudson.remoting.Channel@1675652c:Slave-TestProcessTest" Id=6122 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) Ping thread for channel hudson.remoting.Channel@50923d0b:Slave-TestProcessBuild "Ping thread for channel hudson.remoting.Channel@50923d0b:Slave-TestProcessBuild" Id=6007 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) Ping thread for channel hudson.remoting.Channel@59f73f53:LB3D-Wxpp64sp2 "Ping thread for channel hudson.remoting.Channel@59f73f53:LB3D-Wxpp64sp2" Id=12179 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) Ping thread for channel hudson.remoting.Channel@5ade8f7a:LB3D-Wxpp32sp3 "Ping thread for channel hudson.remoting.Channel@5ade8f7a:LB3D-Wxpp32sp3" Id=12152 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) pool-1-svnkit-thread-1 "pool-1-svnkit-thread-1" Id=411 Group=main WAITING on java.util.concurrent.SynchronousQueue$TransferStack@1bcf9209 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@1bcf9209 at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) pool-1-svnkit-thread-2 "pool-1-svnkit-thread-2" Id=2322 Group=main WAITING on java.util.concurrent.SynchronousQueue$TransferStack@1bcf9209 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@1bcf9209 at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) pool-4-thread-1272 "pool-4-thread-1272" Id=12398 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9 at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) pool-4-thread-1273 "pool-4-thread-1273" Id=12399 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9 at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) pool-4-thread-1274 "pool-4-thread-1274" Id=12417 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9 at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) pool-4-thread-1275 "pool-4-thread-1275" Id=12418 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9 at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) pool-9-thread-1 "pool-9-thread-1" Id=44 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2ba02595 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2ba02595 at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) RequestHandlerThread 1 Selector0 "RequestHandlerThread 1 Selector0" Id=15 Group=main RUNNABLE (in native) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source) at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source) at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source) locked sun.nio.ch.Util$2@2fea3452 locked java.util.Collections$UnmodifiableSet@e2f4d5c locked sun.nio.ch.WindowsSelectorImpl@7ed452e4 at sun.nio.ch.SelectorImpl.select(Unknown Source) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@4da2d209 RequestHandlerThread 2 Acceptor0 SelectChannelConnector@0.0.0.0:8095 "RequestHandlerThread 2 Acceptor0 SelectChannelConnector@0.0.0.0:8095" Id=16 Group=main RUNNABLE (in native) at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source) locked java.lang.Object@33e5bfc7 at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@5db00667 RequestHandlerThread 603 "RequestHandlerThread 603 " Id=12388 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) RequestHandlerThread 605 "RequestHandlerThread 605 " Id=12391 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) RequestHandlerThread 607 "RequestHandlerThread 607 " Id=12427 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) RequestHandlerThread 608 "RequestHandlerThread 608 " Id=12431 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) SocketListener(tcskjira.local.) "SocketListener(tcskjira.local.)" Id=46 Group=main RUNNABLE (in native) at java.net.TwoStacksPlainDatagramSocketImpl.receive0(Native Method) locked java.net.TwoStacksPlainDatagramSocketImpl@62836b78 at java.net.TwoStacksPlainDatagramSocketImpl.receive(Unknown Source) locked java.net.TwoStacksPlainDatagramSocketImpl@62836b78 at java.net.DatagramSocket.receive(Unknown Source) locked java.net.DatagramPacket@6756aa7b locked java.net.MulticastSocket@6ad2a63d at javax.jmdns.impl.SocketListener.run(SocketListener.java:41) TCP slave agent listener port=0 "TCP slave agent listener port=0" Id=30 Group=main RUNNABLE (in native) at java.net.TwoStacksPlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(Unknown Source) at java.net.PlainSocketImpl.accept(Unknown Source) locked java.net.SocksSocketImpl@7ef39e7a at java.net.ServerSocket.implAccept(Unknown Source) at java.net.ServerSocket.accept(Unknown Source) at hudson.TcpSlaveAgentListener.run(TcpSlaveAgentListener.java:93) ViewJob reload thread "ViewJob reload thread" Id=28 Group=main TIMED_WAITING on java.util.LinkedHashSet@39d0518e at java.lang.Object.wait(Native Method) waiting on java.util.LinkedHashSet@39d0518e at hudson.model.ViewJob$ReloadThread.getNext(ViewJob.java:176) at hudson.model.ViewJob$ReloadThread.run(ViewJob.java:193) Waiting to acquire c:\Jenkins\workspace\ipprotection_32_64 : SCM polling for hudson.matrix.MatrixProject@3b277576 [ipprotection_32_64] "Waiting to acquire c:\Jenkins\workspace\ipprotection_32_64 : SCM polling for hudson.matrix.MatrixProject@3b277576 [ipprotection_32_64] " Id=12065 Group=main WAITING on hudson.slaves.WorkspaceList@195952ce at java.lang.Object.wait(Native Method) waiting on hudson.slaves.WorkspaceList@195952ce at java.lang.Object.wait(Object.java:503) at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:232) at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:211) at hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1580) at hudson.model.AbstractProject._poll(AbstractProject.java:1558) at hudson.model.AbstractProject.poll(AbstractProject.java:1490) at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:439) at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:468) at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@3435c217 Attach Listener "Attach Listener" Id=5 Group=system RUNNABLE Finalizer "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@4017e342 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.ReferenceQueue$Lock@4017e342 at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) Java2D Disposer "Java2D Disposer" Id=12 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@1bf08062 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.ReferenceQueue$Lock@1bf08062 at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at sun.java2d.Disposer.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Reference Handler "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@604922ed at java.lang.Object.wait(Native Method) waiting on java.lang.ref.Reference$Lock@604922ed at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) Signal Dispatcher "Signal Dispatcher" Id=4 Group=system RUNNABLE CAAV5R19x64 N/A offline LB3D-Wxpp64sp2 Channel reader thread: channel "Channel reader thread: channel" Id=11 Group=main RUNNABLE (in native) at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) locked java.io.BufferedInputStream@1c2ba649 at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77) at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) main "main" Id=1 Group=main WAITING on hudson.remoting.Engine@7a0136b0 at java.lang.Object.wait(Native Method) waiting on hudson.remoting.Engine@7a0136b0 at java.lang.Thread.join(Unknown Source) at java.lang.Thread.join(Unknown Source) at hudson.remoting.jnlp.Main.main(Main.java:137) at hudson.remoting.jnlp.Main._main(Main.java:130) at hudson.remoting.Launcher.run(Launcher.java:223) at hudson.remoting.Launcher.main(Launcher.java:192) Ping thread for channel hudson.remoting.Channel@28509953:channel "Ping thread for channel hudson.remoting.Channel@28509953:channel" Id=13 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) pool-1-thread-18 "pool-1-thread-18" Id=30 Group=main RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(Unknown Source) at hudson.Functions.getThreadInfos(Functions.java:1131) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@59d9c141 pool-1-thread-2 "pool-1-thread-2" Id=14 Group=main RUNNABLE at com.sun.jna.Native.initIDs(Native Method) at com.sun.jna.Native.<clinit>(Native.java:122) at org.tmatesoft.svn.core.internal.util.jna.JNALibraryLoader.<clinit>(JNALibraryLoader.java:50) at org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:93) at org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196) at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1869) at org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57) at hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:975) at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:952) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:885) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:872) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2420) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@5133066c pool-1-thread-5 "pool-1-thread-5" Id=17 Group=main RUNNABLE at com.sun.jna.Structure.<clinit>(Structure.java:134) at org.jvnet.hudson.Windows.monitor(Windows.java:40) at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:110) at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:100) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@786dee4c Thread-1 "Thread-1" Id=10 Group=main WAITING on hudson.remoting.Channel@28509953 at java.lang.Object.wait(Native Method) waiting on hudson.remoting.Channel@28509953 at java.lang.Object.wait(Object.java:503) at hudson.remoting.Channel.join(Channel.java:842) at hudson.remoting.Engine.run(Engine.java:271) Attach Listener "Attach Listener" Id=4 Group=system RUNNABLE Finalizer "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@75b21111 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.ReferenceQueue$Lock@75b21111 at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) Reference Handler "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@7d7eaf74 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.Reference$Lock@7d7eaf74 at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) Slave-TestProcessBuild Channel reader thread: channel "Channel reader thread: channel" Id=11 Group=main RUNNABLE (in native) at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) locked java.io.BufferedInputStream@74784829 at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77) at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) main "main" Id=1 Group=main WAITING on hudson.remoting.Engine@7f25ab41 at java.lang.Object.wait(Native Method) waiting on hudson.remoting.Engine@7f25ab41 at java.lang.Thread.join(Unknown Source) at java.lang.Thread.join(Unknown Source) at hudson.remoting.jnlp.Main.main(Main.java:137) at hudson.remoting.jnlp.Main._main(Main.java:130) at hudson.remoting.Launcher.run(Launcher.java:223) at hudson.remoting.Launcher.main(Launcher.java:192) Ping thread for channel hudson.remoting.Channel@1c06d198:channel "Ping thread for channel hudson.remoting.Channel@1c06d198:channel" Id=13 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) pool-1-thread-802 "pool-1-thread-802" Id=916 Group=main RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(Unknown Source) at hudson.Functions.getThreadInfos(Functions.java:1131) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@13c5810e Thread-1 "Thread-1" Id=10 Group=main WAITING on hudson.remoting.Channel@1c06d198 at java.lang.Object.wait(Native Method) waiting on hudson.remoting.Channel@1c06d198 at java.lang.Object.wait(Object.java:503) at hudson.remoting.Channel.join(Channel.java:842) at hudson.remoting.Engine.run(Engine.java:271) Attach Listener "Attach Listener" Id=4 Group=system RUNNABLE Finalizer "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@4c9b77c at java.lang.Object.wait(Native Method) waiting on java.lang.ref.ReferenceQueue$Lock@4c9b77c at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) Reference Handler "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@572d3058 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.Reference$Lock@572d3058 at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) LB3D-Wxpp32sp3 Channel reader thread: channel "Channel reader thread: channel" Id=10 Group=main RUNNABLE (in native) at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) locked java.io.BufferedInputStream@b9293d at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77) at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) main "main" Id=1 Group=main WAITING on hudson.remoting.Engine@1059a9a at java.lang.Object.wait(Native Method) waiting on hudson.remoting.Engine@1059a9a at java.lang.Thread.join(Unknown Source) at java.lang.Thread.join(Unknown Source) at hudson.remoting.jnlp.Main.main(Main.java:137) at hudson.remoting.jnlp.Main._main(Main.java:130) at hudson.remoting.Launcher.run(Launcher.java:223) at hudson.remoting.Launcher.main(Launcher.java:192) Ping thread for channel hudson.remoting.Channel@41f871:channel "Ping thread for channel hudson.remoting.Channel@41f871:channel" Id=12 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) pool-1-thread-8 "pool-1-thread-8" Id=42 Group=main RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(Unknown Source) at hudson.Functions.getThreadInfos(Functions.java:1131) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@5322ec Thread-1 "Thread-1" Id=9 Group=main WAITING on hudson.remoting.Channel@41f871 at java.lang.Object.wait(Native Method) waiting on hudson.remoting.Channel@41f871 at java.lang.Object.wait(Object.java:503) at hudson.remoting.Channel.join(Channel.java:842) at hudson.remoting.Engine.run(Engine.java:271) Attach Listener "Attach Listener" Id=4 Group=system RUNNABLE Finalizer "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@429b8e at java.lang.Object.wait(Native Method) waiting on java.lang.ref.ReferenceQueue$Lock@429b8e at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) Reference Handler "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@f2c136 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.Reference$Lock@f2c136 at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) Slave-TestProcessTest Channel reader thread: channel "Channel reader thread: channel" Id=10 Group=main RUNNABLE (in native) at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) locked java.io.BufferedInputStream@18e02f3 at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77) at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) main "main" Id=1 Group=main WAITING on hudson.remoting.Engine@13ac592 at java.lang.Object.wait(Native Method) waiting on hudson.remoting.Engine@13ac592 at java.lang.Thread.join(Unknown Source) at java.lang.Thread.join(Unknown Source) at hudson.remoting.jnlp.Main.main(Main.java:137) at hudson.remoting.jnlp.Main._main(Main.java:130) at hudson.remoting.Launcher.run(Launcher.java:223) at hudson.remoting.Launcher.main(Launcher.java:192) Ping thread for channel hudson.remoting.Channel@1a32902:channel "Ping thread for channel hudson.remoting.Channel@1a32902:channel" Id=12 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) pool-1-svnkit-thread-1 "pool-1-svnkit-thread-1" Id=60 Group=main WAITING on java.util.concurrent.SynchronousQueue$TransferStack@a0f65b at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@a0f65b at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) pool-1-thread-79 "pool-1-thread-79" Id=135 Group=main RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(Unknown Source) at hudson.Functions.getThreadInfos(Functions.java:1131) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@1232e8d Thread-1 "Thread-1" Id=9 Group=main WAITING on hudson.remoting.Channel@1a32902 at java.lang.Object.wait(Native Method) waiting on hudson.remoting.Channel@1a32902 at java.lang.Object.wait(Object.java:503) at hudson.remoting.Channel.join(Channel.java:842) at hudson.remoting.Engine.run(Engine.java:271) Attach Listener "Attach Listener" Id=4 Group=system RUNNABLE Finalizer "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@63bafc at java.lang.Object.wait(Native Method) waiting on java.lang.ref.ReferenceQueue$Lock@63bafc at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) Reference Handler "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@1893737 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.Reference$Lock@1893737 at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)

          Valentin Marin added a comment - - edited

          I've also ran into a similar issue in a sense that it also involved stuck builds and "Failed to monitor <slave-name> for Free Swap Space" exception.

          However, the exception details are a bit different:

          WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor
          Failed to monitor <slave-name> for Free Swap Space
          java.util.concurrent.ExecutionException: java.lang.UnsatisfiedLinkError: com.sun.jna.Native.malloc(J)J
          at hudson.remoting.Channel$3.adapt(Channel.java:747)
          at hudson.remoting.Channel$3.adapt(Channel.java:742)
          at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282)
          Caused by: java.lang.UnsatisfiedLinkError: com.sun.jna.Native.malloc(J)J
          at com.sun.jna.Native.malloc(Native Method)
          at com.sun.jna.Memory.malloc(Memory.java:690)
          at com.sun.jna.Memory.(Memory.java:79)
          at com.sun.jna.Structure$AutoAllocated.(Structure.java:1366)
          at com.sun.jna.Structure.autoAllocate(Structure.java:231)
          at com.sun.jna.Structure.allocateMemory(Structure.java:302)
          at com.sun.jna.Structure.(Structure.java:179)
          at com.sun.jna.Structure.(Structure.java:169)
          at com.sun.jna.Structure.(Structure.java:165)
          at com.sun.jna.Structure.(Structure.java:156)
          at org.jvnet.hudson.Windows$MEMORYSTATUSEX.(Windows.java:55)
          at org.jvnet.hudson.Windows.monitor(Windows.java:40)
          at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:110)
          at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:100)
          at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:326)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at java.lang.Thread.run(Unknown Source)

          What I also noticed a couple of times is that java.exe process on master starts hobbing all of the available RAM plus it's thread count rises.

          I'm running Jenkins ver. 1.530 on a 2008 R2 server, nodes (all of them are windows servers 2003 x86) are managed via vSphere Cloud plug-in and are being controlled as a Windows service using administrator account.

          Valentin Marin added a comment - - edited I've also ran into a similar issue in a sense that it also involved stuck builds and "Failed to monitor <slave-name> for Free Swap Space" exception. However, the exception details are a bit different: WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor Failed to monitor <slave-name> for Free Swap Space java.util.concurrent.ExecutionException: java.lang.UnsatisfiedLinkError: com.sun.jna.Native.malloc(J)J at hudson.remoting.Channel$3.adapt(Channel.java:747) at hudson.remoting.Channel$3.adapt(Channel.java:742) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282) Caused by: java.lang.UnsatisfiedLinkError: com.sun.jna.Native.malloc(J)J at com.sun.jna.Native.malloc(Native Method) at com.sun.jna.Memory.malloc(Memory.java:690) at com.sun.jna.Memory.(Memory.java:79) at com.sun.jna.Structure$AutoAllocated.(Structure.java:1366) at com.sun.jna.Structure.autoAllocate(Structure.java:231) at com.sun.jna.Structure.allocateMemory(Structure.java:302) at com.sun.jna.Structure.(Structure.java:179) at com.sun.jna.Structure.(Structure.java:169) at com.sun.jna.Structure.(Structure.java:165) at com.sun.jna.Structure.(Structure.java:156) at org.jvnet.hudson.Windows$MEMORYSTATUSEX.(Windows.java:55) at org.jvnet.hudson.Windows.monitor(Windows.java:40) at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:110) at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:100) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) What I also noticed a couple of times is that java.exe process on master starts hobbing all of the available RAM plus it's thread count rises. I'm running Jenkins ver. 1.530 on a 2008 R2 server, nodes (all of them are windows servers 2003 x86) are managed via vSphere Cloud plug-in and are being controlled as a Windows service using administrator account.

          Jason Swager added a comment -

          I'm also encountering this problem. Same symptoms. Jenkins 1.538, Windows 2008 R2 server, vSphere Cloud plugin. All the slaves that have these symptoms are showing this only after a slave restart - if they are always on, they don't have this problem. So far, only seen on Windows slaves; they use the JNLP agent starting mechanism. Several versions of Windows; Windows 7, Windows 8, Windows XP.

          Jason Swager added a comment - I'm also encountering this problem. Same symptoms. Jenkins 1.538, Windows 2008 R2 server, vSphere Cloud plugin. All the slaves that have these symptoms are showing this only after a slave restart - if they are always on, they don't have this problem. So far, only seen on Windows slaves; they use the JNLP agent starting mechanism. Several versions of Windows; Windows 7, Windows 8, Windows XP.

          Pavel Kudrys added a comment -

          Hi guys,

          Good to hear I'm not alone in this trouble! Any idea who we should contact to look at this issue? I think my company would no mind to pay a "service fee" to get this issue fixed.

          Yesterday, I changed the way of starting the slave service. Instead of automatically started service, I'm now using a batch file executed during the windows start. Sadly, this did not help. Night build got stuck again with one slave and sure enough, the log contained several "Failed to monitor SlaveName for Free Swap Space" errors.

          Pavel Kudrys added a comment - Hi guys, Good to hear I'm not alone in this trouble! Any idea who we should contact to look at this issue? I think my company would no mind to pay a "service fee" to get this issue fixed. Yesterday, I changed the way of starting the slave service. Instead of automatically started service, I'm now using a batch file executed during the windows start. Sadly, this did not help. Night build got stuck again with one slave and sure enough, the log contained several "Failed to monitor SlaveName for Free Swap Space" errors.

          Jason Swager added a comment -

          Sadly, I have no idea where to start looking for a solution. Since this only happens on our large production system and not our smaller test system, I can't effectively try solutions. I was going to try a fresh Jenkins with just the vSphere plugin, then slowly add plugins to match production.

          Jason Swager added a comment - Sadly, I have no idea where to start looking for a solution. Since this only happens on our large production system and not our smaller test system, I can't effectively try solutions. I was going to try a fresh Jenkins with just the vSphere plugin, then slowly add plugins to match production.

          Guys, are you running Jenkins on built-in Winstone or Apache Tomcat?

          Valentin Marin added a comment - Guys, are you running Jenkins on built-in Winstone or Apache Tomcat?

          Pavel Kudrys added a comment -

          Hi Valentin,

          We are running it on Winstone.

          Pavel Kudrys added a comment - Hi Valentin, We are running it on Winstone.

          Jason Swager added a comment -

          Winstone as well

          Jason Swager added a comment - Winstone as well

          Valentin Marin added a comment - - edited

          Same here. I've got a hunch though, that the exception is somehow related to it, so I'm going to set up Tomcat and see if it makes any difference.

          Valentin Marin added a comment - - edited Same here. I've got a hunch though, that the exception is somehow related to it, so I'm going to set up Tomcat and see if it makes any difference.

          Valentin Marin added a comment - - edited

          So I've set Jenkins up to run on Tomcat 7 up and upgraded 'archive artifacts' plug-in to latest version for a good measure (since all of my builds are using that and it is known to be greedy with RAM) and what do you know, 3 days without stalled builds and counting. Will let you guys know if the issue indeed went away.

          Valentin Marin added a comment - - edited So I've set Jenkins up to run on Tomcat 7 up and upgraded 'archive artifacts' plug-in to latest version for a good measure (since all of my builds are using that and it is known to be greedy with RAM) and what do you know, 3 days without stalled builds and counting. Will let you guys know if the issue indeed went away.

          After running the setup for a while, I came to the conclusion that the issue is still there, although with reduced amount of fails.

          Valentin Marin added a comment - After running the setup for a while, I came to the conclusion that the issue is still there, although with reduced amount of fails.

          VladDRussian added a comment -

          Thought i'd throw in my errors. We are seeing HTTP stop responding and visualVM can't take a dump once it fails. Memory is fine, but threads just keep increasing, once they hit 550-600 active it just stops and says memory issue won't respond till the master is restarted.
          Jenkins 1.484, win2k3 (master) win2k8/win2k3 (slaves)

          Feb 25, 2014 9:21:06 AM hudson.model.DownloadService$Downloadable doPostBack
          INFO: Obtained the updated data file for hudson.tools.JDKInstaller
          Feb 25, 2014 9:21:06 AM hudson.model.DownloadService$Downloadable doPostBack
          INFO: Obtained the updated data file for hudson.tasks.Ant.AntInstaller
          Feb 25, 2014 9:21:06 AM org.kohsuke.stapler.compression.CompressionFilter reportException
          WARNING: Untrapped servlet exception
          java.io.IOException: Unable to delete E:\jenkins\updates\hudson.tasks.Maven.MavenInstaller
          at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:112)
          at hudson.util.TextFile.write(TextFile.java:81)
          at hudson.model.DownloadService$Downloadable.doPostBack(DownloadService.java:260)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
          at java.lang.reflect.Method.invoke(Unknown Source)
          at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:288)
          at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:151)
          at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:90)
          at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111)
          at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659)
          at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241)
          at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659)
          at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:384)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659)
          at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:203)
          at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:488)
          at org.kohsuke.stapler.Stapler.service(Stapler.java:162)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
          at winstone.ServletConfiguration.execute(ServletConfiguration.java:248)
          at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
          at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
          at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
          at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
          at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
          at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
          at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
          at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
          at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
          at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
          at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
          at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
          at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
          at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
          at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
          at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
          at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
          at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
          at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
          at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
          at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)
          at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)
          at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
          at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at java.lang.Thread.run(Unknown Source)

          FREE DISK SPACE MONITORING STILL IN PROGRESS, FAILED, THREADS LEFT, MAXING
          Feb 21, 2014 4:55:20 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
          INFO: Accepted connection #12 from /172.16.69.140:64901
          Feb 21, 2014 4:55:21 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
          INFO: Accepted connection #13 from /172.16.69.82:61763
          Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record <init>
          WARNING: Previous Free Swap Space monitoring activity still in progress. Interrupting
          Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record <init>
          WARNING: Previous Free Disk Space monitoring activity still in progress. Interrupting
          Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record <init>
          WARNING: Previous Free Temp Space monitoring activity still in progress. Interrupting
          Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run
          WARNING: Node monitoring FFX-MOM-BLD10 for Free Swap Space aborted.
          java.lang.InterruptedException
          at java.lang.Object.wait(Native Method)
          at hudson.remoting.Request.call(Request.java:146)
          at hudson.remoting.Channel.call(Channel.java:664)
          at hudson.node_monitors.SwapSpaceMonitor$1.monitor(SwapSpaceMonitor.java:83)
          at hudson.node_monitors.SwapSpaceMonitor$1.monitor(SwapSpaceMonitor.java:81)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219)
          Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run
          WARNING: Node monitoring FFX-MOM-BLD5 for Free Temp Space aborted.
          java.lang.InterruptedException
          at java.lang.Object.wait(Native Method)
          at hudson.remoting.Request.call(Request.java:146)
          at hudson.remoting.Channel.call(Channel.java:664)
          at hudson.FilePath.act(FilePath.java:840)
          at hudson.FilePath.act(FilePath.java:824)
          at hudson.node_monitors.TemporarySpaceMonitor$1.getFreeSpace(TemporarySpaceMonitor.java:73)
          at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:169)
          at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:49)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219)
          Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run
          WARNING: Node monitoring FFX-MOM-BLD5 for Free Disk Space aborted.
          java.lang.InterruptedException
          at java.lang.Object.wait(Native Method)
          at hudson.remoting.Request.call(Request.java:146)
          at hudson.remoting.Channel.call(Channel.java:664)
          at hudson.FilePath.act(FilePath.java:840)
          at hudson.FilePath.act(FilePath.java:824)
          at hudson.node_monitors.DiskSpaceMonitor$1.getFreeSpace(DiskSpaceMonitor.java:71)
          at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:169)
          at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:49)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219)

          BELOW LEAVING OPEN THREADS AND MAXING????
          Feb 20, 2014 6:32:49 AM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run
          WARNING: Failed to monitor FFX-MOM-BLD10 for Free Temp Space
          hudson.util.IOException2: remote file operation failed: E:\jenkins at hudson.remoting.Channel@18b622:FFX-MOM-BLD10
          at hudson.FilePath.act(FilePath.java:847)
          at hudson.FilePath.act(FilePath.java:824)
          at hudson.node_monitors.TemporarySpaceMonitor$1.getFreeSpace(TemporarySpaceMonitor.java:73)
          at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:169)
          at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:49)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219)
          Caused by: java.io.IOException: Remote call on FFX-MOM-BLD10 failed
          at hudson.remoting.Channel.call(Channel.java:673)
          at hudson.FilePath.act(FilePath.java:840)
          ... 5 more
          Caused by: java.lang.Error: Failed to deserialize the Callable object.
          at hudson.remoting.UserRequest.perform(UserRequest.java:104)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:326)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at hudson.remoting.Engine$1$1.run(Engine.java:60)
          at java.lang.Thread.run(Unknown Source)
          Caused by: java.lang.reflect.UndeclaredThrowableException
          at $Proxy6.fetch2(Unknown Source)
          at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:122)
          at java.lang.ClassLoader.loadClass(Unknown Source)
          at java.lang.ClassLoader.loadClass(Unknown Source)
          at java.lang.ClassLoader.loadClassInternal(Unknown Source)
          at java.lang.Class.getDeclaredMethods0(Native Method)
          at java.lang.Class.privateGetDeclaredMethods(Unknown Source)
          at java.lang.Class.getDeclaredMethod(Unknown Source)
          at java.io.ObjectStreamClass.getPrivateMethod(Unknown Source)
          at java.io.ObjectStreamClass.access$1700(Unknown Source)
          at java.io.ObjectStreamClass$2.run(Unknown Source)
          at java.security.AccessController.doPrivileged(Native Method)
          at java.io.ObjectStreamClass.<init>(Unknown Source)
          at java.io.ObjectStreamClass.lookup(Unknown Source)
          at java.io.ObjectStreamClass.initNonProxy(Unknown Source)
          at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)
          at java.io.ObjectInputStream.readClassDesc(Unknown Source)
          at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
          at java.io.ObjectInputStream.readObject0(Unknown Source)
          at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
          at java.io.ObjectInputStream.readSerialData(Unknown Source)
          at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
          at java.io.ObjectInputStream.readObject0(Unknown Source)
          at java.io.ObjectInputStream.readObject(Unknown Source)
          at hudson.remoting.UserRequest.deserialize(UserRequest.java:182)
          at hudson.remoting.UserRequest.perform(UserRequest.java:98)
          ... 9 more
          Caused by: java.lang.InterruptedException
          at java.lang.Object.wait(Native Method)
          at hudson.remoting.Request.call(Request.java:146)
          at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:160)
          ... 35 more
          Feb 20, 2014 6:32:49 AM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run
          WARNING: Failed to monitor FFX-MOM-BLD12 for Free Swap Space
          java.io.IOException: Remote call on FFX-MOM-BLD12 failed
          at hudson.remoting.Channel.call(Channel.java:673)
          at hudson.node_monitors.SwapSpaceMonitor$1.monitor(SwapSpaceMonitor.java:83)
          at hudson.node_monitors.SwapSpaceMonitor$1.monitor(SwapSpaceMonitor.java:81)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219)
          Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.jvnet.hudson.Windows$Kernel32
          at org.jvnet.hudson.Windows.monitor(Windows.java:41)
          at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:113)
          at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:99)
          at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:326)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          at hudson.remoting.Engine$1$1.run(Engine.java:60)
          at java.lang.Thread.run(Thread.java:619)
          Feb 20, 2014 7:32:49 AM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run
          WARNING: Failed to monitor FFX-MOM-BLD10 for Free Temp Space
          hudson.util.IOException2: remote file operation failed: E:\jenkins at hudson.remoting.Channel@18b622:FFX-MOM-BLD10
          at hudson.FilePath.act(FilePath.java:847)
          at hudson.FilePath.act(FilePath.java:824)
          at hudson.node_monitors.TemporarySpaceMonitor$1.getFreeSpace(TemporarySpaceMonitor.java:73)
          at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:169)
          at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:49)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219)
          Caused by: java.io.IOException: Remote call on FFX-MOM-BLD10 failed
          at hudson.remoting.Channel.call(Channel.java:673)
          at hudson.FilePath.act(FilePath.java:840)
          ... 5 more
          Caused by: java.lang.Error: Failed to deserialize the Callable object.
          at hudson.remoting.UserRequest.perform(UserRequest.java:104)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:326)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at hudson.remoting.Engine$1$1.run(Engine.java:60)
          at java.lang.Thread.run(Unknown Source)
          Caused by: java.lang.reflect.UndeclaredThrowableException
          at $Proxy6.fetch2(Unknown Source)
          at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:122)
          at java.lang.ClassLoader.loadClass(Unknown Source)
          at java.lang.ClassLoader.loadClass(Unknown Source)
          at java.lang.ClassLoader.loadClassInternal(Unknown Source)
          at java.lang.Class.getDeclaredMethods0(Native Method)
          at java.lang.Class.privateGetDeclaredMethods(Unknown Source)
          at java.lang.Class.getDeclaredMethod(Unknown Source)
          at java.io.ObjectStreamClass.getPrivateMethod(Unknown Source)
          at java.io.ObjectStreamClass.access$1700(Unknown Source)
          at java.io.ObjectStreamClass$2.run(Unknown Source)
          at java.security.AccessController.doPrivileged(Native Method)
          at java.io.ObjectStreamClass.<init>(Unknown Source)
          at java.io.ObjectStreamClass.lookup(Unknown Source)
          at java.io.ObjectStreamClass.initNonProxy(Unknown Source)
          at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)
          at java.io.ObjectInputStream.readClassDesc(Unknown Source)
          at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
          at java.io.ObjectInputStream.readObject0(Unknown Source)
          at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
          at java.io.ObjectInputStream.readSerialData(Unknown Source)
          at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
          at java.io.ObjectInputStream.readObject0(Unknown Source)
          at java.io.ObjectInputStream.readObject(Unknown Source)
          at hudson.remoting.UserRequest.deserialize(UserRequest.java:182)
          at hudson.remoting.UserRequest.perform(UserRequest.java:98)
          ... 9 more
          Caused by: java.lang.InterruptedException
          at java.lang.Object.wait(Native Method)
          at hudson.remoting.Request.call(Request.java:146)
          at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:160)
          ... 35 more

          Feb 20, 2014 7:50:42 AM winstone.Logger logInternal
          SEVERE: Error during HTTP listener init or shutdown
          java.lang.OutOfMemoryError: unable to create new native thread
          at java.lang.Thread.start0(Native Method)
          at java.lang.Thread.start(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
          at winstone.BoundedExecutorService.scheduleNext(BoundedExecutorService.java:74)
          at winstone.BoundedExecutorService.execute(BoundedExecutorService.java:64)
          at java.util.concurrent.AbstractExecutorService.submit(Unknown Source)
          at winstone.ObjectPool.handleRequest(ObjectPool.java:102)
          at winstone.HttpListener.run(HttpListener.java:144)
          at java.lang.Thread.run(Unknown Source)
          Feb 20, 2014 7:50:45 AM org.kohsuke.stapler.compression.CompressionFilter reportException
          WARNING: Untrapped servlet exception
          winstone.ClientSocketException: Failed to write to client
          at winstone.ClientOutputStream.write(ClientOutputStream.java:41)
          at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
          at java.io.BufferedOutputStream.flush(Unknown Source)
          at winstone.WinstoneOutputStream.commit(WinstoneOutputStream.java:167)
          at winstone.WinstoneOutputStream.commit(WinstoneOutputStream.java:123)
          at winstone.WinstoneOutputStream.write(WinstoneOutputStream.java:112)
          at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:421)
          at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:249)
          at org.kohsuke.stapler.Stapler.service(Stapler.java:152)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
          at winstone.ServletConfiguration.execute(ServletConfiguration.java:248)
          at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
          at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
          at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
          at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
          at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
          at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
          at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
          at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
          at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
          at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
          at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
          at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
          at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
          at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
          at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
          at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
          at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
          at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
          at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
          at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
          at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)
          at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)
          at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
          at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at java.lang.Thread.run(Unknown Source)
          Caused by: java.net.SocketException: Connection reset by peer: socket write error
          at java.net.SocketOutputStream.socketWrite0(Native Method)
          at java.net.SocketOutputStream.socketWrite(Unknown Source)
          at java.net.SocketOutputStream.write(Unknown Source)
          at winstone.ClientOutputStream.write(ClientOutputStream.java:39)
          ... 57 more

          REACHED LIMIT AND FAIL
          Feb 23, 2014 5:07:53 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started Workspace clean-up
          Feb 23, 2014 5:07:56 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished Workspace clean-up. 3156 ms
          Feb 23, 2014 7:08:07 AM hudson.triggers.SafeTimerTask run
          SEVERE: Timer task hudson.node_monitors.AbstractNodeMonitorDescriptor$1@142971b failed
          java.lang.OutOfMemoryError: unable to create new native thread
          at java.lang.Thread.start0(Native Method)
          at java.lang.Thread.start(Unknown Source)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor.triggerUpdate(AbstractNodeMonitorDescriptor.java:181)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor$1.doRun(AbstractNodeMonitorDescriptor.java:78)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
          at java.util.TimerThread.mainLoop(Unknown Source)
          at java.util.TimerThread.run(Unknown Source)
          Feb 23, 2014 7:15:08 AM winstone.Logger logInternal
          WARNING: Untrapped Error in Servlet
          java.lang.OutOfMemoryError: unable to create new native thread
          at java.lang.Thread.start0(Native Method)
          at java.lang.Thread.start(Unknown Source)
          at com4j.ComThread.<init>(ComThread.java:63)
          at com4j.ComThread.<init>(ComThread.java:21)
          at com4j.ComThread$1.initialValue(ComThread.java:31)
          at com4j.ComThread$1.initialValue(ComThread.java:26)
          at java.lang.ThreadLocal.setInitialValue(Unknown Source)
          at java.lang.ThreadLocal.get(Unknown Source)
          at com4j.ComThread.get(ComThread.java:39)
          at com4j.Task.execute(Task.java:25)
          at com4j.COM4J.createInstance(COM4J.java:97)
          at com4j.COM4J.createInstance(COM4J.java:72)
          at com4j.typelibs.ado20.ClassFactory.createCommand(ClassFactory.java:17)
          at hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider.getDnOfUserOrGroup(ActiveDirectoryAuthenticationProvider.java:162)
          at hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider.retrieveUser(ActiveDirectoryAuthenticationProvider.java:61)
          at org.acegisecurity.providers.dao.AbstractUserDetailsAuthenticationProvider.authenticate(AbstractUserDetailsAuthenticationProvider.java:119)
          at org.acegisecurity.providers.ProviderManager.doAuthentication(ProviderManager.java:195)
          at org.acegisecurity.AbstractAuthenticationManager.authenticate(AbstractAuthenticationManager.java:45)
          at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:138)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
          at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
          at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
          at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
          at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
          at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
          at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
          at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
          at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
          at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
          at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
          at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
          at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)
          at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)
          at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
          at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at java.lang.Thread.run(Unknown Source)
          Feb 23, 2014 7:15:08 AM winstone.Logger logInternal
          WARNING: Untrapped Error in Servlet
          java.lang.OutOfMemoryError: unable to create new native thread
          at java.lang.Thread.start0(Native Method)
          at java.lang.Thread.start(Unknown Source)
          at com4j.ComThread.<init>(ComThread.java:63)
          at com4j.ComThread.<init>(ComThread.java:21)
          at com4j.ComThread$1.initialValue(ComThread.java:31)
          at com4j.ComThread$1.initialValue(ComThread.java:26)
          at java.lang.ThreadLocal.setInitialValue(Unknown Source)
          at java.lang.ThreadLocal.get(Unknown Source)
          at com4j.ComThread.get(ComThread.java:39)
          at com4j.Task.execute(Task.java:25)
          at com4j.COM4J.createInstance(COM4J.java:97)
          at com4j.COM4J.createInstance(COM4J.java:72)
          at com4j.typelibs.ado20.ClassFactory.createCommand(ClassFactory.java:17)
          at hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider.getDnOfUserOrGroup(ActiveDirectoryAuthenticationProvider.java:162)
          at hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider.retrieveUser(ActiveDirectoryAuthenticationProvider.java:61)
          at org.acegisecurity.providers.dao.AbstractUserDetailsAuthenticationProvider.authenticate(AbstractUserDetailsAuthenticationProvider.java:119)
          at org.acegisecurity.providers.ProviderManager.doAuthentication(ProviderManager.java:195)
          at org.acegisecurity.AbstractAuthenticationManager.authenticate(AbstractAuthenticationManager.java:45)
          at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:138)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
          at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
          at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
          at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
          at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
          at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
          at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
          at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
          at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
          at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
          at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
          at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
          at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)
          at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)
          at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
          at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at java.lang.Thread.run(Unknown Source)
          Feb 23, 2014 7:15:08 AM winstone.Logger logInternal
          SEVERE: Error during HTTP listener init or shutdown
          java.lang.OutOfMemoryError: unable to create new native thread
          at java.lang.Thread.start0(Native Method)
          at java.lang.Thread.start(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
          at winstone.BoundedExecutorService.scheduleNext(BoundedExecutorService.java:74)
          at winstone.BoundedExecutorService.execute(BoundedExecutorService.java:64)
          at java.util.concurrent.AbstractExecutorService.submit(Unknown Source)
          at winstone.ObjectPool.handleRequest(ObjectPool.java:102)
          at winstone.HttpListener.run(HttpListener.java:144)
          at java.lang.Thread.run(Unknown Source)
          Feb 23, 2014 8:08:07 AM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record <init>
          WARNING: Previous Free Swap Space monitoring activity still in progress. Interrupting
          Feb 23, 2014 2:53:11 PM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started Fingerprint cleanup
          Feb 23, 2014 2:53:11 PM hudson.model.FingerprintCleanupThread execute
          INFO: Cleaned up 0 records

          VladDRussian added a comment - Thought i'd throw in my errors. We are seeing HTTP stop responding and visualVM can't take a dump once it fails. Memory is fine, but threads just keep increasing, once they hit 550-600 active it just stops and says memory issue won't respond till the master is restarted. Jenkins 1.484, win2k3 (master) win2k8/win2k3 (slaves) Feb 25, 2014 9:21:06 AM hudson.model.DownloadService$Downloadable doPostBack INFO: Obtained the updated data file for hudson.tools.JDKInstaller Feb 25, 2014 9:21:06 AM hudson.model.DownloadService$Downloadable doPostBack INFO: Obtained the updated data file for hudson.tasks.Ant.AntInstaller Feb 25, 2014 9:21:06 AM org.kohsuke.stapler.compression.CompressionFilter reportException WARNING: Untrapped servlet exception java.io.IOException: Unable to delete E:\jenkins\updates\hudson.tasks.Maven.MavenInstaller at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:112) at hudson.util.TextFile.write(TextFile.java:81) at hudson.model.DownloadService$Downloadable.doPostBack(DownloadService.java:260) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:288) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:151) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:90) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659) at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:384) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:203) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:488) at org.kohsuke.stapler.Stapler.service(Stapler.java:162) at javax.servlet.http.HttpServlet.service(HttpServlet.java:45) at winstone.ServletConfiguration.execute(ServletConfiguration.java:248) at winstone.RequestDispatcher.forward(RequestDispatcher.java:333) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) FREE DISK SPACE MONITORING STILL IN PROGRESS, FAILED, THREADS LEFT, MAXING Feb 21, 2014 4:55:20 PM hudson.TcpSlaveAgentListener$ConnectionHandler run INFO: Accepted connection #12 from /172.16.69.140:64901 Feb 21, 2014 4:55:21 PM hudson.TcpSlaveAgentListener$ConnectionHandler run INFO: Accepted connection #13 from /172.16.69.82:61763 Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record <init> WARNING: Previous Free Swap Space monitoring activity still in progress. Interrupting Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record <init> WARNING: Previous Free Disk Space monitoring activity still in progress. Interrupting Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record <init> WARNING: Previous Free Temp Space monitoring activity still in progress. Interrupting Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run WARNING: Node monitoring FFX-MOM-BLD10 for Free Swap Space aborted. java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:146) at hudson.remoting.Channel.call(Channel.java:664) at hudson.node_monitors.SwapSpaceMonitor$1.monitor(SwapSpaceMonitor.java:83) at hudson.node_monitors.SwapSpaceMonitor$1.monitor(SwapSpaceMonitor.java:81) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219) Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run WARNING: Node monitoring FFX-MOM-BLD5 for Free Temp Space aborted. java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:146) at hudson.remoting.Channel.call(Channel.java:664) at hudson.FilePath.act(FilePath.java:840) at hudson.FilePath.act(FilePath.java:824) at hudson.node_monitors.TemporarySpaceMonitor$1.getFreeSpace(TemporarySpaceMonitor.java:73) at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:169) at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:49) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219) Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run WARNING: Node monitoring FFX-MOM-BLD5 for Free Disk Space aborted. java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:146) at hudson.remoting.Channel.call(Channel.java:664) at hudson.FilePath.act(FilePath.java:840) at hudson.FilePath.act(FilePath.java:824) at hudson.node_monitors.DiskSpaceMonitor$1.getFreeSpace(DiskSpaceMonitor.java:71) at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:169) at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:49) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219) BELOW LEAVING OPEN THREADS AND MAXING???? Feb 20, 2014 6:32:49 AM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run WARNING: Failed to monitor FFX-MOM-BLD10 for Free Temp Space hudson.util.IOException2: remote file operation failed: E:\jenkins at hudson.remoting.Channel@18b622:FFX-MOM-BLD10 at hudson.FilePath.act(FilePath.java:847) at hudson.FilePath.act(FilePath.java:824) at hudson.node_monitors.TemporarySpaceMonitor$1.getFreeSpace(TemporarySpaceMonitor.java:73) at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:169) at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:49) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219) Caused by: java.io.IOException: Remote call on FFX-MOM-BLD10 failed at hudson.remoting.Channel.call(Channel.java:673) at hudson.FilePath.act(FilePath.java:840) ... 5 more Caused by: java.lang.Error: Failed to deserialize the Callable object. at hudson.remoting.UserRequest.perform(UserRequest.java:104) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:60) at java.lang.Thread.run(Unknown Source) Caused by: java.lang.reflect.UndeclaredThrowableException at $Proxy6.fetch2(Unknown Source) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:122) at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClassInternal(Unknown Source) at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Unknown Source) at java.lang.Class.getDeclaredMethod(Unknown Source) at java.io.ObjectStreamClass.getPrivateMethod(Unknown Source) at java.io.ObjectStreamClass.access$1700(Unknown Source) at java.io.ObjectStreamClass$2.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.<init>(Unknown Source) at java.io.ObjectStreamClass.lookup(Unknown Source) at java.io.ObjectStreamClass.initNonProxy(Unknown Source) at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source) at java.io.ObjectInputStream.readClassDesc(Unknown Source) at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.defaultReadFields(Unknown Source) at java.io.ObjectInputStream.readSerialData(Unknown Source) at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.UserRequest.deserialize(UserRequest.java:182) at hudson.remoting.UserRequest.perform(UserRequest.java:98) ... 9 more Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:146) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:160) ... 35 more Feb 20, 2014 6:32:49 AM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run WARNING: Failed to monitor FFX-MOM-BLD12 for Free Swap Space java.io.IOException: Remote call on FFX-MOM-BLD12 failed at hudson.remoting.Channel.call(Channel.java:673) at hudson.node_monitors.SwapSpaceMonitor$1.monitor(SwapSpaceMonitor.java:83) at hudson.node_monitors.SwapSpaceMonitor$1.monitor(SwapSpaceMonitor.java:81) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219) Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.jvnet.hudson.Windows$Kernel32 at org.jvnet.hudson.Windows.monitor(Windows.java:41) at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:113) at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:99) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at hudson.remoting.Engine$1$1.run(Engine.java:60) at java.lang.Thread.run(Thread.java:619) Feb 20, 2014 7:32:49 AM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run WARNING: Failed to monitor FFX-MOM-BLD10 for Free Temp Space hudson.util.IOException2: remote file operation failed: E:\jenkins at hudson.remoting.Channel@18b622:FFX-MOM-BLD10 at hudson.FilePath.act(FilePath.java:847) at hudson.FilePath.act(FilePath.java:824) at hudson.node_monitors.TemporarySpaceMonitor$1.getFreeSpace(TemporarySpaceMonitor.java:73) at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:169) at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:49) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219) Caused by: java.io.IOException: Remote call on FFX-MOM-BLD10 failed at hudson.remoting.Channel.call(Channel.java:673) at hudson.FilePath.act(FilePath.java:840) ... 5 more Caused by: java.lang.Error: Failed to deserialize the Callable object. at hudson.remoting.UserRequest.perform(UserRequest.java:104) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:60) at java.lang.Thread.run(Unknown Source) Caused by: java.lang.reflect.UndeclaredThrowableException at $Proxy6.fetch2(Unknown Source) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:122) at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClassInternal(Unknown Source) at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Unknown Source) at java.lang.Class.getDeclaredMethod(Unknown Source) at java.io.ObjectStreamClass.getPrivateMethod(Unknown Source) at java.io.ObjectStreamClass.access$1700(Unknown Source) at java.io.ObjectStreamClass$2.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.<init>(Unknown Source) at java.io.ObjectStreamClass.lookup(Unknown Source) at java.io.ObjectStreamClass.initNonProxy(Unknown Source) at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source) at java.io.ObjectInputStream.readClassDesc(Unknown Source) at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.defaultReadFields(Unknown Source) at java.io.ObjectInputStream.readSerialData(Unknown Source) at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.UserRequest.deserialize(UserRequest.java:182) at hudson.remoting.UserRequest.perform(UserRequest.java:98) ... 9 more Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:146) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:160) ... 35 more Feb 20, 2014 7:50:42 AM winstone.Logger logInternal SEVERE: Error during HTTP listener init or shutdown java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) at winstone.BoundedExecutorService.scheduleNext(BoundedExecutorService.java:74) at winstone.BoundedExecutorService.execute(BoundedExecutorService.java:64) at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) at winstone.ObjectPool.handleRequest(ObjectPool.java:102) at winstone.HttpListener.run(HttpListener.java:144) at java.lang.Thread.run(Unknown Source) Feb 20, 2014 7:50:45 AM org.kohsuke.stapler.compression.CompressionFilter reportException WARNING: Untrapped servlet exception winstone.ClientSocketException: Failed to write to client at winstone.ClientOutputStream.write(ClientOutputStream.java:41) at java.io.BufferedOutputStream.flushBuffer(Unknown Source) at java.io.BufferedOutputStream.flush(Unknown Source) at winstone.WinstoneOutputStream.commit(WinstoneOutputStream.java:167) at winstone.WinstoneOutputStream.commit(WinstoneOutputStream.java:123) at winstone.WinstoneOutputStream.write(WinstoneOutputStream.java:112) at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:421) at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:249) at org.kohsuke.stapler.Stapler.service(Stapler.java:152) at javax.servlet.http.HttpServlet.service(HttpServlet.java:45) at winstone.ServletConfiguration.execute(ServletConfiguration.java:248) at winstone.RequestDispatcher.forward(RequestDispatcher.java:333) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketException: Connection reset by peer: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(Unknown Source) at java.net.SocketOutputStream.write(Unknown Source) at winstone.ClientOutputStream.write(ClientOutputStream.java:39) ... 57 more REACHED LIMIT AND FAIL Feb 23, 2014 5:07:53 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started Workspace clean-up Feb 23, 2014 5:07:56 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished Workspace clean-up. 3156 ms Feb 23, 2014 7:08:07 AM hudson.triggers.SafeTimerTask run SEVERE: Timer task hudson.node_monitors.AbstractNodeMonitorDescriptor$1@142971b failed java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Unknown Source) at hudson.node_monitors.AbstractNodeMonitorDescriptor.triggerUpdate(AbstractNodeMonitorDescriptor.java:181) at hudson.node_monitors.AbstractNodeMonitorDescriptor$1.doRun(AbstractNodeMonitorDescriptor.java:78) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.TimerThread.mainLoop(Unknown Source) at java.util.TimerThread.run(Unknown Source) Feb 23, 2014 7:15:08 AM winstone.Logger logInternal WARNING: Untrapped Error in Servlet java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Unknown Source) at com4j.ComThread.<init>(ComThread.java:63) at com4j.ComThread.<init>(ComThread.java:21) at com4j.ComThread$1.initialValue(ComThread.java:31) at com4j.ComThread$1.initialValue(ComThread.java:26) at java.lang.ThreadLocal.setInitialValue(Unknown Source) at java.lang.ThreadLocal.get(Unknown Source) at com4j.ComThread.get(ComThread.java:39) at com4j.Task.execute(Task.java:25) at com4j.COM4J.createInstance(COM4J.java:97) at com4j.COM4J.createInstance(COM4J.java:72) at com4j.typelibs.ado20.ClassFactory.createCommand(ClassFactory.java:17) at hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider.getDnOfUserOrGroup(ActiveDirectoryAuthenticationProvider.java:162) at hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider.retrieveUser(ActiveDirectoryAuthenticationProvider.java:61) at org.acegisecurity.providers.dao.AbstractUserDetailsAuthenticationProvider.authenticate(AbstractUserDetailsAuthenticationProvider.java:119) at org.acegisecurity.providers.ProviderManager.doAuthentication(ProviderManager.java:195) at org.acegisecurity.AbstractAuthenticationManager.authenticate(AbstractAuthenticationManager.java:45) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:138) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Feb 23, 2014 7:15:08 AM winstone.Logger logInternal WARNING: Untrapped Error in Servlet java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Unknown Source) at com4j.ComThread.<init>(ComThread.java:63) at com4j.ComThread.<init>(ComThread.java:21) at com4j.ComThread$1.initialValue(ComThread.java:31) at com4j.ComThread$1.initialValue(ComThread.java:26) at java.lang.ThreadLocal.setInitialValue(Unknown Source) at java.lang.ThreadLocal.get(Unknown Source) at com4j.ComThread.get(ComThread.java:39) at com4j.Task.execute(Task.java:25) at com4j.COM4J.createInstance(COM4J.java:97) at com4j.COM4J.createInstance(COM4J.java:72) at com4j.typelibs.ado20.ClassFactory.createCommand(ClassFactory.java:17) at hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider.getDnOfUserOrGroup(ActiveDirectoryAuthenticationProvider.java:162) at hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider.retrieveUser(ActiveDirectoryAuthenticationProvider.java:61) at org.acegisecurity.providers.dao.AbstractUserDetailsAuthenticationProvider.authenticate(AbstractUserDetailsAuthenticationProvider.java:119) at org.acegisecurity.providers.ProviderManager.doAuthentication(ProviderManager.java:195) at org.acegisecurity.AbstractAuthenticationManager.authenticate(AbstractAuthenticationManager.java:45) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:138) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Feb 23, 2014 7:15:08 AM winstone.Logger logInternal SEVERE: Error during HTTP listener init or shutdown java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) at winstone.BoundedExecutorService.scheduleNext(BoundedExecutorService.java:74) at winstone.BoundedExecutorService.execute(BoundedExecutorService.java:64) at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) at winstone.ObjectPool.handleRequest(ObjectPool.java:102) at winstone.HttpListener.run(HttpListener.java:144) at java.lang.Thread.run(Unknown Source) Feb 23, 2014 8:08:07 AM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record <init> WARNING: Previous Free Swap Space monitoring activity still in progress. Interrupting Feb 23, 2014 2:53:11 PM hudson.model.AsyncPeriodicWork$1 run INFO: Started Fingerprint cleanup Feb 23, 2014 2:53:11 PM hudson.model.FingerprintCleanupThread execute INFO: Cleaned up 0 records

          I am facing the same or similar problem.
          I have Jenkins linux master and slaves on the Proxmox Windows XP machines.
          Time to time, a slave stuck.
          The cosole output ends with line - building remotely on ...
          No SVN checkout is executed after, the build is totally stuck.
          The slave says: Slave successfully connected and online

          I have to restart the slave machine and abort the job. If i only abort the job, the next build is stuck too.

          Jan Řezníček added a comment - I am facing the same or similar problem. I have Jenkins linux master and slaves on the Proxmox Windows XP machines. Time to time, a slave stuck. The cosole output ends with line - building remotely on ... No SVN checkout is executed after, the build is totally stuck. The slave says: Slave successfully connected and online I have to restart the slave machine and abort the job. If i only abort the job, the next build is stuck too.

          This is caused by a deadlock with the subversion plugin and the "Free swap space" monitor option. To prevent this issue you can go to Manage Jenkins > Manage Nodes > Configure and disable this option.

          Steven Christou added a comment - This is caused by a deadlock with the subversion plugin and the "Free swap space" monitor option. To prevent this issue you can go to Manage Jenkins > Manage Nodes > Configure and disable this option.

          Daniel Beck added a comment -

          schristou

          To prevent this issue you can go to Manage Jenkins > Manage Nodes > Configure and disable this option.

          Aren't node monitors still run, just won't take nodes offline that fail the check?

          Daniel Beck added a comment - schristou To prevent this issue you can go to Manage Jenkins > Manage Nodes > Configure and disable this option. Aren't node monitors still run, just won't take nodes offline that fail the check?

          Dave Taddei added a comment -

          We are seeing this intermittently on Jenkins 1.656 with a windows 7 remote slave. We have disabled the Free swap space monitoring option but still see the issue.

          Dave Taddei added a comment - We are seeing this intermittently on Jenkins 1.656 with a windows 7 remote slave. We have disabled the Free swap space monitoring option but still see the issue.

          BBVA added a comment -

          We are also experiencing this problem. We have disabled all monitoring on the slaves.

          We updated to the latest SVN plugin 2.5.7, including SVNKit 1.8.11

          Seems like the problem is a deadlock in the startup of the windows slave. We are using cygwin/ssh connections, if that matters. Looking at the stack traces with SVN involved*:

          The java method getApplicationDataPath() is loading the JNA, which is synchronized in all of it's static methods. So as this is the first time the class is refenced is also doing the static initialization <clinit>. Looks like the Native.loadLibrary is stuck and producing the blockage, to all the threads using SVN.

          *) The actual thread stack dump. Involving SVN.

          "pool-1-thread-3 for channel" prio=5 RUNNABLE
          com.sun.jna.Native.initIDs(Native Method)
          com.sun.jna.Native.<clinit>(Native.java:122)
          org.tmatesoft.svn.core.internal.util.jna.JNALibraryLoader.<clinit>(JNALibraryLoader.java:50)
          org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:113)
          org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196)
          org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1948)
          org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57)
          hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:1055)
          hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1032)
          hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:965)
          hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:945)
          hudson.FilePath$FileCallableWrapper.call(FilePath.java:2577)
          hudson.remoting.UserRequest.perform(UserRequest.java:121)
          hudson.remoting.UserRequest.perform(UserRequest.java:49)
          hudson.remoting.Request$2.run(Request.java:324)
          hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
          java.util.concurrent.FutureTask.run(FutureTask.java:262)
          java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          java.lang.Thread.run(Thread.java:745)

          "pool-1-thread-52 for channel" prio=5 RUNNABLE
          org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:113)
          org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196)
          org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1948)
          org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57)
          hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:1055)
          hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1032)
          hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:965)
          hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:945)
          hudson.FilePath$FileCallableWrapper.call(FilePath.java:2577)
          hudson.remoting.UserRequest.perform(UserRequest.java:121)
          hudson.remoting.UserRequest.perform(UserRequest.java:49)
          hudson.remoting.Request$2.run(Request.java:324)
          hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
          java.util.concurrent.FutureTask.run(FutureTask.java:262)
          java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          java.lang.Thread.run(Thread.java:745)

          BBVA added a comment - We are also experiencing this problem. We have disabled all monitoring on the slaves. We updated to the latest SVN plugin 2.5.7, including SVNKit 1.8.11 Seems like the problem is a deadlock in the startup of the windows slave. We are using cygwin/ssh connections, if that matters. Looking at the stack traces with SVN involved*: The java method getApplicationDataPath() is loading the JNA, which is synchronized in all of it's static methods. So as this is the first time the class is refenced is also doing the static initialization <clinit>. Looks like the Native.loadLibrary is stuck and producing the blockage, to all the threads using SVN. *) The actual thread stack dump. Involving SVN. "pool-1-thread-3 for channel" prio=5 RUNNABLE com.sun.jna.Native.initIDs(Native Method) com.sun.jna.Native.<clinit>(Native.java:122) org.tmatesoft.svn.core.internal.util.jna.JNALibraryLoader.<clinit>(JNALibraryLoader.java:50) org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:113) org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196) org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1948) org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57) hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:1055) hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1032) hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:965) hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:945) hudson.FilePath$FileCallableWrapper.call(FilePath.java:2577) hudson.remoting.UserRequest.perform(UserRequest.java:121) hudson.remoting.UserRequest.perform(UserRequest.java:49) hudson.remoting.Request$2.run(Request.java:324) hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) java.util.concurrent.FutureTask.run(FutureTask.java:262) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) "pool-1-thread-52 for channel" prio=5 RUNNABLE org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:113) org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196) org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1948) org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57) hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:1055) hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1032) hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:965) hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:945) hudson.FilePath$FileCallableWrapper.call(FilePath.java:2577) hudson.remoting.UserRequest.perform(UserRequest.java:121) hudson.remoting.UserRequest.perform(UserRequest.java:49) hudson.remoting.Request$2.run(Request.java:324) hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) java.util.concurrent.FutureTask.run(FutureTask.java:262) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745)

          BBVA added a comment -

          @See https://issues.tmatesoft.com/issue/SGT-451

          Slaves need to force load of jna classes at startup.

          BBVA added a comment - @See https://issues.tmatesoft.com/issue/SGT-451 Slaves need to force load of jna classes at startup.

          BBVA added a comment -

          FYI, we have a patch for jenkins/remoting-2.47 that allows to by-pass this issue by defining property hudson.remoting.RemoteClassLoader.force to com.sun.jna.Native. This forces the load of the offending class early on at any UserRequest with a new ClassLoader.

          We have tested this patch timed job every 5 minutes, with a dormant slave that needs to be waken. We will be requesting merge in github master shortly.

          BBVA added a comment - FYI, we have a patch for jenkins/remoting-2.47 that allows to by-pass this issue by defining property hudson.remoting.RemoteClassLoader.force to com.sun.jna.Native . This forces the load of the offending class early on at any UserRequest with a new ClassLoader. We have tested this patch timed job every 5 minutes, with a dormant slave that needs to be waken. We will be requesting merge in github master shortly.

          Daniel Beck added a comment -

          FYI oleg_nenashev See above.

          Daniel Beck added a comment - FYI oleg_nenashev See above.

          Dorian Daumiller added a comment - - edited

          Deactivating the (all) monitorings on the Jenkins master did not help us. Before reconnecting the swarm client, we see this:

          Curiously, we also still see the monitoring in the Jenkins log:
          {{
          May 09, 2016 9:38:39 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor

          Failed to monitor ****-ed055395 for Free Swap Space
          java.util.concurrent.TimeoutException
          at hudson.remoting.Request$1.get(Request.java:273)
          at hudson.remoting.Request$1.get(Request.java:207)
          at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
          }}

          On the build node, amidst tons of threads, we see two that are referencing org.tmatesoft.svn.core.internal.util.jna:

          {{pool-1-thread-2 for channel

          "pool-1-thread-2 for channel" Id=16 Group=main RUNNABLE
          at com.sun.jna.Native.initIDs(Native Method)
          at com.sun.jna.Native.<clinit>(Native.java:148)
          at org.tmatesoft.svn.core.internal.util.jna.JNALibraryLoader.<clinit>(JNALibraryLoader.java:50)
          at org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:113)
          at org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196)
          at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1948)
          at org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57)
          at hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:1055)
          at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1032)
          at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:965)
          at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:945)
          at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2719)
          at hudson.remoting.UserRequest.perform(UserRequest.java:121)
          at hudson.remoting.UserRequest.perform(UserRequest.java:49)
          at hudson.remoting.Request$2.run(Request.java:324)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at hudson.remoting.Engine$1$1.run(Engine.java:63)
          at java.lang.Thread.run(Unknown Source)

          Number of locked synchronizers = 1

          • java.util.concurrent.ThreadPoolExecutor$Worker@5f599d56
            }}
            {{
            pool-1-thread-916 for channel

          "pool-1-thread-916 for channel" Id=930 Group=main RUNNABLE
          at org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:113)
          at org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196)
          at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1948)
          at org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57)
          at hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:1055)
          at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1032)
          at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:965)
          at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:945)
          at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2719)
          at hudson.remoting.UserRequest.perform(UserRequest.java:121)
          at hudson.remoting.UserRequest.perform(UserRequest.java:49)
          at hudson.remoting.Request$2.run(Request.java:324)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at hudson.remoting.Engine$1$1.run(Engine.java:63)
          at java.lang.Thread.run(Unknown Source)

          Number of locked synchronizers = 1

          • java.util.concurrent.ThreadPoolExecutor$Worker@772c6f4f
            }}

          After reconnecting two swarm clients (but not restarting Jenkins), one builds, one hangs, showing similar traces.
          From the hanging client:

          {{pool-1-thread-6 for channel

          "pool-1-thread-6 for channel" Id=20 Group=main RUNNABLE
          at com.sun.jna.Pointer.<clinit>(Pointer.java:41)
          at com.sun.jna.Structure.<clinit>(Structure.java:2078)
          at org.jvnet.hudson.Windows.monitor(Windows.java:42)
          at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:113)
          at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:103)
          at hudson.remoting.UserRequest.perform(UserRequest.java:121)
          at hudson.remoting.UserRequest.perform(UserRequest.java:49)
          at hudson.remoting.Request$2.run(Request.java:324)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at hudson.remoting.Engine$1$1.run(Engine.java:63)
          at java.lang.Thread.run(Unknown Source)

          Number of locked synchronizers = 1

          • java.util.concurrent.ThreadPoolExecutor$Worker@2e201c5c}}

          It seems the FreeSwapMonitoring starts a new thread on the node every time it is started (we can start it manually using the "Refresh Status" Button in the BuildExecutor Status view).
          We see that some threads are actually closed, so it's not that all threads hang, but for every time we hit refresh status, we see a new instance of at ```org.jvnet.hudson.Windows.monitor``` (if we wait some minute between hitting that button)

          Dorian Daumiller added a comment - - edited Deactivating the (all) monitorings on the Jenkins master did not help us. Before reconnecting the swarm client, we see this: Curiously, we also still see the monitoring in the Jenkins log: {{ May 09, 2016 9:38:39 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor Failed to monitor ****-ed055395 for Free Swap Space java.util.concurrent.TimeoutException at hudson.remoting.Request$1.get(Request.java:273) at hudson.remoting.Request$1.get(Request.java:207) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) }} On the build node, amidst tons of threads, we see two that are referencing org.tmatesoft.svn.core.internal.util.jna : {{pool-1-thread-2 for channel "pool-1-thread-2 for channel" Id=16 Group=main RUNNABLE at com.sun.jna.Native.initIDs(Native Method) at com.sun.jna.Native.<clinit>(Native.java:148) at org.tmatesoft.svn.core.internal.util.jna.JNALibraryLoader.<clinit>(JNALibraryLoader.java:50) at org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:113) at org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196) at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1948) at org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57) at hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:1055) at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1032) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:965) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:945) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2719) at hudson.remoting.UserRequest.perform(UserRequest.java:121) at hudson.remoting.UserRequest.perform(UserRequest.java:49) at hudson.remoting.Request$2.run(Request.java:324) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@5f599d56 }} {{ pool-1-thread-916 for channel "pool-1-thread-916 for channel" Id=930 Group=main RUNNABLE at org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:113) at org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196) at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1948) at org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57) at hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:1055) at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1032) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:965) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:945) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2719) at hudson.remoting.UserRequest.perform(UserRequest.java:121) at hudson.remoting.UserRequest.perform(UserRequest.java:49) at hudson.remoting.Request$2.run(Request.java:324) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@772c6f4f }} After reconnecting two swarm clients (but not restarting Jenkins), one builds, one hangs, showing similar traces. From the hanging client: {{pool-1-thread-6 for channel "pool-1-thread-6 for channel" Id=20 Group=main RUNNABLE at com.sun.jna.Pointer.<clinit>(Pointer.java:41) at com.sun.jna.Structure.<clinit>(Structure.java:2078) at org.jvnet.hudson.Windows.monitor(Windows.java:42) at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:113) at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:103) at hudson.remoting.UserRequest.perform(UserRequest.java:121) at hudson.remoting.UserRequest.perform(UserRequest.java:49) at hudson.remoting.Request$2.run(Request.java:324) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@2e201c5c}} It seems the FreeSwapMonitoring starts a new thread on the node every time it is started (we can start it manually using the "Refresh Status" Button in the BuildExecutor Status view). We see that some threads are actually closed, so it's not that all threads hang, but for every time we hit refresh status, we see a new instance of at ```org.jvnet.hudson.Windows.monitor``` (if we wait some minute between hitting that button)

          Code changed in jenkins
          User: Jenkins BBVA
          Path:
          src/main/java/hudson/remoting/UserRequest.java
          http://jenkins-ci.org/commit/remoting/8e0a8667a04dfd8d3509a08dc693de31cd5d2e77
          Log:
          JENKINS-19445 Force class load on UserRequest (#82)

          This is a workaround to force class load of any class. This helps prevent deadlock on windows nodes when using JNA and Subversion.
          Use property hudson.remoting.RemoteClassLoader.force to name the class to load. Ideally this forced load should happen earlier on the startup, however the classloader isn't available.

          • Update UserRequest as per code review comments

          Fixes based on code review of pull-request: https://github.com/jenkinsci/remoting/pull/82

          • Fix Logger.log() invocation

          Fix Logger.log() invocation: needs Object[], not vargs.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jenkins BBVA Path: src/main/java/hudson/remoting/UserRequest.java http://jenkins-ci.org/commit/remoting/8e0a8667a04dfd8d3509a08dc693de31cd5d2e77 Log: JENKINS-19445 Force class load on UserRequest (#82) Force class load on UserRequest, see JENKINS-19445 This is a workaround to force class load of any class. This helps prevent deadlock on windows nodes when using JNA and Subversion. Use property hudson.remoting.RemoteClassLoader.force to name the class to load. Ideally this forced load should happen earlier on the startup, however the classloader isn't available. Update UserRequest as per code review comments Fixes based on code review of pull-request: https://github.com/jenkinsci/remoting/pull/82 Fix Logger.log() invocation Fix Logger.log() invocation: needs Object[], not vargs.

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          pom.xml
          http://jenkins-ci.org/commit/jenkins/409438f36dc80f20964fb16f8d88041e11ba4ed4
          Log:
          [JENKINS-19445, JENKINS-34213, JENKINS-34808, JENKINS-34121] Bump remoting to 2.59. (#2344)

          • [JENKINS-19445, JENKINS-34213, JENKINS-34808] Bump remoting to 2.58.

          Changes:

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: pom.xml http://jenkins-ci.org/commit/jenkins/409438f36dc80f20964fb16f8d88041e11ba4ed4 Log: [JENKINS-19445, JENKINS-34213, JENKINS-34808, JENKINS-34121] Bump remoting to 2.59. (#2344) [JENKINS-19445, JENKINS-34213, JENKINS-34808] Bump remoting to 2.58. Changes: JENKINS-34213 ( https://issues.jenkins-ci.org/browse/JENKINS-34213 ) - Ensure that the unexporter cleans up whatever it can each sweep ( https://github.com/jenkinsci/remoting/pull/81 ) JENKINS-19445 ( https://issues.jenkins-ci.org/browse/JENKINS-19445 ) Force class load on UserRequest in order to prevent deadlock on windows nodes when using JNA and Subversion ( https://github.com/jenkinsci/remoting/pull/81 ) JENKINS-34808 ( https://issues.jenkins-ci.org/browse/JENKINS-34808 ) - Allow user to adjust socket timeout ( https://github.com/jenkinsci/remoting/pull/68 ) JENKINS-34121 - Upgrade remoting to 2.59

          Can we agree on having this in LTS?

          Dorian Daumiller added a comment - Can we agree on having this in LTS?

          Oleg Nenashev added a comment -

          ddaumiller
          Jenkins 2 LTS - yes
          1.651.3 - under review: https://github.com/jenkinsci/jenkins/pull/2357

          Oleg Nenashev added a comment - ddaumiller Jenkins 2 LTS - yes 1.651.3 - under review: https://github.com/jenkinsci/jenkins/pull/2357

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          pom.xml
          http://jenkins-ci.org/commit/jenkins/12e79963cca5122351943ee107f65c3ad91a2e25
          Log:
          [JENKINS-19445, JENKINS-34213, JENKINS-34808, JENKINS-34121] Bump remoting to 2.59. (#2344)

          • [JENKINS-19445, JENKINS-34213, JENKINS-34808] Bump remoting to 2.58.

          Changes:

          (cherry picked from commit 409438f36dc80f20964fb16f8d88041e11ba4ed4)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: pom.xml http://jenkins-ci.org/commit/jenkins/12e79963cca5122351943ee107f65c3ad91a2e25 Log: [JENKINS-19445, JENKINS-34213, JENKINS-34808, JENKINS-34121] Bump remoting to 2.59. (#2344) [JENKINS-19445, JENKINS-34213, JENKINS-34808] Bump remoting to 2.58. Changes: JENKINS-34213 ( https://issues.jenkins-ci.org/browse/JENKINS-34213 ) - Ensure that the unexporter cleans up whatever it can each sweep ( https://github.com/jenkinsci/remoting/pull/81 ) JENKINS-19445 ( https://issues.jenkins-ci.org/browse/JENKINS-19445 ) Force class load on UserRequest in order to prevent deadlock on windows nodes when using JNA and Subversion ( https://github.com/jenkinsci/remoting/pull/81 ) JENKINS-34808 ( https://issues.jenkins-ci.org/browse/JENKINS-34808 ) - Allow user to adjust socket timeout ( https://github.com/jenkinsci/remoting/pull/68 ) JENKINS-34121 - Upgrade remoting to 2.59 (cherry picked from commit 409438f36dc80f20964fb16f8d88041e11ba4ed4)

          Oleg Nenashev added a comment -

          Released as Jenkins-2.4

          Oleg Nenashev added a comment - Released as Jenkins-2.4

          Dave Taddei added a comment - - edited

          We are showing this happening on the latest jenkins build (2.17) and latest slave jar (2.60) on a Windows 7 64-bit slave. Appears to be an intermittent fault but almost always occurs when an on-demand build is taking place.

          Dave Taddei added a comment - - edited We are showing this happening on the latest jenkins build (2.17) and latest slave jar (2.60) on a Windows 7 64-bit slave. Appears to be an intermittent fault but almost always occurs when an on-demand build is taking place.

          Daniel Beck added a comment -

          drtaddei

          latest jenkins build … latest slave jar

          Please be more specific. What version?

          Daniel Beck added a comment - drtaddei latest jenkins build … latest slave jar Please be more specific. What version?

          BBVA added a comment -

          Dave Taddei, can you confirm you are using the property hudson.remoting.RemoteClassLoader.force to force the load of com.sun.jna.Native. Also that you can see that the class is being loaded via the log of the slave.

          BBVA added a comment - Dave Taddei, can you confirm you are using the property hudson.remoting.RemoteClassLoader.force to force the load of com.sun.jna.Native. Also that you can see that the class is being loaded via the log of the slave.

          Dave Taddei added a comment - - edited

          Hi BBVA, it does not appear so but I am unsure where to look. Should this be set in the JAVA_ARGS for the master, slave or both? I was also under the impression that had been added to release 2.47 so didn't set anything.

          Dave Taddei added a comment - - edited Hi BBVA, it does not appear so but I am unsure where to look. Should this be set in the JAVA_ARGS for the master, slave or both? I was also under the impression that had been added to release 2.47 so didn't set anything.

          BBVA added a comment -

          drtaddei set JVM Options of the slave with: -Dhudson.remoting.RemoteClassLoader.force=com.sun.jna.Native

          BBVA added a comment - drtaddei set JVM Options of the slave with: -Dhudson.remoting.RemoteClassLoader.force=com.sun.jna.Native

          Greg Smith added a comment -

          We are using dynamic slaves, with the vcenter plugin. The vcenter plugin does not allow the specifications of any JVM options.

          At least, I do not believe it does. Could the above change be made permanent, or set as the default for all slaves some how?

          Greg Smith added a comment - We are using dynamic slaves, with the vcenter plugin. The vcenter plugin does not allow the specifications of any JVM options. At least, I do not believe it does. Could the above change be made permanent, or set as the default for all slaves some how?

          Jesse Glick added a comment -

          gregcovertsmith et al.—see the last comment I left in JENKINS-39179: I wrote a little plugin which might work around JNA deadlocks without needing individual configuration of static agents.

          Jesse Glick added a comment - gregcovertsmith et al.—see the last comment I left in  JENKINS-39179 : I wrote a little plugin which might work around JNA deadlocks without needing individual configuration of static agents.

            schristou Steven Christou
            odklizec Pavel Kudrys
            Votes:
            14 Vote for this issue
            Watchers:
            26 Start watching this issue

              Created:
              Updated: