Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19445

Jobs randomly stuck with "building remotely on slave-name" message

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Hi folks,

      Last week, I started to experience a weird issue with freezing/endless waiting of building jobs on slave machines. It first started with WinXP 32 bit slave, but I'm now experiencing the same problem also with WinXP 64 bit slave.

      The symptoms are like this...
      The job is started by SCM change or predefined time.
      Both slave machines are correctly started (the last line, in their respective logs, says "Slave successfully connected and online").
      Now the problem is, that the last line in console says "Building remotely on slave-name..." and there is just scrolling the progress circle. Nothing else happens. No GIT or SVN activity, no building, no testing, simply nothing and it stays like that until I kill the job manually.

      Sometimes, the problem happens with just one machine, and the other one runs OK. But today it struck both machines.

      Last week, when the problem started, I tried to update the Jenkins and both slaves with the most recent versions and latest plugins (except the "Git Client Plugin", because of a known issue with building history). Few hours all worked great, then the problem reappeared.

      Please see the attached files with logs and thread dumps from both slave machines.

      What struck me is the high number of threads started on both machines, which looks really weird? But I don't understand Jenkins that deep so I can't say for sure if it's normal or not?

      Any idea what to try or how to solve this behavior?

      Have a nice day,
      Pavel

        Attachments

        1. LB3D-MASTER_05_09_13.txt
          53 kB
        2. LB3D-MASTER.txt
          41 kB
        3. LB3D-Wxpp32sp3_05_09_13.txt
          6 kB
        4. LB3D-Wxpp32sp3.txt
          89 kB
        5. LB3D-Wxpp64sp2_05_09_13.txt
          7 kB
        6. LB3D-Wxpp64sp2.txt
          92 kB
        7. remoting-2.47.patch
          2 kB
        8. threadump.txt
          9 kB

          Issue Links

            Activity

            Hide
            odklizec Pavel Kudrys added a comment - - edited

            I've added also master thread dump.

            And one more weird thing I just noticed on 64bit machine, which just stuck again.
            At some point, Jenkins console says this this...
            ←[33mSep 04, 2013 12:52:34 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescr
            iptor monitor
            WARNING: Failed to monitor LB3D-Wxpp64sp2 for Free Swap Space
            java.util.concurrent.TimeoutException
            at hudson.remoting.Request$1.get(Request.java:275)
            at hudson.remoting.Request$1.get(Request.java:210)
            at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
            at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(Abstr
            actAsyncNodeMonitorDescriptor.java:97)
            at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(Abstrac
            tNodeMonitorDescriptor.java:282)

            The weird thing is, that I've disabled the "Free Swap Space" monitoring in Jenkins Node configuration section. So how the "Free Swap Space" monitoring could fail on that slave machine? Could it be a cause of broken master-slave connection or maybe corrupted threads?

            Show
            odklizec Pavel Kudrys added a comment - - edited I've added also master thread dump. And one more weird thing I just noticed on 64bit machine, which just stuck again. At some point, Jenkins console says this this... ←[33mSep 04, 2013 12:52:34 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescr iptor monitor WARNING: Failed to monitor LB3D-Wxpp64sp2 for Free Swap Space java.util.concurrent.TimeoutException at hudson.remoting.Request$1.get(Request.java:275) at hudson.remoting.Request$1.get(Request.java:210) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(Abstr actAsyncNodeMonitorDescriptor.java:97) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(Abstrac tNodeMonitorDescriptor.java:282) The weird thing is, that I've disabled the "Free Swap Space" monitoring in Jenkins Node configuration section. So how the "Free Swap Space" monitoring could fail on that slave machine? Could it be a cause of broken master-slave connection or maybe corrupted threads?
            Hide
            odklizec Pavel Kudrys added a comment -

            Hi again. One more piece to the puzzle.

            This is what I saw in log while attempting to run a Jenkins job...

            Sep 05, 2013 7:55:24 AM INFO hudson.slaves.RetentionStrategy$Demand check
            Launching computer LB3D-Wxpp32sp3 as it has been in demand for 1 min 34 sec
            Sep 05, 2013 7:55:24 AM INFO hudson.slaves.RetentionStrategy$Demand check
            Launching computer LB3D-Wxpp64sp2 as it has been in demand for 1 min 34 sec
            Sep 05, 2013 7:55:24 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog
            [LB3D-Wxpp32sp3] Starting Virtual Machine...
            Sep 05, 2013 7:55:24 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog
            [LB3D-Wxpp64sp2] Starting Virtual Machine...
            Sep 05, 2013 7:55:24 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog
            [LB3D-Wxpp32sp3] Powering on VM
            Sep 05, 2013 7:55:27 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog
            [LB3D-Wxpp64sp2] Powering on VM
            Sep 05, 2013 7:55:55 AM INFO hudson.model.AsyncPeriodicWork$1 run
            Started PeriodicBackup
            Sep 05, 2013 7:55:55 AM INFO hudson.model.AsyncPeriodicWork$1 run
            Finished PeriodicBackup. 0 ms
            Sep 05, 2013 7:56:35 AM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
            Accepted connection #1 from /172.28.63.75:1034
            Sep 05, 2013 7:56:54 AM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
            Accepted connection #2 from /172.28.63.95:1047
            Sep 05, 2013 7:56:55 AM INFO hudson.model.AsyncPeriodicWork$1 run
            Started PeriodicBackup
            Sep 05, 2013 7:56:55 AM INFO hudson.model.AsyncPeriodicWork$1 run
            Finished PeriodicBackup. 0 ms
            Sep 05, 2013 7:57:30 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor
            Failed to monitor LB3D-Wxpp64sp2 for Free Swap Space
            java.util.concurrent.TimeoutException
            at hudson.remoting.Request$1.get(Request.java:275)
            at hudson.remoting.Request$1.get(Request.java:210)
            at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
            at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97)
            at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282)

            As you can see, two slaves are powered ON and successfully connected. But after few seconds, the second VM LB3D-Wxpp64sp2 experiences a problem with Free Swap Space monitoring and from now on, nothing else happens on that machine. The last line in slave console shows Building remotely on LB3D-Wxpp64sp2 in workspace etc... and the spinning wheel. The other one successfully updates SVN and GIT and continues with building.

            Could this Free Swap Space monitoring error really be a root of the problem? Is there a way to disable this monitoring for good? And yes, as I mentioned in my previous comment, I tried to disable the monitoring in Jenkins Node configuration section but apparently, the monitoring is still active on a problematic VM?

            Show
            odklizec Pavel Kudrys added a comment - Hi again. One more piece to the puzzle. This is what I saw in log while attempting to run a Jenkins job... Sep 05, 2013 7:55:24 AM INFO hudson.slaves.RetentionStrategy$Demand check Launching computer LB3D-Wxpp32sp3 as it has been in demand for 1 min 34 sec Sep 05, 2013 7:55:24 AM INFO hudson.slaves.RetentionStrategy$Demand check Launching computer LB3D-Wxpp64sp2 as it has been in demand for 1 min 34 sec Sep 05, 2013 7:55:24 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog [LB3D-Wxpp32sp3] Starting Virtual Machine... Sep 05, 2013 7:55:24 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog [LB3D-Wxpp64sp2] Starting Virtual Machine... Sep 05, 2013 7:55:24 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog [LB3D-Wxpp32sp3] Powering on VM Sep 05, 2013 7:55:27 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog [LB3D-Wxpp64sp2] Powering on VM Sep 05, 2013 7:55:55 AM INFO hudson.model.AsyncPeriodicWork$1 run Started PeriodicBackup Sep 05, 2013 7:55:55 AM INFO hudson.model.AsyncPeriodicWork$1 run Finished PeriodicBackup. 0 ms Sep 05, 2013 7:56:35 AM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run Accepted connection #1 from /172.28.63.75:1034 Sep 05, 2013 7:56:54 AM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run Accepted connection #2 from /172.28.63.95:1047 Sep 05, 2013 7:56:55 AM INFO hudson.model.AsyncPeriodicWork$1 run Started PeriodicBackup Sep 05, 2013 7:56:55 AM INFO hudson.model.AsyncPeriodicWork$1 run Finished PeriodicBackup. 0 ms Sep 05, 2013 7:57:30 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor Failed to monitor LB3D-Wxpp64sp2 for Free Swap Space java.util.concurrent.TimeoutException at hudson.remoting.Request$1.get(Request.java:275) at hudson.remoting.Request$1.get(Request.java:210) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282) As you can see, two slaves are powered ON and successfully connected. But after few seconds, the second VM LB3D-Wxpp64sp2 experiences a problem with Free Swap Space monitoring and from now on, nothing else happens on that machine. The last line in slave console shows Building remotely on LB3D-Wxpp64sp2 in workspace etc... and the spinning wheel. The other one successfully updates SVN and GIT and continues with building. Could this Free Swap Space monitoring error really be a root of the problem? Is there a way to disable this monitoring for good? And yes, as I mentioned in my previous comment, I tried to disable the monitoring in Jenkins Node configuration section but apparently, the monitoring is still active on a problematic VM?
            Hide
            odklizec Pavel Kudrys added a comment - - edited

            OK, I'm now even more convinced that the free swap space monitoring has something to do with my problem. During today's night build, the error message appeared on both slaves and sure enough, building on both machines got stuck on "Building remotely on slave-name..." and spinning wheel.

            Sep 06, 2013 1:28:15 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor

            Failed to monitor LB3D-Wxpp64sp2 for Free Swap Space
            java.util.concurrent.TimeoutException
            at hudson.remoting.Request$1.get(Request.java:275)
            at hudson.remoting.Request$1.get(Request.java:210)
            at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
            at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97)
            at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282)

            Sep 06, 2013 2:08:39 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor

            Failed to monitor LB3D-Wxpp32sp3 for Free Disk Space
            java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
            at hudson.remoting.Request$1.get(Request.java:278)
            at hudson.remoting.Request$1.get(Request.java:210)
            at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
            at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97)
            at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282)
            Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
            at hudson.remoting.Request.abort(Request.java:299)
            at hudson.remoting.Channel.terminate(Channel.java:774)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
            Caused by: java.net.SocketException: Connection reset
            at java.net.SocketInputStream.read(Unknown Source)
            at java.net.SocketInputStream.read(Unknown Source)
            at java.io.BufferedInputStream.fill(Unknown Source)
            at java.io.BufferedInputStream.read(Unknown Source)
            at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
            at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
            at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
            at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
            at java.io.ObjectInputStream.readObject0(Unknown Source)
            at java.io.ObjectInputStream.readObject(Unknown Source)
            at hudson.remoting.Command.readFrom(Command.java:92)
            at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

            Any idea how to kill this free swap space monitoring for good?

            Show
            odklizec Pavel Kudrys added a comment - - edited OK, I'm now even more convinced that the free swap space monitoring has something to do with my problem. During today's night build, the error message appeared on both slaves and sure enough, building on both machines got stuck on "Building remotely on slave-name..." and spinning wheel. Sep 06, 2013 1:28:15 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor Failed to monitor LB3D-Wxpp64sp2 for Free Swap Space java.util.concurrent.TimeoutException at hudson.remoting.Request$1.get(Request.java:275) at hudson.remoting.Request$1.get(Request.java:210) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282) Sep 06, 2013 2:08:39 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor Failed to monitor LB3D-Wxpp32sp3 for Free Disk Space java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset at hudson.remoting.Request$1.get(Request.java:278) at hudson.remoting.Request$1.get(Request.java:210) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282) Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset at hudson.remoting.Request.abort(Request.java:299) at hudson.remoting.Channel.terminate(Channel.java:774) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69) Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77) at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) Any idea how to kill this free swap space monitoring for good?
            Hide
            odklizec Pavel Kudrys added a comment -

            OK. One more addition to the problem. I reconfigured both slaves to keep them online as much as possible and so far (5 days), no problem with job start.

            Unfortuantely, from the long perspective, this is not a very good solution. Slaves which are still online eat the server memory even if there is no need for them to stay online. Therefore, we would like to have them online only on demand and then shut them off after the build and test is finished. But this is currently impossible because of problem with randomly stuck slaves during their start (most probably caused by "free swap space monitoring). Anyone please?

            Show
            odklizec Pavel Kudrys added a comment - OK. One more addition to the problem. I reconfigured both slaves to keep them online as much as possible and so far (5 days), no problem with job start. Unfortuantely, from the long perspective, this is not a very good solution. Slaves which are still online eat the server memory even if there is no need for them to stay online. Therefore, we would like to have them online only on demand and then shut them off after the build and test is finished. But this is currently impossible because of problem with randomly stuck slaves during their start (most probably caused by "free swap space monitoring). Anyone please?
            Hide
            odklizec Pavel Kudrys added a comment -

            Anyone please? The workaround with always enabled slaves works quite OK, but I hate to have the slaves permanently running. It's relative OK in case of just two slaves, but I'm planning to add few more slaves and then it would become a real problem.

            Show
            odklizec Pavel Kudrys added a comment - Anyone please? The workaround with always enabled slaves works quite OK, but I hate to have the slaves permanently running. It's relative OK in case of just two slaves, but I'm planning to add few more slaves and then it would become a real problem.
            Hide
            odklizec Pavel Kudrys added a comment -

            One month later, once again updated Jenkins and all plugins, but nothing has changed. Jobs still randomly stuck with "building remotely on slave-name" message. Anyone please? Am I alone with such kind of issue?

            Show
            odklizec Pavel Kudrys added a comment - One month later, once again updated Jenkins and all plugins, but nothing has changed. Jobs still randomly stuck with "building remotely on slave-name" message. Anyone please? Am I alone with such kind of issue?
            Hide
            odklizec Pavel Kudrys added a comment -

            It happened again today Any idea? Anyone please...

            This is what I see in jenkins-slave.err.log:

            Oct 31, 2013 7:41:16 AM hudson.remoting.jnlp.Main$CuiListener <init>
            INFO: Jenkins agent is running in headless mode.
            Oct 31, 2013 7:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
            INFO: Locating server among http://tcskjira:8095/
            Oct 31, 2013 7:41:17 AM hudson.remoting.jnlp.Main$CuiListener status
            INFO: Connecting to tcskjira:1695
            Oct 31, 2013 7:41:17 AM hudson.remoting.jnlp.Main$CuiListener status
            INFO: Handshaking
            Oct 31, 2013 7:41:17 AM hudson.remoting.jnlp.Main$CuiListener status
            INFO: Connected

            This is what I see in jenkins-slave.wrapper.log:

            2013-10-31 07:41:09 - Starting C:\Program Files\Java\jre7\bin\java.exe -Xrs -jar "C:\Jenkins\slave.jar" -jnlpUrl http://tcskjira:8095/computer/LB3D-Wxpp64sp2/slave-agent.jnlp -secret 1ba0fb4096bc76c5ef1332c6368cd286a76074134a7091cd7603e93ec938b060
            2013-10-31 07:41:09 - Started 1576

            This is what I see in Jenkins Log:

            Oct 31, 2013 7:40:34 AM INFO hudson.slaves.RetentionStrategy$Demand check

            Launching computer LB3D-Wxpp64sp2 as it has been in demand for 39 sec

            Oct 31, 2013 7:40:34 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog

            [LB3D-Wxpp64sp2] Starting Virtual Machine...

            Oct 31, 2013 7:40:35 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog

            [LB3D-Wxpp64sp2] Powering on VM

            Oct 31, 2013 7:40:51 AM INFO hudson.model.AsyncPeriodicWork$1 run

            Started PeriodicBackup

            Oct 31, 2013 7:40:51 AM INFO hudson.model.AsyncPeriodicWork$1 run

            Finished PeriodicBackup. 0 ms

            Oct 31, 2013 7:41:16 AM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run

            Accepted connection #40 from /172.28.63.95:1034

            Oct 31, 2013 7:41:23 AM WARNING hudson.node_monitors.ResponseTimeMonitor$1 monitor

            Making CAAV5R19x64 offline because it’s not responding

            Oct 31, 2013 7:41:23 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog

            [CAAV5R19x64] Already disconnecting on a separate thread

            Oct 31, 2013 7:41:51 AM INFO hudson.model.AsyncPeriodicWork$1 run

            Started PeriodicBackup

            Oct 31, 2013 7:41:51 AM INFO hudson.model.AsyncPeriodicWork$1 run

            Finished PeriodicBackup. 0 ms

            Oct 31, 2013 7:41:52 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor

            Failed to monitor LB3D-Wxpp64sp2 for Free Swap Space
            java.util.concurrent.TimeoutException
            at hudson.remoting.Request$1.get(Request.java:275)
            at hudson.remoting.Request$1.get(Request.java:210)
            at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
            at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97)
            at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282)

            Slave Log says this:

            JNLP agent connected from /172.28.63.95
            <===[JENKINS REMOTING CAPACITY]===>Slave.jar version: 2.32
            This is a Windows slave
            Slave successfully connected and online
            followed by spinning wheel...

            Slave Thread Dump:

            Channel reader thread: channel

            "Channel reader thread: channel" Id=11 Group=main RUNNABLE (in native)
            at java.net.SocketInputStream.socketRead0(Native Method)
            at java.net.SocketInputStream.read(Unknown Source)
            at java.net.SocketInputStream.read(Unknown Source)
            at java.io.BufferedInputStream.fill(Unknown Source)
            at java.io.BufferedInputStream.read(Unknown Source)

            • locked java.io.BufferedInputStream@1c2ba649
              at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
              at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
              at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
              at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
              at java.io.ObjectInputStream.readObject0(Unknown Source)
              at java.io.ObjectInputStream.readObject(Unknown Source)
              at hudson.remoting.Command.readFrom(Command.java:92)
              at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
              at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

            main

            "main" Id=1 Group=main WAITING on hudson.remoting.Engine@7a0136b0
            at java.lang.Object.wait(Native Method)

            • waiting on hudson.remoting.Engine@7a0136b0
              at java.lang.Thread.join(Unknown Source)
              at java.lang.Thread.join(Unknown Source)
              at hudson.remoting.jnlp.Main.main(Main.java:137)
              at hudson.remoting.jnlp.Main._main(Main.java:130)
              at hudson.remoting.Launcher.run(Launcher.java:223)
              at hudson.remoting.Launcher.main(Launcher.java:192)

            Ping thread for channel hudson.remoting.Channel@28509953:channel

            "Ping thread for channel hudson.remoting.Channel@28509953:channel" Id=13 Group=main TIMED_WAITING
            at java.lang.Thread.sleep(Native Method)
            at hudson.remoting.PingThread.run(PingThread.java:86)

            pool-1-thread-17

            "pool-1-thread-17" Id=29 Group=main RUNNABLE
            at sun.management.ThreadImpl.dumpThreads0(Native Method)
            at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
            at hudson.Functions.getThreadInfos(Functions.java:1131)
            at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96)
            at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
            at hudson.remoting.UserRequest.perform(UserRequest.java:118)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:326)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
            at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at hudson.remoting.Engine$1$1.run(Engine.java:63)
            at java.lang.Thread.run(Unknown Source)

            Number of locked synchronizers = 1

            • java.util.concurrent.ThreadPoolExecutor$Worker@493df066

            pool-1-thread-2

            "pool-1-thread-2" Id=14 Group=main RUNNABLE
            at com.sun.jna.Native.initIDs(Native Method)
            at com.sun.jna.Native.<clinit>(Native.java:122)
            at org.tmatesoft.svn.core.internal.util.jna.JNALibraryLoader.<clinit>(JNALibraryLoader.java:50)
            at org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:93)
            at org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196)
            at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1869)
            at org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57)
            at hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:975)
            at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:952)
            at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:885)
            at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:872)
            at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2420)
            at hudson.remoting.UserRequest.perform(UserRequest.java:118)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:326)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
            at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at hudson.remoting.Engine$1$1.run(Engine.java:63)
            at java.lang.Thread.run(Unknown Source)

            Number of locked synchronizers = 1

            • java.util.concurrent.ThreadPoolExecutor$Worker@5133066c

            pool-1-thread-5

            "pool-1-thread-5" Id=17 Group=main RUNNABLE
            at com.sun.jna.Structure.<clinit>(Structure.java:134)
            at org.jvnet.hudson.Windows.monitor(Windows.java:40)
            at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:110)
            at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:100)
            at hudson.remoting.UserRequest.perform(UserRequest.java:118)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:326)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
            at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at hudson.remoting.Engine$1$1.run(Engine.java:63)
            at java.lang.Thread.run(Unknown Source)

            Number of locked synchronizers = 1

            • java.util.concurrent.ThreadPoolExecutor$Worker@786dee4c

            Thread-1

            "Thread-1" Id=10 Group=main WAITING on hudson.remoting.Channel@28509953
            at java.lang.Object.wait(Native Method)

            • waiting on hudson.remoting.Channel@28509953
              at java.lang.Object.wait(Object.java:503)
              at hudson.remoting.Channel.join(Channel.java:842)
              at hudson.remoting.Engine.run(Engine.java:271)

            Attach Listener

            "Attach Listener" Id=4 Group=system RUNNABLE

            Finalizer

            "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@75b21111
            at java.lang.Object.wait(Native Method)

            • waiting on java.lang.ref.ReferenceQueue$Lock@75b21111
              at java.lang.ref.ReferenceQueue.remove(Unknown Source)
              at java.lang.ref.ReferenceQueue.remove(Unknown Source)
              at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

            Reference Handler

            "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@7d7eaf74
            at java.lang.Object.wait(Native Method)

            • waiting on java.lang.ref.Reference$Lock@7d7eaf74
              at java.lang.Object.wait(Object.java:503)
              at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)

            System Thread Dump:

            Thread Dump
            AWT-Windows

            "AWT-Windows" Id=614 Group=main RUNNABLE (in native)
            at sun.awt.windows.WToolkit.eventLoop(Native Method)
            at sun.awt.windows.WToolkit.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

            Channel reader thread: LB3D-Wxpp32sp3

            "Channel reader thread: LB3D-Wxpp32sp3" Id=12151 Group=main RUNNABLE
            at java.net.SocketInputStream.socketRead0(Native Method)
            at java.net.SocketInputStream.read(Unknown Source)
            at java.net.SocketInputStream.read(Unknown Source)
            at java.io.BufferedInputStream.fill(Unknown Source)
            at java.io.BufferedInputStream.read(Unknown Source)

            • locked java.io.BufferedInputStream@7a7ec776
              at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
              at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
              at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
              at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
              at java.io.ObjectInputStream.readObject0(Unknown Source)
              at java.io.ObjectInputStream.readObject(Unknown Source)
              at hudson.remoting.Command.readFrom(Command.java:92)
              at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
              at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

            Channel reader thread: LB3D-Wxpp64sp2

            "Channel reader thread: LB3D-Wxpp64sp2" Id=12178 Group=main RUNNABLE
            at java.net.SocketInputStream.socketRead0(Native Method)
            at java.net.SocketInputStream.read(Unknown Source)
            at java.net.SocketInputStream.read(Unknown Source)
            at java.io.BufferedInputStream.fill(Unknown Source)
            at java.io.BufferedInputStream.read(Unknown Source)

            • locked java.io.BufferedInputStream@764058b0
              at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
              at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
              at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
              at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
              at java.io.ObjectInputStream.readObject0(Unknown Source)
              at java.io.ObjectInputStream.readObject(Unknown Source)
              at hudson.remoting.Command.readFrom(Command.java:92)
              at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
              at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

            Channel reader thread: Slave-TestProcessBuild

            "Channel reader thread: Slave-TestProcessBuild" Id=6006 Group=main RUNNABLE
            at java.net.SocketInputStream.socketRead0(Native Method)
            at java.net.SocketInputStream.read(Unknown Source)
            at java.net.SocketInputStream.read(Unknown Source)
            at java.io.BufferedInputStream.fill(Unknown Source)
            at java.io.BufferedInputStream.read(Unknown Source)

            • locked java.io.BufferedInputStream@3563374a
              at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
              at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
              at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
              at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
              at java.io.ObjectInputStream.readObject0(Unknown Source)
              at java.io.ObjectInputStream.readObject(Unknown Source)
              at hudson.remoting.Command.readFrom(Command.java:92)
              at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
              at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

            Channel reader thread: Slave-TestProcessTest

            "Channel reader thread: Slave-TestProcessTest" Id=6121 Group=main RUNNABLE
            at java.net.SocketInputStream.socketRead0(Native Method)
            at java.net.SocketInputStream.read(Unknown Source)
            at java.net.SocketInputStream.read(Unknown Source)
            at java.io.BufferedInputStream.fill(Unknown Source)
            at java.io.BufferedInputStream.read(Unknown Source)

            • locked java.io.BufferedInputStream@374b5f62
              at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
              at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
              at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
              at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
              at java.io.ObjectInputStream.readObject0(Unknown Source)
              at java.io.ObjectInputStream.readObject(Unknown Source)
              at hudson.remoting.Command.readFrom(Command.java:92)
              at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
              at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

            DestroyJavaVM

            "DestroyJavaVM" Id=19 Group=main RUNNABLE

            Executor #0 for LB3D-Wxpp32sp3 : executing ipprotection_32_64 #171

            "Executor #0 for LB3D-Wxpp32sp3 : executing ipprotection_32_64 #171" Id=10955 Group=main TIMED_WAITING
            at java.lang.Thread.sleep(Native Method)
            at hudson.matrix.DefaultMatrixExecutionStrategyImpl.waitForCompletion(DefaultMatrixExecutionStrategyImpl.java:290)
            at hudson.matrix.DefaultMatrixExecutionStrategyImpl.run(DefaultMatrixExecutionStrategyImpl.java:141)
            at hudson.matrix.MatrixBuild$MatrixBuildExecution.doRun(MatrixBuild.java:355)
            at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:562)
            at hudson.model.Run.execute(Run.java:1665)
            at hudson.matrix.MatrixBuild.run(MatrixBuild.java:304)
            at hudson.model.ResourceController.execute(ResourceController.java:88)
            at hudson.model.Executor.run(Executor.java:230)

            Executor #0 for LB3D-Wxpp64sp2 : executing ipprotection_32_64 » LB3D-Wxpp64sp2 #171 / waiting for hudson.remoting.Channel@59f73f53:LB3D-Wxpp64sp2

            "Executor #0 for LB3D-Wxpp64sp2 : executing ipprotection_32_64 » LB3D-Wxpp64sp2 #171 / waiting for hudson.remoting.Channel@59f73f53:LB3D-Wxpp64sp2" Id=11106 Group=main TIMED_WAITING on hudson.remoting.UserRequest@3690a4a8
            at java.lang.Object.wait(Native Method)

            • waiting on hudson.remoting.UserRequest@3690a4a8
              at hudson.remoting.Request.call(Request.java:146)
              at hudson.remoting.Channel.call(Channel.java:714)
              at hudson.FilePath.act(FilePath.java:901)
              at hudson.FilePath.act(FilePath.java:885)
              at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:848)
              at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:786)
              at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:117)
              at hudson.model.AbstractProject.checkout(AbstractProject.java:1411)
              at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:652)
              at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
              at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:557)
              at hudson.model.Run.execute(Run.java:1665)
              at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
              at hudson.model.ResourceController.execute(ResourceController.java:88)
              at hudson.model.Executor.run(Executor.java:230)

            Handling GET /threadDump : RequestHandlerThread595

            "Handling GET /threadDump : RequestHandlerThread595" Id=12350 Group=main TIMED_WAITING on java.util.concurrent.FutureTask$Sync@4deadcb6
            at sun.misc.Unsafe.park(Native Method)

            • waiting on java.util.concurrent.FutureTask$Sync@4deadcb6
              at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(Unknown Source)
              at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
              at java.util.concurrent.FutureTask.get(Unknown Source)
              at hudson.remoting.LocalChannel$2.get(LocalChannel.java:81)
              at jenkins.model.Jenkins.getAllThreadDumps(Jenkins.java:2919)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
              at java.lang.reflect.Method.invoke(Unknown Source)
              at org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258)
              at org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104)
              at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
              at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
              at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
              at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
              at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
              at org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61)
              at org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsIterator(ExpressionSupport.java:94)
              at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:89)
              at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
              at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
              at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
              at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
              at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
              at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
              at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
              at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
              at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
              at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
              at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
              at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
              at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:727)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:841)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:630)
              at org.kohsuke.stapler.Stapler.service(Stapler.java:224)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
              at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
              at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
              at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:208)
              at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:181)
              at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86)
              at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84)
              at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
              at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
              at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
              at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
              at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
              at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
              at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
              at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
              at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
              at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:64)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
              at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
              at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
              at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
              at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
              at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
              at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
              at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
              at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
              at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
              at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
              at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
              at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
              at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
              at org.eclipse.jetty.server.Server.handle(Server.java:370)
              at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
              at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
              at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
              at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
              at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
              at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
              at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
              at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
              at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)

            Number of locked synchronizers = 1

            • java.util.concurrent.ThreadPoolExecutor$Worker@620c96ba

            HashSessionScavenger-0

            "HashSessionScavenger-0" Id=11 Group=main TIMED_WAITING on java.util.TaskQueue@4a7cb042
            at java.lang.Object.wait(Native Method)

            • waiting on java.util.TaskQueue@4a7cb042
              at java.util.TimerThread.mainLoop(Unknown Source)
              at java.util.TimerThread.run(Unknown Source)

            javamelody

            "javamelody" Id=27 Group=main TIMED_WAITING on java.util.TaskQueue@1b9a5f49
            at java.lang.Object.wait(Native Method)

            • waiting on java.util.TaskQueue@1b9a5f49
              at java.util.TimerThread.mainLoop(Unknown Source)
              at java.util.TimerThread.run(Unknown Source)

            Jenkins cron thread

            "Jenkins cron thread" Id=14 Group=main TIMED_WAITING on java.util.TaskQueue@258e37c9
            at java.lang.Object.wait(Native Method)

            • waiting on java.util.TaskQueue@258e37c9
              at java.util.TimerThread.mainLoop(Unknown Source)
              at java.util.TimerThread.run(Unknown Source)

            Jenkins UDP 33848 monitoring thread

            "Jenkins UDP 33848 monitoring thread" Id=31 Group=main RUNNABLE (in native)
            at java.net.TwoStacksPlainDatagramSocketImpl.receive0(Native Method)

            • locked java.net.TwoStacksPlainDatagramSocketImpl@1f036f93
              at java.net.TwoStacksPlainDatagramSocketImpl.receive(Unknown Source)
            • locked java.net.TwoStacksPlainDatagramSocketImpl@1f036f93
              at java.net.DatagramSocket.receive(Unknown Source)
            • locked java.net.DatagramPacket@7279ad5b
            • locked java.net.MulticastSocket@12646c14
              at hudson.UDPBroadcastThread.run(UDPBroadcastThread.java:82)

            Jenkins-Remoting-Thread-1218

            "Jenkins-Remoting-Thread-1218" Id=12433 Group=main RUNNABLE
            at sun.management.ThreadImpl.dumpThreads0(Native Method)
            at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
            at hudson.Functions.getThreadInfos(Functions.java:1131)
            at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96)
            at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
            at hudson.remoting.LocalChannel$1.call(LocalChannel.java:52)
            at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

            Number of locked synchronizers = 1

            • java.util.concurrent.ThreadPoolExecutor$Worker@1904332a

            Jenkins-Remoting-Thread-1219

            "Jenkins-Remoting-Thread-1219" Id=12434 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@2baacf98
            at sun.misc.Unsafe.park(Native Method)

            • waiting on java.util.concurrent.SynchronousQueue$TransferStack@2baacf98
              at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
              at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)

            Jenkins-Remoting-Thread-1220

            "Jenkins-Remoting-Thread-1220" Id=12435 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@2baacf98
            at sun.misc.Unsafe.park(Native Method)

            • waiting on java.util.concurrent.SynchronousQueue$TransferStack@2baacf98
              at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
              at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)

            JmDNS(tcskjira.local.).State.Timer

            "JmDNS(tcskjira.local.).State.Timer" Id=48 Group=main TIMED_WAITING on java.util.TaskQueue@587c9b22
            at java.lang.Object.wait(Native Method)

            • waiting on java.util.TaskQueue@587c9b22
              at java.util.TimerThread.mainLoop(Unknown Source)
              at java.util.TimerThread.run(Unknown Source)

            JmDNS(tcskjira.local.).Timer

            "JmDNS(tcskjira.local.).Timer" Id=47 Group=main TIMED_WAITING on java.util.TaskQueue@5cad0290
            at java.lang.Object.wait(Native Method)

            • waiting on java.util.TaskQueue@5cad0290
              at java.util.TimerThread.mainLoop(Unknown Source)
              at java.util.TimerThread.run(Unknown Source)

            jnr.ffi.util.ref.internal.Finalizer

            "jnr.ffi.util.ref.internal.Finalizer" Id=5472 Group=main WAITING on java.lang.ref.ReferenceQueue$Lock@4c2f6a8
            at java.lang.Object.wait(Native Method)

            • waiting on java.lang.ref.ReferenceQueue$Lock@4c2f6a8
              at java.lang.ref.ReferenceQueue.remove(Unknown Source)
              at java.lang.ref.ReferenceQueue.remove(Unknown Source)
              at jnr.ffi.util.ref.internal.Finalizer.run(Finalizer.java:128)

            LauncherControlThread[ControlPort=-1]

            "LauncherControlThread[ControlPort=-1]" Id=17 Group=main TIMED_WAITING
            at java.lang.Thread.sleep(Native Method)
            at winstone.Launcher.run(Launcher.java:245)
            at java.lang.Thread.run(Unknown Source)

            NioSocketAcceptor-1

            "NioSocketAcceptor-1" Id=29 Group=main RUNNABLE (in native)
            at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
            at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
            at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
            at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
            at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)

            • locked sun.nio.ch.Util$2@5feec26b
            • locked java.util.Collections$UnmodifiableSet@29e1ac62
            • locked sun.nio.ch.WindowsSelectorImpl@4defca63
              at sun.nio.ch.SelectorImpl.select(Unknown Source)
              at sun.nio.ch.SelectorImpl.select(Unknown Source)
              at org.apache.mina.transport.socket.nio.NioSocketAcceptor.select(NioSocketAcceptor.java:238)
              at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:432)
              at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)

            Number of locked synchronizers = 1

            • java.util.concurrent.ThreadPoolExecutor$Worker@29090dda

            Ping thread for channel hudson.remoting.Channel@1675652c:Slave-TestProcessTest

            "Ping thread for channel hudson.remoting.Channel@1675652c:Slave-TestProcessTest" Id=6122 Group=main TIMED_WAITING
            at java.lang.Thread.sleep(Native Method)
            at hudson.remoting.PingThread.run(PingThread.java:86)

            Ping thread for channel hudson.remoting.Channel@50923d0b:Slave-TestProcessBuild

            "Ping thread for channel hudson.remoting.Channel@50923d0b:Slave-TestProcessBuild" Id=6007 Group=main TIMED_WAITING
            at java.lang.Thread.sleep(Native Method)
            at hudson.remoting.PingThread.run(PingThread.java:86)

            Ping thread for channel hudson.remoting.Channel@59f73f53:LB3D-Wxpp64sp2

            "Ping thread for channel hudson.remoting.Channel@59f73f53:LB3D-Wxpp64sp2" Id=12179 Group=main TIMED_WAITING
            at java.lang.Thread.sleep(Native Method)
            at hudson.remoting.PingThread.run(PingThread.java:86)

            Ping thread for channel hudson.remoting.Channel@5ade8f7a:LB3D-Wxpp32sp3

            "Ping thread for channel hudson.remoting.Channel@5ade8f7a:LB3D-Wxpp32sp3" Id=12152 Group=main TIMED_WAITING
            at java.lang.Thread.sleep(Native Method)
            at hudson.remoting.PingThread.run(PingThread.java:86)

            pool-1-svnkit-thread-1

            "pool-1-svnkit-thread-1" Id=411 Group=main WAITING on java.util.concurrent.SynchronousQueue$TransferStack@1bcf9209
            at sun.misc.Unsafe.park(Native Method)

            • waiting on java.util.concurrent.SynchronousQueue$TransferStack@1bcf9209
              at java.util.concurrent.locks.LockSupport.park(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
              at java.util.concurrent.SynchronousQueue.take(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)

            pool-1-svnkit-thread-2

            "pool-1-svnkit-thread-2" Id=2322 Group=main WAITING on java.util.concurrent.SynchronousQueue$TransferStack@1bcf9209
            at sun.misc.Unsafe.park(Native Method)

            • waiting on java.util.concurrent.SynchronousQueue$TransferStack@1bcf9209
              at java.util.concurrent.locks.LockSupport.park(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
              at java.util.concurrent.SynchronousQueue.take(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)

            pool-4-thread-1272

            "pool-4-thread-1272" Id=12398 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9
            at sun.misc.Unsafe.park(Native Method)

            • waiting on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9
              at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
              at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)

            pool-4-thread-1273

            "pool-4-thread-1273" Id=12399 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9
            at sun.misc.Unsafe.park(Native Method)

            • waiting on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9
              at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
              at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)

            pool-4-thread-1274

            "pool-4-thread-1274" Id=12417 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9
            at sun.misc.Unsafe.park(Native Method)

            • waiting on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9
              at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
              at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)

            pool-4-thread-1275

            "pool-4-thread-1275" Id=12418 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9
            at sun.misc.Unsafe.park(Native Method)

            • waiting on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9
              at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
              at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)

            pool-9-thread-1

            "pool-9-thread-1" Id=44 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2ba02595
            at sun.misc.Unsafe.park(Native Method)

            • waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2ba02595
              at java.util.concurrent.locks.LockSupport.park(Unknown Source)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)

            RequestHandlerThread1 Selector0

            "RequestHandlerThread1 Selector0" Id=15 Group=main RUNNABLE (in native)
            at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
            at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
            at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
            at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
            at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)

            • locked sun.nio.ch.Util$2@2fea3452
            • locked java.util.Collections$UnmodifiableSet@e2f4d5c
            • locked sun.nio.ch.WindowsSelectorImpl@7ed452e4
              at sun.nio.ch.SelectorImpl.select(Unknown Source)
              at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
              at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
              at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)

            Number of locked synchronizers = 1

            • java.util.concurrent.ThreadPoolExecutor$Worker@4da2d209

            RequestHandlerThread2 Acceptor0 SelectChannelConnector@0.0.0.0:8095

            "RequestHandlerThread2 Acceptor0 SelectChannelConnector@0.0.0.0:8095" Id=16 Group=main RUNNABLE (in native)
            at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
            at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)

            • locked java.lang.Object@33e5bfc7
              at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
              at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
              at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)

            Number of locked synchronizers = 1

            • java.util.concurrent.ThreadPoolExecutor$Worker@5db00667

            RequestHandlerThread603

            "RequestHandlerThread603" Id=12388 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d
            at sun.misc.Unsafe.park(Native Method)

            • waiting on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d
              at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
              at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)

            RequestHandlerThread605

            "RequestHandlerThread605" Id=12391 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d
            at sun.misc.Unsafe.park(Native Method)

            • waiting on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d
              at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
              at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)

            RequestHandlerThread607

            "RequestHandlerThread607" Id=12427 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d
            at sun.misc.Unsafe.park(Native Method)

            • waiting on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d
              at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
              at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)

            RequestHandlerThread608

            "RequestHandlerThread608" Id=12431 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d
            at sun.misc.Unsafe.park(Native Method)

            • waiting on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d
              at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
              at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)

            SocketListener(tcskjira.local.)

            "SocketListener(tcskjira.local.)" Id=46 Group=main RUNNABLE (in native)
            at java.net.TwoStacksPlainDatagramSocketImpl.receive0(Native Method)

            • locked java.net.TwoStacksPlainDatagramSocketImpl@62836b78
              at java.net.TwoStacksPlainDatagramSocketImpl.receive(Unknown Source)
            • locked java.net.TwoStacksPlainDatagramSocketImpl@62836b78
              at java.net.DatagramSocket.receive(Unknown Source)
            • locked java.net.DatagramPacket@6756aa7b
            • locked java.net.MulticastSocket@6ad2a63d
              at javax.jmdns.impl.SocketListener.run(SocketListener.java:41)

            TCP slave agent listener port=0

            "TCP slave agent listener port=0" Id=30 Group=main RUNNABLE (in native)
            at java.net.TwoStacksPlainSocketImpl.socketAccept(Native Method)
            at java.net.AbstractPlainSocketImpl.accept(Unknown Source)
            at java.net.PlainSocketImpl.accept(Unknown Source)

            • locked java.net.SocksSocketImpl@7ef39e7a
              at java.net.ServerSocket.implAccept(Unknown Source)
              at java.net.ServerSocket.accept(Unknown Source)
              at hudson.TcpSlaveAgentListener.run(TcpSlaveAgentListener.java:93)

            ViewJob reload thread

            "ViewJob reload thread" Id=28 Group=main TIMED_WAITING on java.util.LinkedHashSet@39d0518e
            at java.lang.Object.wait(Native Method)

            • waiting on java.util.LinkedHashSet@39d0518e
              at hudson.model.ViewJob$ReloadThread.getNext(ViewJob.java:176)
              at hudson.model.ViewJob$ReloadThread.run(ViewJob.java:193)

            Waiting to acquire c:\Jenkins\workspace\ipprotection_32_64 : SCM polling for hudson.matrix.MatrixProject@3b277576[ipprotection_32_64]

            "Waiting to acquire c:\Jenkins\workspace\ipprotection_32_64 : SCM polling for hudson.matrix.MatrixProject@3b277576[ipprotection_32_64]" Id=12065 Group=main WAITING on hudson.slaves.WorkspaceList@195952ce
            at java.lang.Object.wait(Native Method)

            • waiting on hudson.slaves.WorkspaceList@195952ce
              at java.lang.Object.wait(Object.java:503)
              at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:232)
              at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:211)
              at hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1580)
              at hudson.model.AbstractProject._poll(AbstractProject.java:1558)
              at hudson.model.AbstractProject.poll(AbstractProject.java:1490)
              at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:439)
              at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:468)
              at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
              at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
              at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
              at java.util.concurrent.FutureTask.run(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)

            Number of locked synchronizers = 1

            • java.util.concurrent.ThreadPoolExecutor$Worker@3435c217

            Attach Listener

            "Attach Listener" Id=5 Group=system RUNNABLE

            Finalizer

            "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@4017e342
            at java.lang.Object.wait(Native Method)

            • waiting on java.lang.ref.ReferenceQueue$Lock@4017e342
              at java.lang.ref.ReferenceQueue.remove(Unknown Source)
              at java.lang.ref.ReferenceQueue.remove(Unknown Source)
              at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

            Java2D Disposer

            "Java2D Disposer" Id=12 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@1bf08062
            at java.lang.Object.wait(Native Method)

            • waiting on java.lang.ref.ReferenceQueue$Lock@1bf08062
              at java.lang.ref.ReferenceQueue.remove(Unknown Source)
              at java.lang.ref.ReferenceQueue.remove(Unknown Source)
              at sun.java2d.Disposer.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)

            Reference Handler

            "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@604922ed
            at java.lang.Object.wait(Native Method)

            • waiting on java.lang.ref.Reference$Lock@604922ed
              at java.lang.Object.wait(Object.java:503)
              at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)

            Signal Dispatcher

            "Signal Dispatcher" Id=4 Group=system RUNNABLE

            CAAV5R19x64
            N/A

            offline

            LB3D-Wxpp64sp2
            Channel reader thread: channel

            "Channel reader thread: channel" Id=11 Group=main RUNNABLE (in native)
            at java.net.SocketInputStream.socketRead0(Native Method)
            at java.net.SocketInputStream.read(Unknown Source)
            at java.net.SocketInputStream.read(Unknown Source)
            at java.io.BufferedInputStream.fill(Unknown Source)
            at java.io.BufferedInputStream.read(Unknown Source)

            • locked java.io.BufferedInputStream@1c2ba649
              at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
              at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
              at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
              at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
              at java.io.ObjectInputStream.readObject0(Unknown Source)
              at java.io.ObjectInputStream.readObject(Unknown Source)
              at hudson.remoting.Command.readFrom(Command.java:92)
              at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
              at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

            main

            "main" Id=1 Group=main WAITING on hudson.remoting.Engine@7a0136b0
            at java.lang.Object.wait(Native Method)

            • waiting on hudson.remoting.Engine@7a0136b0
              at java.lang.Thread.join(Unknown Source)
              at java.lang.Thread.join(Unknown Source)
              at hudson.remoting.jnlp.Main.main(Main.java:137)
              at hudson.remoting.jnlp.Main._main(Main.java:130)
              at hudson.remoting.Launcher.run(Launcher.java:223)
              at hudson.remoting.Launcher.main(Launcher.java:192)

            Ping thread for channel hudson.remoting.Channel@28509953:channel

            "Ping thread for channel hudson.remoting.Channel@28509953:channel" Id=13 Group=main TIMED_WAITING
            at java.lang.Thread.sleep(Native Method)
            at hudson.remoting.PingThread.run(PingThread.java:86)

            pool-1-thread-18

            "pool-1-thread-18" Id=30 Group=main RUNNABLE
            at sun.management.ThreadImpl.dumpThreads0(Native Method)
            at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
            at hudson.Functions.getThreadInfos(Functions.java:1131)
            at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96)
            at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
            at hudson.remoting.UserRequest.perform(UserRequest.java:118)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:326)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
            at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at hudson.remoting.Engine$1$1.run(Engine.java:63)
            at java.lang.Thread.run(Unknown Source)

            Number of locked synchronizers = 1

            • java.util.concurrent.ThreadPoolExecutor$Worker@59d9c141

            pool-1-thread-2

            "pool-1-thread-2" Id=14 Group=main RUNNABLE
            at com.sun.jna.Native.initIDs(Native Method)
            at com.sun.jna.Native.<clinit>(Native.java:122)
            at org.tmatesoft.svn.core.internal.util.jna.JNALibraryLoader.<clinit>(JNALibraryLoader.java:50)
            at org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:93)
            at org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196)
            at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1869)
            at org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57)
            at hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:975)
            at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:952)
            at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:885)
            at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:872)
            at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2420)
            at hudson.remoting.UserRequest.perform(UserRequest.java:118)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:326)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
            at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at hudson.remoting.Engine$1$1.run(Engine.java:63)
            at java.lang.Thread.run(Unknown Source)

            Number of locked synchronizers = 1

            • java.util.concurrent.ThreadPoolExecutor$Worker@5133066c

            pool-1-thread-5

            "pool-1-thread-5" Id=17 Group=main RUNNABLE
            at com.sun.jna.Structure.<clinit>(Structure.java:134)
            at org.jvnet.hudson.Windows.monitor(Windows.java:40)
            at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:110)
            at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:100)
            at hudson.remoting.UserRequest.perform(UserRequest.java:118)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:326)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
            at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at hudson.remoting.Engine$1$1.run(Engine.java:63)
            at java.lang.Thread.run(Unknown Source)

            Number of locked synchronizers = 1

            • java.util.concurrent.ThreadPoolExecutor$Worker@786dee4c

            Thread-1

            "Thread-1" Id=10 Group=main WAITING on hudson.remoting.Channel@28509953
            at java.lang.Object.wait(Native Method)

            • waiting on hudson.remoting.Channel@28509953
              at java.lang.Object.wait(Object.java:503)
              at hudson.remoting.Channel.join(Channel.java:842)
              at hudson.remoting.Engine.run(Engine.java:271)

            Attach Listener

            "Attach Listener" Id=4 Group=system RUNNABLE

            Finalizer

            "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@75b21111
            at java.lang.Object.wait(Native Method)

            • waiting on java.lang.ref.ReferenceQueue$Lock@75b21111
              at java.lang.ref.ReferenceQueue.remove(Unknown Source)
              at java.lang.ref.ReferenceQueue.remove(Unknown Source)
              at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

            Reference Handler

            "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@7d7eaf74
            at java.lang.Object.wait(Native Method)

            • waiting on java.lang.ref.Reference$Lock@7d7eaf74
              at java.lang.Object.wait(Object.java:503)
              at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)

            Slave-TestProcessBuild
            Channel reader thread: channel

            "Channel reader thread: channel" Id=11 Group=main RUNNABLE (in native)
            at java.net.SocketInputStream.socketRead0(Native Method)
            at java.net.SocketInputStream.read(Unknown Source)
            at java.net.SocketInputStream.read(Unknown Source)
            at java.io.BufferedInputStream.fill(Unknown Source)
            at java.io.BufferedInputStream.read(Unknown Source)

            • locked java.io.BufferedInputStream@74784829
              at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
              at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
              at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
              at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
              at java.io.ObjectInputStream.readObject0(Unknown Source)
              at java.io.ObjectInputStream.readObject(Unknown Source)
              at hudson.remoting.Command.readFrom(Command.java:92)
              at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
              at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

            main

            "main" Id=1 Group=main WAITING on hudson.remoting.Engine@7f25ab41
            at java.lang.Object.wait(Native Method)

            • waiting on hudson.remoting.Engine@7f25ab41
              at java.lang.Thread.join(Unknown Source)
              at java.lang.Thread.join(Unknown Source)
              at hudson.remoting.jnlp.Main.main(Main.java:137)
              at hudson.remoting.jnlp.Main._main(Main.java:130)
              at hudson.remoting.Launcher.run(Launcher.java:223)
              at hudson.remoting.Launcher.main(Launcher.java:192)

            Ping thread for channel hudson.remoting.Channel@1c06d198:channel

            "Ping thread for channel hudson.remoting.Channel@1c06d198:channel" Id=13 Group=main TIMED_WAITING
            at java.lang.Thread.sleep(Native Method)
            at hudson.remoting.PingThread.run(PingThread.java:86)

            pool-1-thread-802

            "pool-1-thread-802" Id=916 Group=main RUNNABLE
            at sun.management.ThreadImpl.dumpThreads0(Native Method)
            at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
            at hudson.Functions.getThreadInfos(Functions.java:1131)
            at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96)
            at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
            at hudson.remoting.UserRequest.perform(UserRequest.java:118)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:326)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
            at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at hudson.remoting.Engine$1$1.run(Engine.java:63)
            at java.lang.Thread.run(Unknown Source)

            Number of locked synchronizers = 1

            • java.util.concurrent.ThreadPoolExecutor$Worker@13c5810e

            Thread-1

            "Thread-1" Id=10 Group=main WAITING on hudson.remoting.Channel@1c06d198
            at java.lang.Object.wait(Native Method)

            • waiting on hudson.remoting.Channel@1c06d198
              at java.lang.Object.wait(Object.java:503)
              at hudson.remoting.Channel.join(Channel.java:842)
              at hudson.remoting.Engine.run(Engine.java:271)

            Attach Listener

            "Attach Listener" Id=4 Group=system RUNNABLE

            Finalizer

            "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@4c9b77c
            at java.lang.Object.wait(Native Method)

            • waiting on java.lang.ref.ReferenceQueue$Lock@4c9b77c
              at java.lang.ref.ReferenceQueue.remove(Unknown Source)
              at java.lang.ref.ReferenceQueue.remove(Unknown Source)
              at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

            Reference Handler

            "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@572d3058
            at java.lang.Object.wait(Native Method)

            • waiting on java.lang.ref.Reference$Lock@572d3058
              at java.lang.Object.wait(Object.java:503)
              at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)

            LB3D-Wxpp32sp3
            Channel reader thread: channel

            "Channel reader thread: channel" Id=10 Group=main RUNNABLE (in native)
            at java.net.SocketInputStream.socketRead0(Native Method)
            at java.net.SocketInputStream.read(Unknown Source)
            at java.net.SocketInputStream.read(Unknown Source)
            at java.io.BufferedInputStream.fill(Unknown Source)
            at java.io.BufferedInputStream.read(Unknown Source)

            • locked java.io.BufferedInputStream@b9293d
              at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
              at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
              at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
              at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
              at java.io.ObjectInputStream.readObject0(Unknown Source)
              at java.io.ObjectInputStream.readObject(Unknown Source)
              at hudson.remoting.Command.readFrom(Command.java:92)
              at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
              at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

            main

            "main" Id=1 Group=main WAITING on hudson.remoting.Engine@1059a9a
            at java.lang.Object.wait(Native Method)

            • waiting on hudson.remoting.Engine@1059a9a
              at java.lang.Thread.join(Unknown Source)
              at java.lang.Thread.join(Unknown Source)
              at hudson.remoting.jnlp.Main.main(Main.java:137)
              at hudson.remoting.jnlp.Main._main(Main.java:130)
              at hudson.remoting.Launcher.run(Launcher.java:223)
              at hudson.remoting.Launcher.main(Launcher.java:192)

            Ping thread for channel hudson.remoting.Channel@41f871:channel

            "Ping thread for channel hudson.remoting.Channel@41f871:channel" Id=12 Group=main TIMED_WAITING
            at java.lang.Thread.sleep(Native Method)
            at hudson.remoting.PingThread.run(PingThread.java:86)

            pool-1-thread-8

            "pool-1-thread-8" Id=42 Group=main RUNNABLE
            at sun.management.ThreadImpl.dumpThreads0(Native Method)
            at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
            at hudson.Functions.getThreadInfos(Functions.java:1131)
            at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96)
            at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
            at hudson.remoting.UserRequest.perform(UserRequest.java:118)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:326)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
            at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at hudson.remoting.Engine$1$1.run(Engine.java:63)
            at java.lang.Thread.run(Unknown Source)

            Number of locked synchronizers = 1

            • java.util.concurrent.ThreadPoolExecutor$Worker@5322ec

            Thread-1

            "Thread-1" Id=9 Group=main WAITING on hudson.remoting.Channel@41f871
            at java.lang.Object.wait(Native Method)

            • waiting on hudson.remoting.Channel@41f871
              at java.lang.Object.wait(Object.java:503)
              at hudson.remoting.Channel.join(Channel.java:842)
              at hudson.remoting.Engine.run(Engine.java:271)

            Attach Listener

            "Attach Listener" Id=4 Group=system RUNNABLE

            Finalizer

            "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@429b8e
            at java.lang.Object.wait(Native Method)

            • waiting on java.lang.ref.ReferenceQueue$Lock@429b8e
              at java.lang.ref.ReferenceQueue.remove(Unknown Source)
              at java.lang.ref.ReferenceQueue.remove(Unknown Source)
              at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

            Reference Handler

            "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@f2c136
            at java.lang.Object.wait(Native Method)

            • waiting on java.lang.ref.Reference$Lock@f2c136
              at java.lang.Object.wait(Object.java:503)
              at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)

            Slave-TestProcessTest
            Channel reader thread: channel

            "Channel reader thread: channel" Id=10 Group=main RUNNABLE (in native)
            at java.net.SocketInputStream.socketRead0(Native Method)
            at java.net.SocketInputStream.read(Unknown Source)
            at java.net.SocketInputStream.read(Unknown Source)
            at java.io.BufferedInputStream.fill(Unknown Source)
            at java.io.BufferedInputStream.read(Unknown Source)

            • locked java.io.BufferedInputStream@18e02f3
              at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
              at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
              at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
              at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
              at java.io.ObjectInputStream.readObject0(Unknown Source)
              at java.io.ObjectInputStream.readObject(Unknown Source)
              at hudson.remoting.Command.readFrom(Command.java:92)
              at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
              at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

            main

            "main" Id=1 Group=main WAITING on hudson.remoting.Engine@13ac592
            at java.lang.Object.wait(Native Method)

            • waiting on hudson.remoting.Engine@13ac592
              at java.lang.Thread.join(Unknown Source)
              at java.lang.Thread.join(Unknown Source)
              at hudson.remoting.jnlp.Main.main(Main.java:137)
              at hudson.remoting.jnlp.Main._main(Main.java:130)
              at hudson.remoting.Launcher.run(Launcher.java:223)
              at hudson.remoting.Launcher.main(Launcher.java:192)

            Ping thread for channel hudson.remoting.Channel@1a32902:channel

            "Ping thread for channel hudson.remoting.Channel@1a32902:channel" Id=12 Group=main TIMED_WAITING
            at java.lang.Thread.sleep(Native Method)
            at hudson.remoting.PingThread.run(PingThread.java:86)

            pool-1-svnkit-thread-1

            "pool-1-svnkit-thread-1" Id=60 Group=main WAITING on java.util.concurrent.SynchronousQueue$TransferStack@a0f65b
            at sun.misc.Unsafe.park(Native Method)

            • waiting on java.util.concurrent.SynchronousQueue$TransferStack@a0f65b
              at java.util.concurrent.locks.LockSupport.park(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
              at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
              at java.util.concurrent.SynchronousQueue.take(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)

            pool-1-thread-79

            "pool-1-thread-79" Id=135 Group=main RUNNABLE
            at sun.management.ThreadImpl.dumpThreads0(Native Method)
            at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
            at hudson.Functions.getThreadInfos(Functions.java:1131)
            at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96)
            at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
            at hudson.remoting.UserRequest.perform(UserRequest.java:118)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:326)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
            at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at hudson.remoting.Engine$1$1.run(Engine.java:63)
            at java.lang.Thread.run(Unknown Source)

            Number of locked synchronizers = 1

            • java.util.concurrent.ThreadPoolExecutor$Worker@1232e8d

            Thread-1

            "Thread-1" Id=9 Group=main WAITING on hudson.remoting.Channel@1a32902
            at java.lang.Object.wait(Native Method)

            • waiting on hudson.remoting.Channel@1a32902
              at java.lang.Object.wait(Object.java:503)
              at hudson.remoting.Channel.join(Channel.java:842)
              at hudson.remoting.Engine.run(Engine.java:271)

            Attach Listener

            "Attach Listener" Id=4 Group=system RUNNABLE

            Finalizer

            "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@63bafc
            at java.lang.Object.wait(Native Method)

            • waiting on java.lang.ref.ReferenceQueue$Lock@63bafc
              at java.lang.ref.ReferenceQueue.remove(Unknown Source)
              at java.lang.ref.ReferenceQueue.remove(Unknown Source)
              at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

            Reference Handler

            "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@1893737
            at java.lang.Object.wait(Native Method)

            • waiting on java.lang.ref.Reference$Lock@1893737
              at java.lang.Object.wait(Object.java:503)
              at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
            Show
            odklizec Pavel Kudrys added a comment - It happened again today Any idea? Anyone please... This is what I see in jenkins-slave.err.log: Oct 31, 2013 7:41:16 AM hudson.remoting.jnlp.Main$CuiListener <init> INFO: Jenkins agent is running in headless mode. Oct 31, 2013 7:41:16 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among http://tcskjira:8095/ Oct 31, 2013 7:41:17 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to tcskjira:1695 Oct 31, 2013 7:41:17 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Oct 31, 2013 7:41:17 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected This is what I see in jenkins-slave.wrapper.log: 2013-10-31 07:41:09 - Starting C:\Program Files\Java\jre7\bin\java.exe -Xrs -jar "C:\Jenkins\slave.jar" -jnlpUrl http://tcskjira:8095/computer/LB3D-Wxpp64sp2/slave-agent.jnlp -secret 1ba0fb4096bc76c5ef1332c6368cd286a76074134a7091cd7603e93ec938b060 2013-10-31 07:41:09 - Started 1576 This is what I see in Jenkins Log: Oct 31, 2013 7:40:34 AM INFO hudson.slaves.RetentionStrategy$Demand check Launching computer LB3D-Wxpp64sp2 as it has been in demand for 39 sec Oct 31, 2013 7:40:34 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog [LB3D-Wxpp64sp2] Starting Virtual Machine... Oct 31, 2013 7:40:35 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog [LB3D-Wxpp64sp2] Powering on VM Oct 31, 2013 7:40:51 AM INFO hudson.model.AsyncPeriodicWork$1 run Started PeriodicBackup Oct 31, 2013 7:40:51 AM INFO hudson.model.AsyncPeriodicWork$1 run Finished PeriodicBackup. 0 ms Oct 31, 2013 7:41:16 AM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run Accepted connection #40 from /172.28.63.95:1034 Oct 31, 2013 7:41:23 AM WARNING hudson.node_monitors.ResponseTimeMonitor$1 monitor Making CAAV5R19x64 offline because it’s not responding Oct 31, 2013 7:41:23 AM INFO org.jenkinsci.plugins.vSphereCloud InternalLog [CAAV5R19x64] Already disconnecting on a separate thread Oct 31, 2013 7:41:51 AM INFO hudson.model.AsyncPeriodicWork$1 run Started PeriodicBackup Oct 31, 2013 7:41:51 AM INFO hudson.model.AsyncPeriodicWork$1 run Finished PeriodicBackup. 0 ms Oct 31, 2013 7:41:52 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor Failed to monitor LB3D-Wxpp64sp2 for Free Swap Space java.util.concurrent.TimeoutException at hudson.remoting.Request$1.get(Request.java:275) at hudson.remoting.Request$1.get(Request.java:210) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282) Slave Log says this: JNLP agent connected from /172.28.63.95 <=== [JENKINS REMOTING CAPACITY] ===>Slave.jar version: 2.32 This is a Windows slave Slave successfully connected and online followed by spinning wheel... Slave Thread Dump: Channel reader thread: channel "Channel reader thread: channel" Id=11 Group=main RUNNABLE (in native) at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) locked java.io.BufferedInputStream@1c2ba649 at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77) at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) main "main" Id=1 Group=main WAITING on hudson.remoting.Engine@7a0136b0 at java.lang.Object.wait(Native Method) waiting on hudson.remoting.Engine@7a0136b0 at java.lang.Thread.join(Unknown Source) at java.lang.Thread.join(Unknown Source) at hudson.remoting.jnlp.Main.main(Main.java:137) at hudson.remoting.jnlp.Main._main(Main.java:130) at hudson.remoting.Launcher.run(Launcher.java:223) at hudson.remoting.Launcher.main(Launcher.java:192) Ping thread for channel hudson.remoting.Channel@28509953:channel "Ping thread for channel hudson.remoting.Channel@28509953:channel" Id=13 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) pool-1-thread-17 "pool-1-thread-17" Id=29 Group=main RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(Unknown Source) at hudson.Functions.getThreadInfos(Functions.java:1131) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@493df066 pool-1-thread-2 "pool-1-thread-2" Id=14 Group=main RUNNABLE at com.sun.jna.Native.initIDs(Native Method) at com.sun.jna.Native.<clinit>(Native.java:122) at org.tmatesoft.svn.core.internal.util.jna.JNALibraryLoader.<clinit>(JNALibraryLoader.java:50) at org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:93) at org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196) at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1869) at org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57) at hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:975) at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:952) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:885) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:872) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2420) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@5133066c pool-1-thread-5 "pool-1-thread-5" Id=17 Group=main RUNNABLE at com.sun.jna.Structure.<clinit>(Structure.java:134) at org.jvnet.hudson.Windows.monitor(Windows.java:40) at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:110) at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:100) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@786dee4c Thread-1 "Thread-1" Id=10 Group=main WAITING on hudson.remoting.Channel@28509953 at java.lang.Object.wait(Native Method) waiting on hudson.remoting.Channel@28509953 at java.lang.Object.wait(Object.java:503) at hudson.remoting.Channel.join(Channel.java:842) at hudson.remoting.Engine.run(Engine.java:271) Attach Listener "Attach Listener" Id=4 Group=system RUNNABLE Finalizer "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@75b21111 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.ReferenceQueue$Lock@75b21111 at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) Reference Handler "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@7d7eaf74 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.Reference$Lock@7d7eaf74 at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) System Thread Dump: Thread Dump AWT-Windows "AWT-Windows" Id=614 Group=main RUNNABLE (in native) at sun.awt.windows.WToolkit.eventLoop(Native Method) at sun.awt.windows.WToolkit.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Channel reader thread: LB3D-Wxpp32sp3 "Channel reader thread: LB3D-Wxpp32sp3" Id=12151 Group=main RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) locked java.io.BufferedInputStream@7a7ec776 at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77) at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) Channel reader thread: LB3D-Wxpp64sp2 "Channel reader thread: LB3D-Wxpp64sp2" Id=12178 Group=main RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) locked java.io.BufferedInputStream@764058b0 at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77) at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) Channel reader thread: Slave-TestProcessBuild "Channel reader thread: Slave-TestProcessBuild" Id=6006 Group=main RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) locked java.io.BufferedInputStream@3563374a at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77) at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) Channel reader thread: Slave-TestProcessTest "Channel reader thread: Slave-TestProcessTest" Id=6121 Group=main RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) locked java.io.BufferedInputStream@374b5f62 at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77) at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) DestroyJavaVM "DestroyJavaVM" Id=19 Group=main RUNNABLE Executor #0 for LB3D-Wxpp32sp3 : executing ipprotection_32_64 #171 "Executor #0 for LB3D-Wxpp32sp3 : executing ipprotection_32_64 #171" Id=10955 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.matrix.DefaultMatrixExecutionStrategyImpl.waitForCompletion(DefaultMatrixExecutionStrategyImpl.java:290) at hudson.matrix.DefaultMatrixExecutionStrategyImpl.run(DefaultMatrixExecutionStrategyImpl.java:141) at hudson.matrix.MatrixBuild$MatrixBuildExecution.doRun(MatrixBuild.java:355) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:562) at hudson.model.Run.execute(Run.java:1665) at hudson.matrix.MatrixBuild.run(MatrixBuild.java:304) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:230) Executor #0 for LB3D-Wxpp64sp2 : executing ipprotection_32_64 » LB3D-Wxpp64sp2 #171 / waiting for hudson.remoting.Channel@59f73f53:LB3D-Wxpp64sp2 "Executor #0 for LB3D-Wxpp64sp2 : executing ipprotection_32_64 » LB3D-Wxpp64sp2 #171 / waiting for hudson.remoting.Channel@59f73f53:LB3D-Wxpp64sp2" Id=11106 Group=main TIMED_WAITING on hudson.remoting.UserRequest@3690a4a8 at java.lang.Object.wait(Native Method) waiting on hudson.remoting.UserRequest@3690a4a8 at hudson.remoting.Request.call(Request.java:146) at hudson.remoting.Channel.call(Channel.java:714) at hudson.FilePath.act(FilePath.java:901) at hudson.FilePath.act(FilePath.java:885) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:848) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:786) at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:117) at hudson.model.AbstractProject.checkout(AbstractProject.java:1411) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:652) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:557) at hudson.model.Run.execute(Run.java:1665) at hudson.matrix.MatrixRun.run(MatrixRun.java:146) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:230) Handling GET /threadDump : RequestHandlerThread 595 "Handling GET /threadDump : RequestHandlerThread 595 " Id=12350 Group=main TIMED_WAITING on java.util.concurrent.FutureTask$Sync@4deadcb6 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.FutureTask$Sync@4deadcb6 at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source) at java.util.concurrent.FutureTask.get(Unknown Source) at hudson.remoting.LocalChannel$2.get(LocalChannel.java:81) at jenkins.model.Jenkins.getAllThreadDumps(Jenkins.java:2919) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258) at org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104) at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83) at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57) at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51) at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80) at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74) at org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61) at org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsIterator(ExpressionSupport.java:94) at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:89) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53) at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:727) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:841) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:630) at org.kohsuke.stapler.Stapler.service(Stapler.java:224) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:208) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:181) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:64) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@620c96ba HashSessionScavenger-0 "HashSessionScavenger-0" Id=11 Group=main TIMED_WAITING on java.util.TaskQueue@4a7cb042 at java.lang.Object.wait(Native Method) waiting on java.util.TaskQueue@4a7cb042 at java.util.TimerThread.mainLoop(Unknown Source) at java.util.TimerThread.run(Unknown Source) javamelody "javamelody" Id=27 Group=main TIMED_WAITING on java.util.TaskQueue@1b9a5f49 at java.lang.Object.wait(Native Method) waiting on java.util.TaskQueue@1b9a5f49 at java.util.TimerThread.mainLoop(Unknown Source) at java.util.TimerThread.run(Unknown Source) Jenkins cron thread "Jenkins cron thread" Id=14 Group=main TIMED_WAITING on java.util.TaskQueue@258e37c9 at java.lang.Object.wait(Native Method) waiting on java.util.TaskQueue@258e37c9 at java.util.TimerThread.mainLoop(Unknown Source) at java.util.TimerThread.run(Unknown Source) Jenkins UDP 33848 monitoring thread "Jenkins UDP 33848 monitoring thread" Id=31 Group=main RUNNABLE (in native) at java.net.TwoStacksPlainDatagramSocketImpl.receive0(Native Method) locked java.net.TwoStacksPlainDatagramSocketImpl@1f036f93 at java.net.TwoStacksPlainDatagramSocketImpl.receive(Unknown Source) locked java.net.TwoStacksPlainDatagramSocketImpl@1f036f93 at java.net.DatagramSocket.receive(Unknown Source) locked java.net.DatagramPacket@7279ad5b locked java.net.MulticastSocket@12646c14 at hudson.UDPBroadcastThread.run(UDPBroadcastThread.java:82) Jenkins-Remoting-Thread-1218 "Jenkins-Remoting-Thread-1218" Id=12433 Group=main RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(Unknown Source) at hudson.Functions.getThreadInfos(Functions.java:1131) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92) at hudson.remoting.LocalChannel$1.call(LocalChannel.java:52) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@1904332a Jenkins-Remoting-Thread-1219 "Jenkins-Remoting-Thread-1219" Id=12434 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@2baacf98 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@2baacf98 at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Jenkins-Remoting-Thread-1220 "Jenkins-Remoting-Thread-1220" Id=12435 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@2baacf98 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@2baacf98 at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) JmDNS(tcskjira.local.).State.Timer "JmDNS(tcskjira.local.).State.Timer" Id=48 Group=main TIMED_WAITING on java.util.TaskQueue@587c9b22 at java.lang.Object.wait(Native Method) waiting on java.util.TaskQueue@587c9b22 at java.util.TimerThread.mainLoop(Unknown Source) at java.util.TimerThread.run(Unknown Source) JmDNS(tcskjira.local.).Timer "JmDNS(tcskjira.local.).Timer" Id=47 Group=main TIMED_WAITING on java.util.TaskQueue@5cad0290 at java.lang.Object.wait(Native Method) waiting on java.util.TaskQueue@5cad0290 at java.util.TimerThread.mainLoop(Unknown Source) at java.util.TimerThread.run(Unknown Source) jnr.ffi.util.ref.internal.Finalizer "jnr.ffi.util.ref.internal.Finalizer" Id=5472 Group=main WAITING on java.lang.ref.ReferenceQueue$Lock@4c2f6a8 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.ReferenceQueue$Lock@4c2f6a8 at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at jnr.ffi.util.ref.internal.Finalizer.run(Finalizer.java:128) LauncherControlThread [ControlPort=-1] "LauncherControlThread [ControlPort=-1] " Id=17 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at winstone.Launcher.run(Launcher.java:245) at java.lang.Thread.run(Unknown Source) NioSocketAcceptor-1 "NioSocketAcceptor-1" Id=29 Group=main RUNNABLE (in native) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source) at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source) at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source) locked sun.nio.ch.Util$2@5feec26b locked java.util.Collections$UnmodifiableSet@29e1ac62 locked sun.nio.ch.WindowsSelectorImpl@4defca63 at sun.nio.ch.SelectorImpl.select(Unknown Source) at sun.nio.ch.SelectorImpl.select(Unknown Source) at org.apache.mina.transport.socket.nio.NioSocketAcceptor.select(NioSocketAcceptor.java:238) at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:432) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@29090dda Ping thread for channel hudson.remoting.Channel@1675652c:Slave-TestProcessTest "Ping thread for channel hudson.remoting.Channel@1675652c:Slave-TestProcessTest" Id=6122 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) Ping thread for channel hudson.remoting.Channel@50923d0b:Slave-TestProcessBuild "Ping thread for channel hudson.remoting.Channel@50923d0b:Slave-TestProcessBuild" Id=6007 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) Ping thread for channel hudson.remoting.Channel@59f73f53:LB3D-Wxpp64sp2 "Ping thread for channel hudson.remoting.Channel@59f73f53:LB3D-Wxpp64sp2" Id=12179 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) Ping thread for channel hudson.remoting.Channel@5ade8f7a:LB3D-Wxpp32sp3 "Ping thread for channel hudson.remoting.Channel@5ade8f7a:LB3D-Wxpp32sp3" Id=12152 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) pool-1-svnkit-thread-1 "pool-1-svnkit-thread-1" Id=411 Group=main WAITING on java.util.concurrent.SynchronousQueue$TransferStack@1bcf9209 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@1bcf9209 at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) pool-1-svnkit-thread-2 "pool-1-svnkit-thread-2" Id=2322 Group=main WAITING on java.util.concurrent.SynchronousQueue$TransferStack@1bcf9209 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@1bcf9209 at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) pool-4-thread-1272 "pool-4-thread-1272" Id=12398 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9 at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) pool-4-thread-1273 "pool-4-thread-1273" Id=12399 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9 at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) pool-4-thread-1274 "pool-4-thread-1274" Id=12417 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9 at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) pool-4-thread-1275 "pool-4-thread-1275" Id=12418 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@5da44db9 at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) pool-9-thread-1 "pool-9-thread-1" Id=44 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2ba02595 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2ba02595 at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) RequestHandlerThread 1 Selector0 "RequestHandlerThread 1 Selector0" Id=15 Group=main RUNNABLE (in native) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source) at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source) at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source) locked sun.nio.ch.Util$2@2fea3452 locked java.util.Collections$UnmodifiableSet@e2f4d5c locked sun.nio.ch.WindowsSelectorImpl@7ed452e4 at sun.nio.ch.SelectorImpl.select(Unknown Source) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@4da2d209 RequestHandlerThread 2 Acceptor0 SelectChannelConnector@0.0.0.0:8095 "RequestHandlerThread 2 Acceptor0 SelectChannelConnector@0.0.0.0:8095" Id=16 Group=main RUNNABLE (in native) at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source) locked java.lang.Object@33e5bfc7 at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@5db00667 RequestHandlerThread 603 "RequestHandlerThread 603 " Id=12388 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) RequestHandlerThread 605 "RequestHandlerThread 605 " Id=12391 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) RequestHandlerThread 607 "RequestHandlerThread 607 " Id=12427 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) RequestHandlerThread 608 "RequestHandlerThread 608 " Id=12431 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@5b231a8d at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) SocketListener(tcskjira.local.) "SocketListener(tcskjira.local.)" Id=46 Group=main RUNNABLE (in native) at java.net.TwoStacksPlainDatagramSocketImpl.receive0(Native Method) locked java.net.TwoStacksPlainDatagramSocketImpl@62836b78 at java.net.TwoStacksPlainDatagramSocketImpl.receive(Unknown Source) locked java.net.TwoStacksPlainDatagramSocketImpl@62836b78 at java.net.DatagramSocket.receive(Unknown Source) locked java.net.DatagramPacket@6756aa7b locked java.net.MulticastSocket@6ad2a63d at javax.jmdns.impl.SocketListener.run(SocketListener.java:41) TCP slave agent listener port=0 "TCP slave agent listener port=0" Id=30 Group=main RUNNABLE (in native) at java.net.TwoStacksPlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(Unknown Source) at java.net.PlainSocketImpl.accept(Unknown Source) locked java.net.SocksSocketImpl@7ef39e7a at java.net.ServerSocket.implAccept(Unknown Source) at java.net.ServerSocket.accept(Unknown Source) at hudson.TcpSlaveAgentListener.run(TcpSlaveAgentListener.java:93) ViewJob reload thread "ViewJob reload thread" Id=28 Group=main TIMED_WAITING on java.util.LinkedHashSet@39d0518e at java.lang.Object.wait(Native Method) waiting on java.util.LinkedHashSet@39d0518e at hudson.model.ViewJob$ReloadThread.getNext(ViewJob.java:176) at hudson.model.ViewJob$ReloadThread.run(ViewJob.java:193) Waiting to acquire c:\Jenkins\workspace\ipprotection_32_64 : SCM polling for hudson.matrix.MatrixProject@3b277576 [ipprotection_32_64] "Waiting to acquire c:\Jenkins\workspace\ipprotection_32_64 : SCM polling for hudson.matrix.MatrixProject@3b277576 [ipprotection_32_64] " Id=12065 Group=main WAITING on hudson.slaves.WorkspaceList@195952ce at java.lang.Object.wait(Native Method) waiting on hudson.slaves.WorkspaceList@195952ce at java.lang.Object.wait(Object.java:503) at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:232) at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:211) at hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1580) at hudson.model.AbstractProject._poll(AbstractProject.java:1558) at hudson.model.AbstractProject.poll(AbstractProject.java:1490) at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:439) at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:468) at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@3435c217 Attach Listener "Attach Listener" Id=5 Group=system RUNNABLE Finalizer "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@4017e342 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.ReferenceQueue$Lock@4017e342 at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) Java2D Disposer "Java2D Disposer" Id=12 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@1bf08062 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.ReferenceQueue$Lock@1bf08062 at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at sun.java2d.Disposer.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Reference Handler "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@604922ed at java.lang.Object.wait(Native Method) waiting on java.lang.ref.Reference$Lock@604922ed at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) Signal Dispatcher "Signal Dispatcher" Id=4 Group=system RUNNABLE CAAV5R19x64 N/A offline LB3D-Wxpp64sp2 Channel reader thread: channel "Channel reader thread: channel" Id=11 Group=main RUNNABLE (in native) at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) locked java.io.BufferedInputStream@1c2ba649 at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77) at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) main "main" Id=1 Group=main WAITING on hudson.remoting.Engine@7a0136b0 at java.lang.Object.wait(Native Method) waiting on hudson.remoting.Engine@7a0136b0 at java.lang.Thread.join(Unknown Source) at java.lang.Thread.join(Unknown Source) at hudson.remoting.jnlp.Main.main(Main.java:137) at hudson.remoting.jnlp.Main._main(Main.java:130) at hudson.remoting.Launcher.run(Launcher.java:223) at hudson.remoting.Launcher.main(Launcher.java:192) Ping thread for channel hudson.remoting.Channel@28509953:channel "Ping thread for channel hudson.remoting.Channel@28509953:channel" Id=13 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) pool-1-thread-18 "pool-1-thread-18" Id=30 Group=main RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(Unknown Source) at hudson.Functions.getThreadInfos(Functions.java:1131) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@59d9c141 pool-1-thread-2 "pool-1-thread-2" Id=14 Group=main RUNNABLE at com.sun.jna.Native.initIDs(Native Method) at com.sun.jna.Native.<clinit>(Native.java:122) at org.tmatesoft.svn.core.internal.util.jna.JNALibraryLoader.<clinit>(JNALibraryLoader.java:50) at org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:93) at org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196) at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1869) at org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57) at hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:975) at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:952) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:885) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:872) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2420) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@5133066c pool-1-thread-5 "pool-1-thread-5" Id=17 Group=main RUNNABLE at com.sun.jna.Structure.<clinit>(Structure.java:134) at org.jvnet.hudson.Windows.monitor(Windows.java:40) at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:110) at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:100) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@786dee4c Thread-1 "Thread-1" Id=10 Group=main WAITING on hudson.remoting.Channel@28509953 at java.lang.Object.wait(Native Method) waiting on hudson.remoting.Channel@28509953 at java.lang.Object.wait(Object.java:503) at hudson.remoting.Channel.join(Channel.java:842) at hudson.remoting.Engine.run(Engine.java:271) Attach Listener "Attach Listener" Id=4 Group=system RUNNABLE Finalizer "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@75b21111 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.ReferenceQueue$Lock@75b21111 at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) Reference Handler "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@7d7eaf74 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.Reference$Lock@7d7eaf74 at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) Slave-TestProcessBuild Channel reader thread: channel "Channel reader thread: channel" Id=11 Group=main RUNNABLE (in native) at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) locked java.io.BufferedInputStream@74784829 at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77) at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) main "main" Id=1 Group=main WAITING on hudson.remoting.Engine@7f25ab41 at java.lang.Object.wait(Native Method) waiting on hudson.remoting.Engine@7f25ab41 at java.lang.Thread.join(Unknown Source) at java.lang.Thread.join(Unknown Source) at hudson.remoting.jnlp.Main.main(Main.java:137) at hudson.remoting.jnlp.Main._main(Main.java:130) at hudson.remoting.Launcher.run(Launcher.java:223) at hudson.remoting.Launcher.main(Launcher.java:192) Ping thread for channel hudson.remoting.Channel@1c06d198:channel "Ping thread for channel hudson.remoting.Channel@1c06d198:channel" Id=13 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) pool-1-thread-802 "pool-1-thread-802" Id=916 Group=main RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(Unknown Source) at hudson.Functions.getThreadInfos(Functions.java:1131) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@13c5810e Thread-1 "Thread-1" Id=10 Group=main WAITING on hudson.remoting.Channel@1c06d198 at java.lang.Object.wait(Native Method) waiting on hudson.remoting.Channel@1c06d198 at java.lang.Object.wait(Object.java:503) at hudson.remoting.Channel.join(Channel.java:842) at hudson.remoting.Engine.run(Engine.java:271) Attach Listener "Attach Listener" Id=4 Group=system RUNNABLE Finalizer "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@4c9b77c at java.lang.Object.wait(Native Method) waiting on java.lang.ref.ReferenceQueue$Lock@4c9b77c at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) Reference Handler "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@572d3058 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.Reference$Lock@572d3058 at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) LB3D-Wxpp32sp3 Channel reader thread: channel "Channel reader thread: channel" Id=10 Group=main RUNNABLE (in native) at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) locked java.io.BufferedInputStream@b9293d at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77) at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) main "main" Id=1 Group=main WAITING on hudson.remoting.Engine@1059a9a at java.lang.Object.wait(Native Method) waiting on hudson.remoting.Engine@1059a9a at java.lang.Thread.join(Unknown Source) at java.lang.Thread.join(Unknown Source) at hudson.remoting.jnlp.Main.main(Main.java:137) at hudson.remoting.jnlp.Main._main(Main.java:130) at hudson.remoting.Launcher.run(Launcher.java:223) at hudson.remoting.Launcher.main(Launcher.java:192) Ping thread for channel hudson.remoting.Channel@41f871:channel "Ping thread for channel hudson.remoting.Channel@41f871:channel" Id=12 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) pool-1-thread-8 "pool-1-thread-8" Id=42 Group=main RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(Unknown Source) at hudson.Functions.getThreadInfos(Functions.java:1131) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@5322ec Thread-1 "Thread-1" Id=9 Group=main WAITING on hudson.remoting.Channel@41f871 at java.lang.Object.wait(Native Method) waiting on hudson.remoting.Channel@41f871 at java.lang.Object.wait(Object.java:503) at hudson.remoting.Channel.join(Channel.java:842) at hudson.remoting.Engine.run(Engine.java:271) Attach Listener "Attach Listener" Id=4 Group=system RUNNABLE Finalizer "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@429b8e at java.lang.Object.wait(Native Method) waiting on java.lang.ref.ReferenceQueue$Lock@429b8e at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) Reference Handler "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@f2c136 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.Reference$Lock@f2c136 at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) Slave-TestProcessTest Channel reader thread: channel "Channel reader thread: channel" Id=10 Group=main RUNNABLE (in native) at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) locked java.io.BufferedInputStream@18e02f3 at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77) at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) main "main" Id=1 Group=main WAITING on hudson.remoting.Engine@13ac592 at java.lang.Object.wait(Native Method) waiting on hudson.remoting.Engine@13ac592 at java.lang.Thread.join(Unknown Source) at java.lang.Thread.join(Unknown Source) at hudson.remoting.jnlp.Main.main(Main.java:137) at hudson.remoting.jnlp.Main._main(Main.java:130) at hudson.remoting.Launcher.run(Launcher.java:223) at hudson.remoting.Launcher.main(Launcher.java:192) Ping thread for channel hudson.remoting.Channel@1a32902:channel "Ping thread for channel hudson.remoting.Channel@1a32902:channel" Id=12 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) pool-1-svnkit-thread-1 "pool-1-svnkit-thread-1" Id=60 Group=main WAITING on java.util.concurrent.SynchronousQueue$TransferStack@a0f65b at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@a0f65b at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) pool-1-thread-79 "pool-1-thread-79" Id=135 Group=main RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(Unknown Source) at hudson.Functions.getThreadInfos(Functions.java:1131) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@1232e8d Thread-1 "Thread-1" Id=9 Group=main WAITING on hudson.remoting.Channel@1a32902 at java.lang.Object.wait(Native Method) waiting on hudson.remoting.Channel@1a32902 at java.lang.Object.wait(Object.java:503) at hudson.remoting.Channel.join(Channel.java:842) at hudson.remoting.Engine.run(Engine.java:271) Attach Listener "Attach Listener" Id=4 Group=system RUNNABLE Finalizer "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@63bafc at java.lang.Object.wait(Native Method) waiting on java.lang.ref.ReferenceQueue$Lock@63bafc at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) Reference Handler "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@1893737 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.Reference$Lock@1893737 at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
            Hide
            vmarin Valentin Marin added a comment - - edited

            I've also ran into a similar issue in a sense that it also involved stuck builds and "Failed to monitor <slave-name> for Free Swap Space" exception.

            However, the exception details are a bit different:

            WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor
            Failed to monitor <slave-name> for Free Swap Space
            java.util.concurrent.ExecutionException: java.lang.UnsatisfiedLinkError: com.sun.jna.Native.malloc(J)J
            at hudson.remoting.Channel$3.adapt(Channel.java:747)
            at hudson.remoting.Channel$3.adapt(Channel.java:742)
            at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
            at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97)
            at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282)
            Caused by: java.lang.UnsatisfiedLinkError: com.sun.jna.Native.malloc(J)J
            at com.sun.jna.Native.malloc(Native Method)
            at com.sun.jna.Memory.malloc(Memory.java:690)
            at com.sun.jna.Memory.(Memory.java:79)
            at com.sun.jna.Structure$AutoAllocated.(Structure.java:1366)
            at com.sun.jna.Structure.autoAllocate(Structure.java:231)
            at com.sun.jna.Structure.allocateMemory(Structure.java:302)
            at com.sun.jna.Structure.(Structure.java:179)
            at com.sun.jna.Structure.(Structure.java:169)
            at com.sun.jna.Structure.(Structure.java:165)
            at com.sun.jna.Structure.(Structure.java:156)
            at org.jvnet.hudson.Windows$MEMORYSTATUSEX.(Windows.java:55)
            at org.jvnet.hudson.Windows.monitor(Windows.java:40)
            at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:110)
            at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:100)
            at hudson.remoting.UserRequest.perform(UserRequest.java:118)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:326)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
            at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

            What I also noticed a couple of times is that java.exe process on master starts hobbing all of the available RAM plus it's thread count rises.

            I'm running Jenkins ver. 1.530 on a 2008 R2 server, nodes (all of them are windows servers 2003 x86) are managed via vSphere Cloud plug-in and are being controlled as a Windows service using administrator account.

            Show
            vmarin Valentin Marin added a comment - - edited I've also ran into a similar issue in a sense that it also involved stuck builds and "Failed to monitor <slave-name> for Free Swap Space" exception. However, the exception details are a bit different: WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor Failed to monitor <slave-name> for Free Swap Space java.util.concurrent.ExecutionException: java.lang.UnsatisfiedLinkError: com.sun.jna.Native.malloc(J)J at hudson.remoting.Channel$3.adapt(Channel.java:747) at hudson.remoting.Channel$3.adapt(Channel.java:742) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282) Caused by: java.lang.UnsatisfiedLinkError: com.sun.jna.Native.malloc(J)J at com.sun.jna.Native.malloc(Native Method) at com.sun.jna.Memory.malloc(Memory.java:690) at com.sun.jna.Memory.(Memory.java:79) at com.sun.jna.Structure$AutoAllocated.(Structure.java:1366) at com.sun.jna.Structure.autoAllocate(Structure.java:231) at com.sun.jna.Structure.allocateMemory(Structure.java:302) at com.sun.jna.Structure.(Structure.java:179) at com.sun.jna.Structure.(Structure.java:169) at com.sun.jna.Structure.(Structure.java:165) at com.sun.jna.Structure.(Structure.java:156) at org.jvnet.hudson.Windows$MEMORYSTATUSEX.(Windows.java:55) at org.jvnet.hudson.Windows.monitor(Windows.java:40) at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:110) at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:100) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) What I also noticed a couple of times is that java.exe process on master starts hobbing all of the available RAM plus it's thread count rises. I'm running Jenkins ver. 1.530 on a 2008 R2 server, nodes (all of them are windows servers 2003 x86) are managed via vSphere Cloud plug-in and are being controlled as a Windows service using administrator account.
            Hide
            jswager1 Jason Swager added a comment -

            I'm also encountering this problem. Same symptoms. Jenkins 1.538, Windows 2008 R2 server, vSphere Cloud plugin. All the slaves that have these symptoms are showing this only after a slave restart - if they are always on, they don't have this problem. So far, only seen on Windows slaves; they use the JNLP agent starting mechanism. Several versions of Windows; Windows 7, Windows 8, Windows XP.

            Show
            jswager1 Jason Swager added a comment - I'm also encountering this problem. Same symptoms. Jenkins 1.538, Windows 2008 R2 server, vSphere Cloud plugin. All the slaves that have these symptoms are showing this only after a slave restart - if they are always on, they don't have this problem. So far, only seen on Windows slaves; they use the JNLP agent starting mechanism. Several versions of Windows; Windows 7, Windows 8, Windows XP.
            Hide
            odklizec Pavel Kudrys added a comment -

            Hi guys,

            Good to hear I'm not alone in this trouble! Any idea who we should contact to look at this issue? I think my company would no mind to pay a "service fee" to get this issue fixed.

            Yesterday, I changed the way of starting the slave service. Instead of automatically started service, I'm now using a batch file executed during the windows start. Sadly, this did not help. Night build got stuck again with one slave and sure enough, the log contained several "Failed to monitor SlaveName for Free Swap Space" errors.

            Show
            odklizec Pavel Kudrys added a comment - Hi guys, Good to hear I'm not alone in this trouble! Any idea who we should contact to look at this issue? I think my company would no mind to pay a "service fee" to get this issue fixed. Yesterday, I changed the way of starting the slave service. Instead of automatically started service, I'm now using a batch file executed during the windows start. Sadly, this did not help. Night build got stuck again with one slave and sure enough, the log contained several "Failed to monitor SlaveName for Free Swap Space" errors.
            Hide
            jswager1 Jason Swager added a comment -

            Sadly, I have no idea where to start looking for a solution. Since this only happens on our large production system and not our smaller test system, I can't effectively try solutions. I was going to try a fresh Jenkins with just the vSphere plugin, then slowly add plugins to match production.

            Show
            jswager1 Jason Swager added a comment - Sadly, I have no idea where to start looking for a solution. Since this only happens on our large production system and not our smaller test system, I can't effectively try solutions. I was going to try a fresh Jenkins with just the vSphere plugin, then slowly add plugins to match production.
            Hide
            vmarin Valentin Marin added a comment -

            Guys, are you running Jenkins on built-in Winstone or Apache Tomcat?

            Show
            vmarin Valentin Marin added a comment - Guys, are you running Jenkins on built-in Winstone or Apache Tomcat?
            Hide
            odklizec Pavel Kudrys added a comment -

            Hi Valentin,

            We are running it on Winstone.

            Show
            odklizec Pavel Kudrys added a comment - Hi Valentin, We are running it on Winstone.
            Hide
            jswager1 Jason Swager added a comment -

            Winstone as well

            Show
            jswager1 Jason Swager added a comment - Winstone as well
            Hide
            vmarin Valentin Marin added a comment - - edited

            Same here. I've got a hunch though, that the exception is somehow related to it, so I'm going to set up Tomcat and see if it makes any difference.

            Show
            vmarin Valentin Marin added a comment - - edited Same here. I've got a hunch though, that the exception is somehow related to it, so I'm going to set up Tomcat and see if it makes any difference.
            Hide
            vmarin Valentin Marin added a comment - - edited

            So I've set Jenkins up to run on Tomcat 7 up and upgraded 'archive artifacts' plug-in to latest version for a good measure (since all of my builds are using that and it is known to be greedy with RAM) and what do you know, 3 days without stalled builds and counting. Will let you guys know if the issue indeed went away.

            Show
            vmarin Valentin Marin added a comment - - edited So I've set Jenkins up to run on Tomcat 7 up and upgraded 'archive artifacts' plug-in to latest version for a good measure (since all of my builds are using that and it is known to be greedy with RAM) and what do you know, 3 days without stalled builds and counting. Will let you guys know if the issue indeed went away.
            Hide
            vmarin Valentin Marin added a comment -

            After running the setup for a while, I came to the conclusion that the issue is still there, although with reduced amount of fails.

            Show
            vmarin Valentin Marin added a comment - After running the setup for a while, I came to the conclusion that the issue is still there, although with reduced amount of fails.
            Hide
            vladdrussian VladDRussian added a comment -

            Thought i'd throw in my errors. We are seeing HTTP stop responding and visualVM can't take a dump once it fails. Memory is fine, but threads just keep increasing, once they hit 550-600 active it just stops and says memory issue won't respond till the master is restarted.
            Jenkins 1.484, win2k3 (master) win2k8/win2k3 (slaves)

            Feb 25, 2014 9:21:06 AM hudson.model.DownloadService$Downloadable doPostBack
            INFO: Obtained the updated data file for hudson.tools.JDKInstaller
            Feb 25, 2014 9:21:06 AM hudson.model.DownloadService$Downloadable doPostBack
            INFO: Obtained the updated data file for hudson.tasks.Ant.AntInstaller
            Feb 25, 2014 9:21:06 AM org.kohsuke.stapler.compression.CompressionFilter reportException
            WARNING: Untrapped servlet exception
            java.io.IOException: Unable to delete E:\jenkins\updates\hudson.tasks.Maven.MavenInstaller
            at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:112)
            at hudson.util.TextFile.write(TextFile.java:81)
            at hudson.model.DownloadService$Downloadable.doPostBack(DownloadService.java:260)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
            at java.lang.reflect.Method.invoke(Unknown Source)
            at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:288)
            at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:151)
            at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:90)
            at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111)
            at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
            at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574)
            at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659)
            at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241)
            at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
            at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574)
            at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659)
            at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:384)
            at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574)
            at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659)
            at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:203)
            at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
            at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574)
            at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659)
            at org.kohsuke.stapler.Stapler.invoke(Stapler.java:488)
            at org.kohsuke.stapler.Stapler.service(Stapler.java:162)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
            at winstone.ServletConfiguration.execute(ServletConfiguration.java:248)
            at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
            at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376)
            at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
            at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
            at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
            at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
            at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
            at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
            at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
            at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
            at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
            at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
            at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
            at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
            at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
            at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
            at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
            at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
            at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
            at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
            at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
            at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)
            at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)
            at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
            at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

            FREE DISK SPACE MONITORING STILL IN PROGRESS, FAILED, THREADS LEFT, MAXING
            Feb 21, 2014 4:55:20 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
            INFO: Accepted connection #12 from /172.16.69.140:64901
            Feb 21, 2014 4:55:21 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
            INFO: Accepted connection #13 from /172.16.69.82:61763
            Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record <init>
            WARNING: Previous Free Swap Space monitoring activity still in progress. Interrupting
            Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record <init>
            WARNING: Previous Free Disk Space monitoring activity still in progress. Interrupting
            Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record <init>
            WARNING: Previous Free Temp Space monitoring activity still in progress. Interrupting
            Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run
            WARNING: Node monitoring FFX-MOM-BLD10 for Free Swap Space aborted.
            java.lang.InterruptedException
            at java.lang.Object.wait(Native Method)
            at hudson.remoting.Request.call(Request.java:146)
            at hudson.remoting.Channel.call(Channel.java:664)
            at hudson.node_monitors.SwapSpaceMonitor$1.monitor(SwapSpaceMonitor.java:83)
            at hudson.node_monitors.SwapSpaceMonitor$1.monitor(SwapSpaceMonitor.java:81)
            at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219)
            Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run
            WARNING: Node monitoring FFX-MOM-BLD5 for Free Temp Space aborted.
            java.lang.InterruptedException
            at java.lang.Object.wait(Native Method)
            at hudson.remoting.Request.call(Request.java:146)
            at hudson.remoting.Channel.call(Channel.java:664)
            at hudson.FilePath.act(FilePath.java:840)
            at hudson.FilePath.act(FilePath.java:824)
            at hudson.node_monitors.TemporarySpaceMonitor$1.getFreeSpace(TemporarySpaceMonitor.java:73)
            at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:169)
            at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:49)
            at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219)
            Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run
            WARNING: Node monitoring FFX-MOM-BLD5 for Free Disk Space aborted.
            java.lang.InterruptedException
            at java.lang.Object.wait(Native Method)
            at hudson.remoting.Request.call(Request.java:146)
            at hudson.remoting.Channel.call(Channel.java:664)
            at hudson.FilePath.act(FilePath.java:840)
            at hudson.FilePath.act(FilePath.java:824)
            at hudson.node_monitors.DiskSpaceMonitor$1.getFreeSpace(DiskSpaceMonitor.java:71)
            at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:169)
            at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:49)
            at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219)

            BELOW LEAVING OPEN THREADS AND MAXING????
            Feb 20, 2014 6:32:49 AM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run
            WARNING: Failed to monitor FFX-MOM-BLD10 for Free Temp Space
            hudson.util.IOException2: remote file operation failed: E:\jenkins at hudson.remoting.Channel@18b622:FFX-MOM-BLD10
            at hudson.FilePath.act(FilePath.java:847)
            at hudson.FilePath.act(FilePath.java:824)
            at hudson.node_monitors.TemporarySpaceMonitor$1.getFreeSpace(TemporarySpaceMonitor.java:73)
            at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:169)
            at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:49)
            at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219)
            Caused by: java.io.IOException: Remote call on FFX-MOM-BLD10 failed
            at hudson.remoting.Channel.call(Channel.java:673)
            at hudson.FilePath.act(FilePath.java:840)
            ... 5 more
            Caused by: java.lang.Error: Failed to deserialize the Callable object.
            at hudson.remoting.UserRequest.perform(UserRequest.java:104)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:326)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
            at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at hudson.remoting.Engine$1$1.run(Engine.java:60)
            at java.lang.Thread.run(Unknown Source)
            Caused by: java.lang.reflect.UndeclaredThrowableException
            at $Proxy6.fetch2(Unknown Source)
            at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:122)
            at java.lang.ClassLoader.loadClass(Unknown Source)
            at java.lang.ClassLoader.loadClass(Unknown Source)
            at java.lang.ClassLoader.loadClassInternal(Unknown Source)
            at java.lang.Class.getDeclaredMethods0(Native Method)
            at java.lang.Class.privateGetDeclaredMethods(Unknown Source)
            at java.lang.Class.getDeclaredMethod(Unknown Source)
            at java.io.ObjectStreamClass.getPrivateMethod(Unknown Source)
            at java.io.ObjectStreamClass.access$1700(Unknown Source)
            at java.io.ObjectStreamClass$2.run(Unknown Source)
            at java.security.AccessController.doPrivileged(Native Method)
            at java.io.ObjectStreamClass.<init>(Unknown Source)
            at java.io.ObjectStreamClass.lookup(Unknown Source)
            at java.io.ObjectStreamClass.initNonProxy(Unknown Source)
            at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)
            at java.io.ObjectInputStream.readClassDesc(Unknown Source)
            at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
            at java.io.ObjectInputStream.readObject0(Unknown Source)
            at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
            at java.io.ObjectInputStream.readSerialData(Unknown Source)
            at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
            at java.io.ObjectInputStream.readObject0(Unknown Source)
            at java.io.ObjectInputStream.readObject(Unknown Source)
            at hudson.remoting.UserRequest.deserialize(UserRequest.java:182)
            at hudson.remoting.UserRequest.perform(UserRequest.java:98)
            ... 9 more
            Caused by: java.lang.InterruptedException
            at java.lang.Object.wait(Native Method)
            at hudson.remoting.Request.call(Request.java:146)
            at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:160)
            ... 35 more
            Feb 20, 2014 6:32:49 AM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run
            WARNING: Failed to monitor FFX-MOM-BLD12 for Free Swap Space
            java.io.IOException: Remote call on FFX-MOM-BLD12 failed
            at hudson.remoting.Channel.call(Channel.java:673)
            at hudson.node_monitors.SwapSpaceMonitor$1.monitor(SwapSpaceMonitor.java:83)
            at hudson.node_monitors.SwapSpaceMonitor$1.monitor(SwapSpaceMonitor.java:81)
            at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219)
            Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.jvnet.hudson.Windows$Kernel32
            at org.jvnet.hudson.Windows.monitor(Windows.java:41)
            at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:113)
            at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:99)
            at hudson.remoting.UserRequest.perform(UserRequest.java:118)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:326)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
            at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
            at java.util.concurrent.FutureTask.run(FutureTask.java:138)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
            at hudson.remoting.Engine$1$1.run(Engine.java:60)
            at java.lang.Thread.run(Thread.java:619)
            Feb 20, 2014 7:32:49 AM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run
            WARNING: Failed to monitor FFX-MOM-BLD10 for Free Temp Space
            hudson.util.IOException2: remote file operation failed: E:\jenkins at hudson.remoting.Channel@18b622:FFX-MOM-BLD10
            at hudson.FilePath.act(FilePath.java:847)
            at hudson.FilePath.act(FilePath.java:824)
            at hudson.node_monitors.TemporarySpaceMonitor$1.getFreeSpace(TemporarySpaceMonitor.java:73)
            at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:169)
            at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:49)
            at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219)
            Caused by: java.io.IOException: Remote call on FFX-MOM-BLD10 failed
            at hudson.remoting.Channel.call(Channel.java:673)
            at hudson.FilePath.act(FilePath.java:840)
            ... 5 more
            Caused by: java.lang.Error: Failed to deserialize the Callable object.
            at hudson.remoting.UserRequest.perform(UserRequest.java:104)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:326)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
            at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at hudson.remoting.Engine$1$1.run(Engine.java:60)
            at java.lang.Thread.run(Unknown Source)
            Caused by: java.lang.reflect.UndeclaredThrowableException
            at $Proxy6.fetch2(Unknown Source)
            at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:122)
            at java.lang.ClassLoader.loadClass(Unknown Source)
            at java.lang.ClassLoader.loadClass(Unknown Source)
            at java.lang.ClassLoader.loadClassInternal(Unknown Source)
            at java.lang.Class.getDeclaredMethods0(Native Method)
            at java.lang.Class.privateGetDeclaredMethods(Unknown Source)
            at java.lang.Class.getDeclaredMethod(Unknown Source)
            at java.io.ObjectStreamClass.getPrivateMethod(Unknown Source)
            at java.io.ObjectStreamClass.access$1700(Unknown Source)
            at java.io.ObjectStreamClass$2.run(Unknown Source)
            at java.security.AccessController.doPrivileged(Native Method)
            at java.io.ObjectStreamClass.<init>(Unknown Source)
            at java.io.ObjectStreamClass.lookup(Unknown Source)
            at java.io.ObjectStreamClass.initNonProxy(Unknown Source)
            at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)
            at java.io.ObjectInputStream.readClassDesc(Unknown Source)
            at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
            at java.io.ObjectInputStream.readObject0(Unknown Source)
            at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
            at java.io.ObjectInputStream.readSerialData(Unknown Source)
            at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
            at java.io.ObjectInputStream.readObject0(Unknown Source)
            at java.io.ObjectInputStream.readObject(Unknown Source)
            at hudson.remoting.UserRequest.deserialize(UserRequest.java:182)
            at hudson.remoting.UserRequest.perform(UserRequest.java:98)
            ... 9 more
            Caused by: java.lang.InterruptedException
            at java.lang.Object.wait(Native Method)
            at hudson.remoting.Request.call(Request.java:146)
            at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:160)
            ... 35 more

            Feb 20, 2014 7:50:42 AM winstone.Logger logInternal
            SEVERE: Error during HTTP listener init or shutdown
            java.lang.OutOfMemoryError: unable to create new native thread
            at java.lang.Thread.start0(Native Method)
            at java.lang.Thread.start(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
            at winstone.BoundedExecutorService.scheduleNext(BoundedExecutorService.java:74)
            at winstone.BoundedExecutorService.execute(BoundedExecutorService.java:64)
            at java.util.concurrent.AbstractExecutorService.submit(Unknown Source)
            at winstone.ObjectPool.handleRequest(ObjectPool.java:102)
            at winstone.HttpListener.run(HttpListener.java:144)
            at java.lang.Thread.run(Unknown Source)
            Feb 20, 2014 7:50:45 AM org.kohsuke.stapler.compression.CompressionFilter reportException
            WARNING: Untrapped servlet exception
            winstone.ClientSocketException: Failed to write to client
            at winstone.ClientOutputStream.write(ClientOutputStream.java:41)
            at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
            at java.io.BufferedOutputStream.flush(Unknown Source)
            at winstone.WinstoneOutputStream.commit(WinstoneOutputStream.java:167)
            at winstone.WinstoneOutputStream.commit(WinstoneOutputStream.java:123)
            at winstone.WinstoneOutputStream.write(WinstoneOutputStream.java:112)
            at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:421)
            at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:249)
            at org.kohsuke.stapler.Stapler.service(Stapler.java:152)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
            at winstone.ServletConfiguration.execute(ServletConfiguration.java:248)
            at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
            at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376)
            at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
            at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
            at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
            at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
            at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
            at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
            at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
            at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
            at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
            at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
            at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
            at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
            at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
            at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
            at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
            at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
            at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
            at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
            at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
            at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)
            at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)
            at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
            at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)
            Caused by: java.net.SocketException: Connection reset by peer: socket write error
            at java.net.SocketOutputStream.socketWrite0(Native Method)
            at java.net.SocketOutputStream.socketWrite(Unknown Source)
            at java.net.SocketOutputStream.write(Unknown Source)
            at winstone.ClientOutputStream.write(ClientOutputStream.java:39)
            ... 57 more

            REACHED LIMIT AND FAIL
            Feb 23, 2014 5:07:53 AM hudson.model.AsyncPeriodicWork$1 run
            INFO: Started Workspace clean-up
            Feb 23, 2014 5:07:56 AM hudson.model.AsyncPeriodicWork$1 run
            INFO: Finished Workspace clean-up. 3156 ms
            Feb 23, 2014 7:08:07 AM hudson.triggers.SafeTimerTask run
            SEVERE: Timer task hudson.node_monitors.AbstractNodeMonitorDescriptor$1@142971b failed
            java.lang.OutOfMemoryError: unable to create new native thread
            at java.lang.Thread.start0(Native Method)
            at java.lang.Thread.start(Unknown Source)
            at hudson.node_monitors.AbstractNodeMonitorDescriptor.triggerUpdate(AbstractNodeMonitorDescriptor.java:181)
            at hudson.node_monitors.AbstractNodeMonitorDescriptor$1.doRun(AbstractNodeMonitorDescriptor.java:78)
            at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
            at java.util.TimerThread.mainLoop(Unknown Source)
            at java.util.TimerThread.run(Unknown Source)
            Feb 23, 2014 7:15:08 AM winstone.Logger logInternal
            WARNING: Untrapped Error in Servlet
            java.lang.OutOfMemoryError: unable to create new native thread
            at java.lang.Thread.start0(Native Method)
            at java.lang.Thread.start(Unknown Source)
            at com4j.ComThread.<init>(ComThread.java:63)
            at com4j.ComThread.<init>(ComThread.java:21)
            at com4j.ComThread$1.initialValue(ComThread.java:31)
            at com4j.ComThread$1.initialValue(ComThread.java:26)
            at java.lang.ThreadLocal.setInitialValue(Unknown Source)
            at java.lang.ThreadLocal.get(Unknown Source)
            at com4j.ComThread.get(ComThread.java:39)
            at com4j.Task.execute(Task.java:25)
            at com4j.COM4J.createInstance(COM4J.java:97)
            at com4j.COM4J.createInstance(COM4J.java:72)
            at com4j.typelibs.ado20.ClassFactory.createCommand(ClassFactory.java:17)
            at hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider.getDnOfUserOrGroup(ActiveDirectoryAuthenticationProvider.java:162)
            at hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider.retrieveUser(ActiveDirectoryAuthenticationProvider.java:61)
            at org.acegisecurity.providers.dao.AbstractUserDetailsAuthenticationProvider.authenticate(AbstractUserDetailsAuthenticationProvider.java:119)
            at org.acegisecurity.providers.ProviderManager.doAuthentication(ProviderManager.java:195)
            at org.acegisecurity.AbstractAuthenticationManager.authenticate(AbstractAuthenticationManager.java:45)
            at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:138)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
            at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
            at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
            at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
            at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
            at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
            at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
            at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
            at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
            at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
            at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
            at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
            at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)
            at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)
            at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
            at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)
            Feb 23, 2014 7:15:08 AM winstone.Logger logInternal
            WARNING: Untrapped Error in Servlet
            java.lang.OutOfMemoryError: unable to create new native thread
            at java.lang.Thread.start0(Native Method)
            at java.lang.Thread.start(Unknown Source)
            at com4j.ComThread.<init>(ComThread.java:63)
            at com4j.ComThread.<init>(ComThread.java:21)
            at com4j.ComThread$1.initialValue(ComThread.java:31)
            at com4j.ComThread$1.initialValue(ComThread.java:26)
            at java.lang.ThreadLocal.setInitialValue(Unknown Source)
            at java.lang.ThreadLocal.get(Unknown Source)
            at com4j.ComThread.get(ComThread.java:39)
            at com4j.Task.execute(Task.java:25)
            at com4j.COM4J.createInstance(COM4J.java:97)
            at com4j.COM4J.createInstance(COM4J.java:72)
            at com4j.typelibs.ado20.ClassFactory.createCommand(ClassFactory.java:17)
            at hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider.getDnOfUserOrGroup(ActiveDirectoryAuthenticationProvider.java:162)
            at hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider.retrieveUser(ActiveDirectoryAuthenticationProvider.java:61)
            at org.acegisecurity.providers.dao.AbstractUserDetailsAuthenticationProvider.authenticate(AbstractUserDetailsAuthenticationProvider.java:119)
            at org.acegisecurity.providers.ProviderManager.doAuthentication(ProviderManager.java:195)
            at org.acegisecurity.AbstractAuthenticationManager.authenticate(AbstractAuthenticationManager.java:45)
            at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:138)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
            at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
            at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
            at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
            at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
            at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
            at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
            at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
            at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
            at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
            at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
            at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
            at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
            at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)
            at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)
            at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
            at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)
            Feb 23, 2014 7:15:08 AM winstone.Logger logInternal
            SEVERE: Error during HTTP listener init or shutdown
            java.lang.OutOfMemoryError: unable to create new native thread
            at java.lang.Thread.start0(Native Method)
            at java.lang.Thread.start(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
            at winstone.BoundedExecutorService.scheduleNext(BoundedExecutorService.java:74)
            at winstone.BoundedExecutorService.execute(BoundedExecutorService.java:64)
            at java.util.concurrent.AbstractExecutorService.submit(Unknown Source)
            at winstone.ObjectPool.handleRequest(ObjectPool.java:102)
            at winstone.HttpListener.run(HttpListener.java:144)
            at java.lang.Thread.run(Unknown Source)
            Feb 23, 2014 8:08:07 AM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record <init>
            WARNING: Previous Free Swap Space monitoring activity still in progress. Interrupting
            Feb 23, 2014 2:53:11 PM hudson.model.AsyncPeriodicWork$1 run
            INFO: Started Fingerprint cleanup
            Feb 23, 2014 2:53:11 PM hudson.model.FingerprintCleanupThread execute
            INFO: Cleaned up 0 records

            Show
            vladdrussian VladDRussian added a comment - Thought i'd throw in my errors. We are seeing HTTP stop responding and visualVM can't take a dump once it fails. Memory is fine, but threads just keep increasing, once they hit 550-600 active it just stops and says memory issue won't respond till the master is restarted. Jenkins 1.484, win2k3 (master) win2k8/win2k3 (slaves) Feb 25, 2014 9:21:06 AM hudson.model.DownloadService$Downloadable doPostBack INFO: Obtained the updated data file for hudson.tools.JDKInstaller Feb 25, 2014 9:21:06 AM hudson.model.DownloadService$Downloadable doPostBack INFO: Obtained the updated data file for hudson.tasks.Ant.AntInstaller Feb 25, 2014 9:21:06 AM org.kohsuke.stapler.compression.CompressionFilter reportException WARNING: Untrapped servlet exception java.io.IOException: Unable to delete E:\jenkins\updates\hudson.tasks.Maven.MavenInstaller at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:112) at hudson.util.TextFile.write(TextFile.java:81) at hudson.model.DownloadService$Downloadable.doPostBack(DownloadService.java:260) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:288) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:151) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:90) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659) at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:384) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:203) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:488) at org.kohsuke.stapler.Stapler.service(Stapler.java:162) at javax.servlet.http.HttpServlet.service(HttpServlet.java:45) at winstone.ServletConfiguration.execute(ServletConfiguration.java:248) at winstone.RequestDispatcher.forward(RequestDispatcher.java:333) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) FREE DISK SPACE MONITORING STILL IN PROGRESS, FAILED, THREADS LEFT, MAXING Feb 21, 2014 4:55:20 PM hudson.TcpSlaveAgentListener$ConnectionHandler run INFO: Accepted connection #12 from /172.16.69.140:64901 Feb 21, 2014 4:55:21 PM hudson.TcpSlaveAgentListener$ConnectionHandler run INFO: Accepted connection #13 from /172.16.69.82:61763 Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record <init> WARNING: Previous Free Swap Space monitoring activity still in progress. Interrupting Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record <init> WARNING: Previous Free Disk Space monitoring activity still in progress. Interrupting Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record <init> WARNING: Previous Free Temp Space monitoring activity still in progress. Interrupting Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run WARNING: Node monitoring FFX-MOM-BLD10 for Free Swap Space aborted. java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:146) at hudson.remoting.Channel.call(Channel.java:664) at hudson.node_monitors.SwapSpaceMonitor$1.monitor(SwapSpaceMonitor.java:83) at hudson.node_monitors.SwapSpaceMonitor$1.monitor(SwapSpaceMonitor.java:81) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219) Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run WARNING: Node monitoring FFX-MOM-BLD5 for Free Temp Space aborted. java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:146) at hudson.remoting.Channel.call(Channel.java:664) at hudson.FilePath.act(FilePath.java:840) at hudson.FilePath.act(FilePath.java:824) at hudson.node_monitors.TemporarySpaceMonitor$1.getFreeSpace(TemporarySpaceMonitor.java:73) at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:169) at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:49) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219) Feb 21, 2014 4:55:26 PM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run WARNING: Node monitoring FFX-MOM-BLD5 for Free Disk Space aborted. java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:146) at hudson.remoting.Channel.call(Channel.java:664) at hudson.FilePath.act(FilePath.java:840) at hudson.FilePath.act(FilePath.java:824) at hudson.node_monitors.DiskSpaceMonitor$1.getFreeSpace(DiskSpaceMonitor.java:71) at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:169) at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:49) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219) BELOW LEAVING OPEN THREADS AND MAXING???? Feb 20, 2014 6:32:49 AM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run WARNING: Failed to monitor FFX-MOM-BLD10 for Free Temp Space hudson.util.IOException2: remote file operation failed: E:\jenkins at hudson.remoting.Channel@18b622:FFX-MOM-BLD10 at hudson.FilePath.act(FilePath.java:847) at hudson.FilePath.act(FilePath.java:824) at hudson.node_monitors.TemporarySpaceMonitor$1.getFreeSpace(TemporarySpaceMonitor.java:73) at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:169) at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:49) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219) Caused by: java.io.IOException: Remote call on FFX-MOM-BLD10 failed at hudson.remoting.Channel.call(Channel.java:673) at hudson.FilePath.act(FilePath.java:840) ... 5 more Caused by: java.lang.Error: Failed to deserialize the Callable object. at hudson.remoting.UserRequest.perform(UserRequest.java:104) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:60) at java.lang.Thread.run(Unknown Source) Caused by: java.lang.reflect.UndeclaredThrowableException at $Proxy6.fetch2(Unknown Source) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:122) at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClassInternal(Unknown Source) at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Unknown Source) at java.lang.Class.getDeclaredMethod(Unknown Source) at java.io.ObjectStreamClass.getPrivateMethod(Unknown Source) at java.io.ObjectStreamClass.access$1700(Unknown Source) at java.io.ObjectStreamClass$2.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.<init>(Unknown Source) at java.io.ObjectStreamClass.lookup(Unknown Source) at java.io.ObjectStreamClass.initNonProxy(Unknown Source) at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source) at java.io.ObjectInputStream.readClassDesc(Unknown Source) at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.defaultReadFields(Unknown Source) at java.io.ObjectInputStream.readSerialData(Unknown Source) at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.UserRequest.deserialize(UserRequest.java:182) at hudson.remoting.UserRequest.perform(UserRequest.java:98) ... 9 more Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:146) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:160) ... 35 more Feb 20, 2014 6:32:49 AM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run WARNING: Failed to monitor FFX-MOM-BLD12 for Free Swap Space java.io.IOException: Remote call on FFX-MOM-BLD12 failed at hudson.remoting.Channel.call(Channel.java:673) at hudson.node_monitors.SwapSpaceMonitor$1.monitor(SwapSpaceMonitor.java:83) at hudson.node_monitors.SwapSpaceMonitor$1.monitor(SwapSpaceMonitor.java:81) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219) Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.jvnet.hudson.Windows$Kernel32 at org.jvnet.hudson.Windows.monitor(Windows.java:41) at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:113) at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:99) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at hudson.remoting.Engine$1$1.run(Engine.java:60) at java.lang.Thread.run(Thread.java:619) Feb 20, 2014 7:32:49 AM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record run WARNING: Failed to monitor FFX-MOM-BLD10 for Free Temp Space hudson.util.IOException2: remote file operation failed: E:\jenkins at hudson.remoting.Channel@18b622:FFX-MOM-BLD10 at hudson.FilePath.act(FilePath.java:847) at hudson.FilePath.act(FilePath.java:824) at hudson.node_monitors.TemporarySpaceMonitor$1.getFreeSpace(TemporarySpaceMonitor.java:73) at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:169) at hudson.node_monitors.DiskSpaceMonitorDescriptor.monitor(DiskSpaceMonitorDescriptor.java:49) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:219) Caused by: java.io.IOException: Remote call on FFX-MOM-BLD10 failed at hudson.remoting.Channel.call(Channel.java:673) at hudson.FilePath.act(FilePath.java:840) ... 5 more Caused by: java.lang.Error: Failed to deserialize the Callable object. at hudson.remoting.UserRequest.perform(UserRequest.java:104) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:60) at java.lang.Thread.run(Unknown Source) Caused by: java.lang.reflect.UndeclaredThrowableException at $Proxy6.fetch2(Unknown Source) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:122) at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClassInternal(Unknown Source) at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Unknown Source) at java.lang.Class.getDeclaredMethod(Unknown Source) at java.io.ObjectStreamClass.getPrivateMethod(Unknown Source) at java.io.ObjectStreamClass.access$1700(Unknown Source) at java.io.ObjectStreamClass$2.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.<init>(Unknown Source) at java.io.ObjectStreamClass.lookup(Unknown Source) at java.io.ObjectStreamClass.initNonProxy(Unknown Source) at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source) at java.io.ObjectInputStream.readClassDesc(Unknown Source) at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.defaultReadFields(Unknown Source) at java.io.ObjectInputStream.readSerialData(Unknown Source) at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.UserRequest.deserialize(UserRequest.java:182) at hudson.remoting.UserRequest.perform(UserRequest.java:98) ... 9 more Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:146) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:160) ... 35 more Feb 20, 2014 7:50:42 AM winstone.Logger logInternal SEVERE: Error during HTTP listener init or shutdown java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) at winstone.BoundedExecutorService.scheduleNext(BoundedExecutorService.java:74) at winstone.BoundedExecutorService.execute(BoundedExecutorService.java:64) at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) at winstone.ObjectPool.handleRequest(ObjectPool.java:102) at winstone.HttpListener.run(HttpListener.java:144) at java.lang.Thread.run(Unknown Source) Feb 20, 2014 7:50:45 AM org.kohsuke.stapler.compression.CompressionFilter reportException WARNING: Untrapped servlet exception winstone.ClientSocketException: Failed to write to client at winstone.ClientOutputStream.write(ClientOutputStream.java:41) at java.io.BufferedOutputStream.flushBuffer(Unknown Source) at java.io.BufferedOutputStream.flush(Unknown Source) at winstone.WinstoneOutputStream.commit(WinstoneOutputStream.java:167) at winstone.WinstoneOutputStream.commit(WinstoneOutputStream.java:123) at winstone.WinstoneOutputStream.write(WinstoneOutputStream.java:112) at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:421) at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:249) at org.kohsuke.stapler.Stapler.service(Stapler.java:152) at javax.servlet.http.HttpServlet.service(HttpServlet.java:45) at winstone.ServletConfiguration.execute(ServletConfiguration.java:248) at winstone.RequestDispatcher.forward(RequestDispatcher.java:333) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketException: Connection reset by peer: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(Unknown Source) at java.net.SocketOutputStream.write(Unknown Source) at winstone.ClientOutputStream.write(ClientOutputStream.java:39) ... 57 more REACHED LIMIT AND FAIL Feb 23, 2014 5:07:53 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started Workspace clean-up Feb 23, 2014 5:07:56 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished Workspace clean-up. 3156 ms Feb 23, 2014 7:08:07 AM hudson.triggers.SafeTimerTask run SEVERE: Timer task hudson.node_monitors.AbstractNodeMonitorDescriptor$1@142971b failed java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Unknown Source) at hudson.node_monitors.AbstractNodeMonitorDescriptor.triggerUpdate(AbstractNodeMonitorDescriptor.java:181) at hudson.node_monitors.AbstractNodeMonitorDescriptor$1.doRun(AbstractNodeMonitorDescriptor.java:78) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.TimerThread.mainLoop(Unknown Source) at java.util.TimerThread.run(Unknown Source) Feb 23, 2014 7:15:08 AM winstone.Logger logInternal WARNING: Untrapped Error in Servlet java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Unknown Source) at com4j.ComThread.<init>(ComThread.java:63) at com4j.ComThread.<init>(ComThread.java:21) at com4j.ComThread$1.initialValue(ComThread.java:31) at com4j.ComThread$1.initialValue(ComThread.java:26) at java.lang.ThreadLocal.setInitialValue(Unknown Source) at java.lang.ThreadLocal.get(Unknown Source) at com4j.ComThread.get(ComThread.java:39) at com4j.Task.execute(Task.java:25) at com4j.COM4J.createInstance(COM4J.java:97) at com4j.COM4J.createInstance(COM4J.java:72) at com4j.typelibs.ado20.ClassFactory.createCommand(ClassFactory.java:17) at hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider.getDnOfUserOrGroup(ActiveDirectoryAuthenticationProvider.java:162) at hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider.retrieveUser(ActiveDirectoryAuthenticationProvider.java:61) at org.acegisecurity.providers.dao.AbstractUserDetailsAuthenticationProvider.authenticate(AbstractUserDetailsAuthenticationProvider.java:119) at org.acegisecurity.providers.ProviderManager.doAuthentication(ProviderManager.java:195) at org.acegisecurity.AbstractAuthenticationManager.authenticate(AbstractAuthenticationManager.java:45) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:138) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Feb 23, 2014 7:15:08 AM winstone.Logger logInternal WARNING: Untrapped Error in Servlet java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Unknown Source) at com4j.ComThread.<init>(ComThread.java:63) at com4j.ComThread.<init>(ComThread.java:21) at com4j.ComThread$1.initialValue(ComThread.java:31) at com4j.ComThread$1.initialValue(ComThread.java:26) at java.lang.ThreadLocal.setInitialValue(Unknown Source) at java.lang.ThreadLocal.get(Unknown Source) at com4j.ComThread.get(ComThread.java:39) at com4j.Task.execute(Task.java:25) at com4j.COM4J.createInstance(COM4J.java:97) at com4j.COM4J.createInstance(COM4J.java:72) at com4j.typelibs.ado20.ClassFactory.createCommand(ClassFactory.java:17) at hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider.getDnOfUserOrGroup(ActiveDirectoryAuthenticationProvider.java:162) at hudson.plugins.active_directory.ActiveDirectoryAuthenticationProvider.retrieveUser(ActiveDirectoryAuthenticationProvider.java:61) at org.acegisecurity.providers.dao.AbstractUserDetailsAuthenticationProvider.authenticate(AbstractUserDetailsAuthenticationProvider.java:119) at org.acegisecurity.providers.ProviderManager.doAuthentication(ProviderManager.java:195) at org.acegisecurity.AbstractAuthenticationManager.authenticate(AbstractAuthenticationManager.java:45) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:138) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Feb 23, 2014 7:15:08 AM winstone.Logger logInternal SEVERE: Error during HTTP listener init or shutdown java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) at winstone.BoundedExecutorService.scheduleNext(BoundedExecutorService.java:74) at winstone.BoundedExecutorService.execute(BoundedExecutorService.java:64) at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) at winstone.ObjectPool.handleRequest(ObjectPool.java:102) at winstone.HttpListener.run(HttpListener.java:144) at java.lang.Thread.run(Unknown Source) Feb 23, 2014 8:08:07 AM hudson.node_monitors.AbstractNodeMonitorDescriptor$Record <init> WARNING: Previous Free Swap Space monitoring activity still in progress. Interrupting Feb 23, 2014 2:53:11 PM hudson.model.AsyncPeriodicWork$1 run INFO: Started Fingerprint cleanup Feb 23, 2014 2:53:11 PM hudson.model.FingerprintCleanupThread execute INFO: Cleaned up 0 records
            Hide
            reznicek Jan Řezníček added a comment -

            I am facing the same or similar problem.
            I have Jenkins linux master and slaves on the Proxmox Windows XP machines.
            Time to time, a slave stuck.
            The cosole output ends with line - building remotely on ...
            No SVN checkout is executed after, the build is totally stuck.
            The slave says: Slave successfully connected and online

            I have to restart the slave machine and abort the job. If i only abort the job, the next build is stuck too.

            Show
            reznicek Jan Řezníček added a comment - I am facing the same or similar problem. I have Jenkins linux master and slaves on the Proxmox Windows XP machines. Time to time, a slave stuck. The cosole output ends with line - building remotely on ... No SVN checkout is executed after, the build is totally stuck. The slave says: Slave successfully connected and online I have to restart the slave machine and abort the job. If i only abort the job, the next build is stuck too.
            Hide
            schristou Steven Christou added a comment -

            This is caused by a deadlock with the subversion plugin and the "Free swap space" monitor option. To prevent this issue you can go to Manage Jenkins > Manage Nodes > Configure and disable this option.

            Show
            schristou Steven Christou added a comment - This is caused by a deadlock with the subversion plugin and the "Free swap space" monitor option. To prevent this issue you can go to Manage Jenkins > Manage Nodes > Configure and disable this option.
            Hide
            danielbeck Daniel Beck added a comment -

            Steven Christou

            To prevent this issue you can go to Manage Jenkins > Manage Nodes > Configure and disable this option.

            Aren't node monitors still run, just won't take nodes offline that fail the check?

            Show
            danielbeck Daniel Beck added a comment - Steven Christou To prevent this issue you can go to Manage Jenkins > Manage Nodes > Configure and disable this option. Aren't node monitors still run, just won't take nodes offline that fail the check?
            Hide
            drtaddei Dave Taddei added a comment -

            We are seeing this intermittently on Jenkins 1.656 with a windows 7 remote slave. We have disabled the Free swap space monitoring option but still see the issue.

            Show
            drtaddei Dave Taddei added a comment - We are seeing this intermittently on Jenkins 1.656 with a windows 7 remote slave. We have disabled the Free swap space monitoring option but still see the issue.
            Hide
            bbva BBVA added a comment -

            We are also experiencing this problem. We have disabled all monitoring on the slaves.

            We updated to the latest SVN plugin 2.5.7, including SVNKit 1.8.11

            Seems like the problem is a deadlock in the startup of the windows slave. We are using cygwin/ssh connections, if that matters. Looking at the stack traces with SVN involved*:

            The java method getApplicationDataPath() is loading the JNA, which is synchronized in all of it's static methods. So as this is the first time the class is refenced is also doing the static initialization <clinit>. Looks like the Native.loadLibrary is stuck and producing the blockage, to all the threads using SVN.

            *) The actual thread stack dump. Involving SVN.

            "pool-1-thread-3 for channel" prio=5 RUNNABLE
            com.sun.jna.Native.initIDs(Native Method)
            com.sun.jna.Native.<clinit>(Native.java:122)
            org.tmatesoft.svn.core.internal.util.jna.JNALibraryLoader.<clinit>(JNALibraryLoader.java:50)
            org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:113)
            org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196)
            org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1948)
            org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57)
            hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:1055)
            hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1032)
            hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:965)
            hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:945)
            hudson.FilePath$FileCallableWrapper.call(FilePath.java:2577)
            hudson.remoting.UserRequest.perform(UserRequest.java:121)
            hudson.remoting.UserRequest.perform(UserRequest.java:49)
            hudson.remoting.Request$2.run(Request.java:324)
            hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
            java.util.concurrent.FutureTask.run(FutureTask.java:262)
            java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            java.lang.Thread.run(Thread.java:745)

            "pool-1-thread-52 for channel" prio=5 RUNNABLE
            org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:113)
            org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196)
            org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1948)
            org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57)
            hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:1055)
            hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1032)
            hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:965)
            hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:945)
            hudson.FilePath$FileCallableWrapper.call(FilePath.java:2577)
            hudson.remoting.UserRequest.perform(UserRequest.java:121)
            hudson.remoting.UserRequest.perform(UserRequest.java:49)
            hudson.remoting.Request$2.run(Request.java:324)
            hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
            java.util.concurrent.FutureTask.run(FutureTask.java:262)
            java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            java.lang.Thread.run(Thread.java:745)

            Show
            bbva BBVA added a comment - We are also experiencing this problem. We have disabled all monitoring on the slaves. We updated to the latest SVN plugin 2.5.7, including SVNKit 1.8.11 Seems like the problem is a deadlock in the startup of the windows slave. We are using cygwin/ssh connections, if that matters. Looking at the stack traces with SVN involved*: The java method getApplicationDataPath() is loading the JNA, which is synchronized in all of it's static methods. So as this is the first time the class is refenced is also doing the static initialization <clinit>. Looks like the Native.loadLibrary is stuck and producing the blockage, to all the threads using SVN. *) The actual thread stack dump. Involving SVN. "pool-1-thread-3 for channel" prio=5 RUNNABLE com.sun.jna.Native.initIDs(Native Method) com.sun.jna.Native.<clinit>(Native.java:122) org.tmatesoft.svn.core.internal.util.jna.JNALibraryLoader.<clinit>(JNALibraryLoader.java:50) org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:113) org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196) org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1948) org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57) hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:1055) hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1032) hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:965) hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:945) hudson.FilePath$FileCallableWrapper.call(FilePath.java:2577) hudson.remoting.UserRequest.perform(UserRequest.java:121) hudson.remoting.UserRequest.perform(UserRequest.java:49) hudson.remoting.Request$2.run(Request.java:324) hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) java.util.concurrent.FutureTask.run(FutureTask.java:262) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) "pool-1-thread-52 for channel" prio=5 RUNNABLE org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:113) org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196) org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1948) org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57) hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:1055) hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1032) hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:965) hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:945) hudson.FilePath$FileCallableWrapper.call(FilePath.java:2577) hudson.remoting.UserRequest.perform(UserRequest.java:121) hudson.remoting.UserRequest.perform(UserRequest.java:49) hudson.remoting.Request$2.run(Request.java:324) hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) java.util.concurrent.FutureTask.run(FutureTask.java:262) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745)
            Hide
            bbva BBVA added a comment -

            @See https://issues.tmatesoft.com/issue/SGT-451

            Slaves need to force load of jna classes at startup.

            Show
            bbva BBVA added a comment - @See https://issues.tmatesoft.com/issue/SGT-451 Slaves need to force load of jna classes at startup.
            Hide
            bbva BBVA added a comment -

            FYI, we have a patch for jenkins/remoting-2.47 that allows to by-pass this issue by defining property hudson.remoting.RemoteClassLoader.force to com.sun.jna.Native. This forces the load of the offending class early on at any UserRequest with a new ClassLoader.

            We have tested this patch timed job every 5 minutes, with a dormant slave that needs to be waken. We will be requesting merge in github master shortly.

            Show
            bbva BBVA added a comment - FYI, we have a patch for jenkins/remoting-2.47 that allows to by-pass this issue by defining property hudson.remoting.RemoteClassLoader.force to com.sun.jna.Native . This forces the load of the offending class early on at any UserRequest with a new ClassLoader. We have tested this patch timed job every 5 minutes, with a dormant slave that needs to be waken. We will be requesting merge in github master shortly.
            Hide
            danielbeck Daniel Beck added a comment -

            FYI Oleg Nenashev See above.

            Show
            danielbeck Daniel Beck added a comment - FYI Oleg Nenashev See above.
            Hide
            ddaumiller Dorian Daumiller added a comment - - edited

            Deactivating the (all) monitorings on the Jenkins master did not help us. Before reconnecting the swarm client, we see this:

            Curiously, we also still see the monitoring in the Jenkins log:
            {{
            May 09, 2016 9:38:39 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor

            Failed to monitor ****-ed055395 for Free Swap Space
            java.util.concurrent.TimeoutException
            at hudson.remoting.Request$1.get(Request.java:273)
            at hudson.remoting.Request$1.get(Request.java:207)
            at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
            at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96)
            at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
            }}

            On the build node, amidst tons of threads, we see two that are referencing org.tmatesoft.svn.core.internal.util.jna:

            {{pool-1-thread-2 for channel

            "pool-1-thread-2 for channel" Id=16 Group=main RUNNABLE
            at com.sun.jna.Native.initIDs(Native Method)
            at com.sun.jna.Native.<clinit>(Native.java:148)
            at org.tmatesoft.svn.core.internal.util.jna.JNALibraryLoader.<clinit>(JNALibraryLoader.java:50)
            at org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:113)
            at org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196)
            at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1948)
            at org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57)
            at hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:1055)
            at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1032)
            at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:965)
            at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:945)
            at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2719)
            at hudson.remoting.UserRequest.perform(UserRequest.java:121)
            at hudson.remoting.UserRequest.perform(UserRequest.java:49)
            at hudson.remoting.Request$2.run(Request.java:324)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at hudson.remoting.Engine$1$1.run(Engine.java:63)
            at java.lang.Thread.run(Unknown Source)

            Number of locked synchronizers = 1

            • java.util.concurrent.ThreadPoolExecutor$Worker@5f599d56
              }}
              {{
              pool-1-thread-916 for channel

            "pool-1-thread-916 for channel" Id=930 Group=main RUNNABLE
            at org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:113)
            at org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196)
            at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1948)
            at org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57)
            at hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:1055)
            at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1032)
            at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:965)
            at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:945)
            at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2719)
            at hudson.remoting.UserRequest.perform(UserRequest.java:121)
            at hudson.remoting.UserRequest.perform(UserRequest.java:49)
            at hudson.remoting.Request$2.run(Request.java:324)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at hudson.remoting.Engine$1$1.run(Engine.java:63)
            at java.lang.Thread.run(Unknown Source)

            Number of locked synchronizers = 1

            • java.util.concurrent.ThreadPoolExecutor$Worker@772c6f4f
              }}

            After reconnecting two swarm clients (but not restarting Jenkins), one builds, one hangs, showing similar traces.
            From the hanging client:

            {{pool-1-thread-6 for channel

            "pool-1-thread-6 for channel" Id=20 Group=main RUNNABLE
            at com.sun.jna.Pointer.<clinit>(Pointer.java:41)
            at com.sun.jna.Structure.<clinit>(Structure.java:2078)
            at org.jvnet.hudson.Windows.monitor(Windows.java:42)
            at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:113)
            at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:103)
            at hudson.remoting.UserRequest.perform(UserRequest.java:121)
            at hudson.remoting.UserRequest.perform(UserRequest.java:49)
            at hudson.remoting.Request$2.run(Request.java:324)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at hudson.remoting.Engine$1$1.run(Engine.java:63)
            at java.lang.Thread.run(Unknown Source)

            Number of locked synchronizers = 1

            • java.util.concurrent.ThreadPoolExecutor$Worker@2e201c5c}}

            It seems the FreeSwapMonitoring starts a new thread on the node every time it is started (we can start it manually using the "Refresh Status" Button in the BuildExecutor Status view).
            We see that some threads are actually closed, so it's not that all threads hang, but for every time we hit refresh status, we see a new instance of at ```org.jvnet.hudson.Windows.monitor``` (if we wait some minute between hitting that button)

            Show
            ddaumiller Dorian Daumiller added a comment - - edited Deactivating the (all) monitorings on the Jenkins master did not help us. Before reconnecting the swarm client, we see this: Curiously, we also still see the monitoring in the Jenkins log: {{ May 09, 2016 9:38:39 AM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor Failed to monitor ****-ed055395 for Free Swap Space java.util.concurrent.TimeoutException at hudson.remoting.Request$1.get(Request.java:273) at hudson.remoting.Request$1.get(Request.java:207) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) }} On the build node, amidst tons of threads, we see two that are referencing org.tmatesoft.svn.core.internal.util.jna : {{pool-1-thread-2 for channel "pool-1-thread-2 for channel" Id=16 Group=main RUNNABLE at com.sun.jna.Native.initIDs(Native Method) at com.sun.jna.Native.<clinit>(Native.java:148) at org.tmatesoft.svn.core.internal.util.jna.JNALibraryLoader.<clinit>(JNALibraryLoader.java:50) at org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:113) at org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196) at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1948) at org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57) at hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:1055) at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1032) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:965) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:945) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2719) at hudson.remoting.UserRequest.perform(UserRequest.java:121) at hudson.remoting.UserRequest.perform(UserRequest.java:49) at hudson.remoting.Request$2.run(Request.java:324) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@5f599d56 }} {{ pool-1-thread-916 for channel "pool-1-thread-916 for channel" Id=930 Group=main RUNNABLE at org.tmatesoft.svn.core.internal.util.jna.SVNWin32Util.getApplicationDataPath(SVNWin32Util.java:113) at org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getApplicationDataPath(SVNJNAUtil.java:196) at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getApplicationDataPath(SVNFileUtil.java:1948) at org.tmatesoft.svn.core.wc.SVNWCUtil.getDefaultConfigurationDirectory(SVNWCUtil.java:57) at hudson.scm.SubversionSCM.createSvnAuthenticationManager(SubversionSCM.java:1055) at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1032) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:965) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:945) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2719) at hudson.remoting.UserRequest.perform(UserRequest.java:121) at hudson.remoting.UserRequest.perform(UserRequest.java:49) at hudson.remoting.Request$2.run(Request.java:324) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@772c6f4f }} After reconnecting two swarm clients (but not restarting Jenkins), one builds, one hangs, showing similar traces. From the hanging client: {{pool-1-thread-6 for channel "pool-1-thread-6 for channel" Id=20 Group=main RUNNABLE at com.sun.jna.Pointer.<clinit>(Pointer.java:41) at com.sun.jna.Structure.<clinit>(Structure.java:2078) at org.jvnet.hudson.Windows.monitor(Windows.java:42) at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:113) at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:103) at hudson.remoting.UserRequest.perform(UserRequest.java:121) at hudson.remoting.UserRequest.perform(UserRequest.java:49) at hudson.remoting.Request$2.run(Request.java:324) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) at java.lang.Thread.run(Unknown Source) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@2e201c5c}} It seems the FreeSwapMonitoring starts a new thread on the node every time it is started (we can start it manually using the "Refresh Status" Button in the BuildExecutor Status view). We see that some threads are actually closed, so it's not that all threads hang, but for every time we hit refresh status, we see a new instance of at ```org.jvnet.hudson.Windows.monitor``` (if we wait some minute between hitting that button)
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jenkins BBVA
            Path:
            src/main/java/hudson/remoting/UserRequest.java
            http://jenkins-ci.org/commit/remoting/8e0a8667a04dfd8d3509a08dc693de31cd5d2e77
            Log:
            JENKINS-19445 Force class load on UserRequest (#82)

            This is a workaround to force class load of any class. This helps prevent deadlock on windows nodes when using JNA and Subversion.
            Use property hudson.remoting.RemoteClassLoader.force to name the class to load. Ideally this forced load should happen earlier on the startup, however the classloader isn't available.

            • Update UserRequest as per code review comments

            Fixes based on code review of pull-request: https://github.com/jenkinsci/remoting/pull/82

            • Fix Logger.log() invocation

            Fix Logger.log() invocation: needs Object[], not vargs.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jenkins BBVA Path: src/main/java/hudson/remoting/UserRequest.java http://jenkins-ci.org/commit/remoting/8e0a8667a04dfd8d3509a08dc693de31cd5d2e77 Log: JENKINS-19445 Force class load on UserRequest (#82) Force class load on UserRequest, see JENKINS-19445 This is a workaround to force class load of any class. This helps prevent deadlock on windows nodes when using JNA and Subversion. Use property hudson.remoting.RemoteClassLoader.force to name the class to load. Ideally this forced load should happen earlier on the startup, however the classloader isn't available. Update UserRequest as per code review comments Fixes based on code review of pull-request: https://github.com/jenkinsci/remoting/pull/82 Fix Logger.log() invocation Fix Logger.log() invocation: needs Object[], not vargs.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            pom.xml
            http://jenkins-ci.org/commit/jenkins/409438f36dc80f20964fb16f8d88041e11ba4ed4
            Log:
            [JENKINS-19445, JENKINS-34213, JENKINS-34808, JENKINS-34121] Bump remoting to 2.59. (#2344)

            • [JENKINS-19445, JENKINS-34213, JENKINS-34808] Bump remoting to 2.58.

            Changes:

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: pom.xml http://jenkins-ci.org/commit/jenkins/409438f36dc80f20964fb16f8d88041e11ba4ed4 Log: [JENKINS-19445, JENKINS-34213, JENKINS-34808, JENKINS-34121] Bump remoting to 2.59. (#2344) [JENKINS-19445, JENKINS-34213, JENKINS-34808] Bump remoting to 2.58. Changes: JENKINS-34213 ( https://issues.jenkins-ci.org/browse/JENKINS-34213 ) - Ensure that the unexporter cleans up whatever it can each sweep ( https://github.com/jenkinsci/remoting/pull/81 ) JENKINS-19445 ( https://issues.jenkins-ci.org/browse/JENKINS-19445 ) Force class load on UserRequest in order to prevent deadlock on windows nodes when using JNA and Subversion ( https://github.com/jenkinsci/remoting/pull/81 ) JENKINS-34808 ( https://issues.jenkins-ci.org/browse/JENKINS-34808 ) - Allow user to adjust socket timeout ( https://github.com/jenkinsci/remoting/pull/68 ) JENKINS-34121 - Upgrade remoting to 2.59
            Hide
            ddaumiller Dorian Daumiller added a comment -

            Can we agree on having this in LTS?

            Show
            ddaumiller Dorian Daumiller added a comment - Can we agree on having this in LTS?
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Dorian Daumiller
            Jenkins 2 LTS - yes
            1.651.3 - under review: https://github.com/jenkinsci/jenkins/pull/2357

            Show
            oleg_nenashev Oleg Nenashev added a comment - Dorian Daumiller Jenkins 2 LTS - yes 1.651.3 - under review: https://github.com/jenkinsci/jenkins/pull/2357
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            pom.xml
            http://jenkins-ci.org/commit/jenkins/12e79963cca5122351943ee107f65c3ad91a2e25
            Log:
            [JENKINS-19445, JENKINS-34213, JENKINS-34808, JENKINS-34121] Bump remoting to 2.59. (#2344)

            • [JENKINS-19445, JENKINS-34213, JENKINS-34808] Bump remoting to 2.58.

            Changes:

            (cherry picked from commit 409438f36dc80f20964fb16f8d88041e11ba4ed4)

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: pom.xml http://jenkins-ci.org/commit/jenkins/12e79963cca5122351943ee107f65c3ad91a2e25 Log: [JENKINS-19445, JENKINS-34213, JENKINS-34808, JENKINS-34121] Bump remoting to 2.59. (#2344) [JENKINS-19445, JENKINS-34213, JENKINS-34808] Bump remoting to 2.58. Changes: JENKINS-34213 ( https://issues.jenkins-ci.org/browse/JENKINS-34213 ) - Ensure that the unexporter cleans up whatever it can each sweep ( https://github.com/jenkinsci/remoting/pull/81 ) JENKINS-19445 ( https://issues.jenkins-ci.org/browse/JENKINS-19445 ) Force class load on UserRequest in order to prevent deadlock on windows nodes when using JNA and Subversion ( https://github.com/jenkinsci/remoting/pull/81 ) JENKINS-34808 ( https://issues.jenkins-ci.org/browse/JENKINS-34808 ) - Allow user to adjust socket timeout ( https://github.com/jenkinsci/remoting/pull/68 ) JENKINS-34121 - Upgrade remoting to 2.59 (cherry picked from commit 409438f36dc80f20964fb16f8d88041e11ba4ed4)
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Released as Jenkins-2.4

            Show
            oleg_nenashev Oleg Nenashev added a comment - Released as Jenkins-2.4
            Hide
            drtaddei Dave Taddei added a comment - - edited

            We are showing this happening on the latest jenkins build (2.17) and latest slave jar (2.60) on a Windows 7 64-bit slave. Appears to be an intermittent fault but almost always occurs when an on-demand build is taking place.

            Show
            drtaddei Dave Taddei added a comment - - edited We are showing this happening on the latest jenkins build (2.17) and latest slave jar (2.60) on a Windows 7 64-bit slave. Appears to be an intermittent fault but almost always occurs when an on-demand build is taking place.
            Hide
            danielbeck Daniel Beck added a comment -

            Dave Taddei

            latest jenkins build … latest slave jar

            Please be more specific. What version?

            Show
            danielbeck Daniel Beck added a comment - Dave Taddei latest jenkins build … latest slave jar Please be more specific. What version?
            Hide
            bbva BBVA added a comment -

            Dave Taddei, can you confirm you are using the property hudson.remoting.RemoteClassLoader.force to force the load of com.sun.jna.Native. Also that you can see that the class is being loaded via the log of the slave.

            Show
            bbva BBVA added a comment - Dave Taddei, can you confirm you are using the property hudson.remoting.RemoteClassLoader.force to force the load of com.sun.jna.Native. Also that you can see that the class is being loaded via the log of the slave.
            Hide
            drtaddei Dave Taddei added a comment - - edited

            Hi BBVA, it does not appear so but I am unsure where to look. Should this be set in the JAVA_ARGS for the master, slave or both? I was also under the impression that had been added to release 2.47 so didn't set anything.

            Show
            drtaddei Dave Taddei added a comment - - edited Hi BBVA, it does not appear so but I am unsure where to look. Should this be set in the JAVA_ARGS for the master, slave or both? I was also under the impression that had been added to release 2.47 so didn't set anything.
            Hide
            bbva BBVA added a comment -

            Dave Taddei set JVM Options of the slave with: -Dhudson.remoting.RemoteClassLoader.force=com.sun.jna.Native

            Show
            bbva BBVA added a comment - Dave Taddei set JVM Options of the slave with: -Dhudson.remoting.RemoteClassLoader.force=com.sun.jna.Native
            Hide
            gregcovertsmith Greg Smith added a comment -

            We are using dynamic slaves, with the vcenter plugin. The vcenter plugin does not allow the specifications of any JVM options.

            At least, I do not believe it does. Could the above change be made permanent, or set as the default for all slaves some how?

            Show
            gregcovertsmith Greg Smith added a comment - We are using dynamic slaves, with the vcenter plugin. The vcenter plugin does not allow the specifications of any JVM options. At least, I do not believe it does. Could the above change be made permanent, or set as the default for all slaves some how?
            Hide
            jglick Jesse Glick added a comment -

            Greg Smith et al.—see the last comment I left in JENKINS-39179: I wrote a little plugin which might work around JNA deadlocks without needing individual configuration of static agents.

            Show
            jglick Jesse Glick added a comment - Greg Smith et al.—see the last comment I left in  JENKINS-39179 : I wrote a little plugin which might work around JNA deadlocks without needing individual configuration of static agents.

              People

              Assignee:
              schristou Steven Christou
              Reporter:
              odklizec Pavel Kudrys
              Votes:
              14 Vote for this issue
              Watchers:
              26 Start watching this issue

                Dates

                Created:
                Updated: