Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34712

"master is offline" preventing Pipeline from executing

      Our own Jenkins Pipeline projects seem to be getting stuck in this state of "master is offline" when attempting to run on our clusters which have zero executors assigned to the master node.

      It's unclear what, past a service restart, will clear this up

      Steps to reproduce:

      1. Start a pipeline job
      2. forcing the master to run out of storage
      3. shutdown master, clear up storage
      4. restart master, confirm it's up
      5. observe that it is still marked as offline for a long time. 30+ minutes

          [JENKINS-34712] "master is offline" preventing Pipeline from executing

          R. Tyler Croy created issue -
          Jesse Glick made changes -
          Component/s New: core [ 15593 ]
          Component/s Original: workflow-plugin [ 18820 ]
          Assignee Original: Jesse Glick [ jglick ]
          Labels New: workflow
          R. Tyler Croy made changes -
          Workflow Original: JNJira [ 170892 ] New: JNJira + In-Review [ 184080 ]
          Andrew Bayer made changes -
          Labels Original: workflow New: pipeline workflow
          Andrew Bayer made changes -
          Labels Original: pipeline workflow New: pipeline
          Ryan Campbell made changes -
          Description Original: Our own Jenkins Pipeline projects seem to be getting stuck in this state of "master is offline" when attempting to run on our clusters which have zero executors assigned to the master node.

          It's unclear what, past a service restart, will clear this up :(

          h3. Thread Dump

          {code}
          Channel reader thread: trusted-agent-1

          "Channel reader thread: trusted-agent-1" Id=87 Group=main WAITING on com.trilead.ssh2.channel.Channel@78493ab5
          at java.lang.Object.wait(Native Method)
          - waiting on com.trilead.ssh2.channel.Channel@78493ab5
          at java.lang.Object.wait(Object.java:503)
          at com.trilead.ssh2.channel.FifoBuffer.read(FifoBuffer.java:212)
          at com.trilead.ssh2.channel.Channel$Output.read(Channel.java:127)
          at com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:936)
          at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58)
          at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:79)
          at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82)
          at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72)
          at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103)
          at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
          at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
          at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

          Channel reader thread: trusted-agent-2

          "Channel reader thread: trusted-agent-2" Id=90 Group=main WAITING on com.trilead.ssh2.channel.Channel@67e91886
          at java.lang.Object.wait(Native Method)
          - waiting on com.trilead.ssh2.channel.Channel@67e91886
          at java.lang.Object.wait(Object.java:503)
          at com.trilead.ssh2.channel.FifoBuffer.read(FifoBuffer.java:212)
          at com.trilead.ssh2.channel.Channel$Output.read(Channel.java:127)
          at com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:936)
          at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58)
          at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:79)
          at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82)
          at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72)
          at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103)
          at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
          at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
          at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

          Computer.threadPoolForRemoting [#8]

          "Computer.threadPoolForRemoting [#8]" Id=132 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@4a44a3c2
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.SynchronousQueue$TransferStack@4a44a3c2
          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
          at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
          at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          Computer.threadPoolForRemoting [#9]

          "Computer.threadPoolForRemoting [#9]" Id=133 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@4a44a3c2
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.SynchronousQueue$TransferStack@4a44a3c2
          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
          at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
          at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          DestroyJavaVM

          "DestroyJavaVM" Id=17 Group=main RUNNABLE

          FilePath.localPool [#4]

          "FilePath.localPool [#4]" Id=137 Group=main RUNNABLE
          at sun.management.ThreadImpl.dumpThreads0(Native Method)
          at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446)
          at hudson.Functions.getThreadInfos(Functions.java:1200)
          at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:98)
          at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:95)
          at hudson.remoting.LocalChannel$1.call(LocalChannel.java:52)
          at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
          at java.util.concurrent.FutureTask.run(FutureTask.java:262)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          Number of locked synchronizers = 1
          - java.util.concurrent.ThreadPoolExecutor$Worker@4dd9587c

          Handling GET /threadDump from 172.31.29.27 : RequestHandlerThread[#4] Jenkins/threadDump.jelly

          "Handling GET /threadDump from 172.31.29.27 : RequestHandlerThread[#4] Jenkins/threadDump.jelly" Id=25 Group=main RUNNABLE
          at java.lang.System.identityHashCode(Native Method)
          at java.io.ObjectOutputStream$HandleTable.hash(ObjectOutputStream.java:2359)
          at java.io.ObjectOutputStream$HandleTable.lookup(ObjectOutputStream.java:2292)
          at java.io.ObjectOutputStream$ReplaceTable.lookup(ObjectOutputStream.java:2398)
          at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1112)
          at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1377)
          at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1173)
          at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
          at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:440)
          at java.lang.Throwable.writeObject(Throwable.java:985)
          - locked java.lang.Exception@3bcc975f
          at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:606)
          at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028)
          at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495)
          at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
          at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
          at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
          at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:440)
          at hudson.remoting.RemoteInvocationHandler.writeObject(RemoteInvocationHandler.java:290)
          at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:606)
          at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028)
          at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495)
          at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
          at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
          at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
          at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
          at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
          at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
          at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
          at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
          at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
          at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
          at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
          at hudson.remoting.Command.writeTo(Command.java:83)
          at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:43)
          at hudson.remoting.Channel.send(Channel.java:582)
          - locked hudson.remoting.Channel@56434d6
          at hudson.remoting.Request.callAsync(Request.java:205)
          at hudson.remoting.Channel.callAsync(Channel.java:807)
          at hudson.util.RemotingDiagnostics.getThreadDumpAsync(RemotingDiagnostics.java:92)
          at jenkins.model.Jenkins.getAllThreadDumps(Jenkins.java:3073)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:606)
          at org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258)
          at org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104)
          at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
          at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
          at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
          at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
          at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
          at org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134)
          at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
          at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
          at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
          at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
          at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
          at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
          at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
          at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
          at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
          at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
          at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
          at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
          at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
          at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
          at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
          at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
          at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
          at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
          at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
          at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
          at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
          at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
          at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
          at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:92)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
          at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
          at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
          at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
          at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
          at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
          at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
          at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
          at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
          at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
          at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
          at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
          at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
          at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
          at org.eclipse.jetty.server.Server.handle(Server.java:370)
          at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
          at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
          at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
          at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
          at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
          at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
          at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
          at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
          at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          Number of locked synchronizers = 1
          - java.util.concurrent.ThreadPoolExecutor$Worker@5f69743c

          HashSessionScavenger-0

          "HashSessionScavenger-0" Id=10 Group=main TIMED_WAITING on java.util.TaskQueue@3031917d
          at java.lang.Object.wait(Native Method)
          - waiting on java.util.TaskQueue@3031917d
          at java.util.TimerThread.mainLoop(Timer.java:552)
          at java.util.TimerThread.run(Timer.java:505)

          Jenkins cron thread

          "Jenkins cron thread" Id=18 Group=main WAITING on java.util.TaskQueue@5fdde855
          at java.lang.Object.wait(Native Method)
          - waiting on java.util.TaskQueue@5fdde855
          at java.lang.Object.wait(Object.java:503)
          at java.util.TimerThread.mainLoop(Timer.java:526)
          at java.util.TimerThread.run(Timer.java:505)

          Jenkins UDP 33848 monitoring thread

          "Jenkins UDP 33848 monitoring thread" Id=98 Group=main RUNNABLE (in native)
          at java.net.PlainDatagramSocketImpl.receive0(Native Method)
          - locked java.net.PlainDatagramSocketImpl@2b23ce36
          at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:146)
          - locked java.net.PlainDatagramSocketImpl@2b23ce36
          at java.net.DatagramSocket.receive(DatagramSocket.java:817)
          - locked java.net.DatagramPacket@3e0a57
          - locked java.net.MulticastSocket@6918d5e0
          at hudson.UDPBroadcastThread.run(UDPBroadcastThread.java:83)

          jenkins.util.Timer [#10]

          "jenkins.util.Timer [#10]" Id=47 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          jenkins.util.Timer [#1]

          "jenkins.util.Timer [#1]" Id=19 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          jenkins.util.Timer [#2]

          "jenkins.util.Timer [#2]" Id=27 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          jenkins.util.Timer [#3]

          "jenkins.util.Timer [#3]" Id=28 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          jenkins.util.Timer [#4]

          "jenkins.util.Timer [#4]" Id=29 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          jenkins.util.Timer [#5]

          "jenkins.util.Timer [#5]" Id=30 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          jenkins.util.Timer [#6]

          "jenkins.util.Timer [#6]" Id=31 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          jenkins.util.Timer [#7]

          "jenkins.util.Timer [#7]" Id=32 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          jenkins.util.Timer [#8]

          "jenkins.util.Timer [#8]" Id=33 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          jenkins.util.Timer [#9]

          "jenkins.util.Timer [#9]" Id=46 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f4a0c6f
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          JmDNS(ip-172-31-50-95.local.).State.Timer

          "JmDNS(ip-172-31-50-95.local.).State.Timer" Id=102 Group=main TIMED_WAITING on java.util.TaskQueue@1a0134b0
          at java.lang.Object.wait(Native Method)
          - waiting on java.util.TaskQueue@1a0134b0
          at java.util.TimerThread.mainLoop(Timer.java:552)
          at java.util.TimerThread.run(Timer.java:505)

          JmDNS(ip-172-31-50-95.local.).Timer

          "JmDNS(ip-172-31-50-95.local.).Timer" Id=101 Group=main TIMED_WAITING on java.util.TaskQueue@452a1c26
          at java.lang.Object.wait(Native Method)
          - waiting on java.util.TaskQueue@452a1c26
          at java.util.TimerThread.mainLoop(Timer.java:552)
          at java.util.TimerThread.run(Timer.java:505)

          LauncherControlThread[ControlPort=-1]

          "LauncherControlThread[ControlPort=-1]" Id=15 Group=main TIMED_WAITING
          at java.lang.Thread.sleep(Native Method)
          at winstone.Launcher.run(Launcher.java:247)
          at java.lang.Thread.run(Thread.java:745)

          NioSocketAcceptor-1

          "NioSocketAcceptor-1" Id=69 Group=main RUNNABLE (in native)
          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
          at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
          at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
          - locked sun.nio.ch.Util$2@5bcae695
          - locked java.util.Collections$UnmodifiableSet@7ee1d295
          - locked sun.nio.ch.EPollSelectorImpl@58b9691
          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
          at org.apache.mina.transport.socket.nio.NioSocketAcceptor.select(NioSocketAcceptor.java:238)
          at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:432)
          at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          Number of locked synchronizers = 1
          - java.util.concurrent.ThreadPoolExecutor$Worker@58b18d61

          Ping thread for channel hudson.remoting.Channel@25d663c8:trusted-agent-2

          "Ping thread for channel hudson.remoting.Channel@25d663c8:trusted-agent-2" Id=92 Group=main TIMED_WAITING
          at java.lang.Thread.sleep(Native Method)
          at hudson.remoting.PingThread.run(PingThread.java:90)

          Ping thread for channel hudson.remoting.Channel@56434d6:trusted-agent-1

          "Ping thread for channel hudson.remoting.Channel@56434d6:trusted-agent-1" Id=91 Group=main TIMED_WAITING
          at java.lang.Thread.sleep(Native Method)
          at hudson.remoting.PingThread.run(PingThread.java:90)

          RemoteInvocationHandler [#1]

          "RemoteInvocationHandler [#1]" Id=86 Group=main TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@fad7ec5
          at java.lang.Object.wait(Native Method)
          - waiting on java.lang.ref.ReferenceQueue$Lock@fad7ec5
          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
          at hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:415)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          at java.util.concurrent.FutureTask.run(FutureTask.java:262)
          at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
          at java.lang.Thread.run(Thread.java:745)

          RequestHandlerThread[#1] Selector0

          "RequestHandlerThread[#1] Selector0" Id=13 Group=main RUNNABLE (in native)
          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
          at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
          at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
          - locked sun.nio.ch.Util$2@1a352a35
          - locked java.util.Collections$UnmodifiableSet@18660b04
          - locked sun.nio.ch.EPollSelectorImpl@535d401f
          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
          at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
          at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
          at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          Number of locked synchronizers = 1
          - java.util.concurrent.ThreadPoolExecutor$Worker@710d806c

          RequestHandlerThread[#2] Acceptor0 SelectChannelConnector@0.0.0.0:8080

          "RequestHandlerThread[#2] Acceptor0 SelectChannelConnector@0.0.0.0:8080" Id=14 Group=main RUNNABLE (in native)
          at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
          at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
          - locked java.lang.Object@61c0559e
          at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
          at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
          at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          Number of locked synchronizers = 1
          - java.util.concurrent.ThreadPoolExecutor$Worker@23897e13

          RequestHandlerThread[#3]

          "RequestHandlerThread[#3]" Id=24 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5a8ff0f
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.SynchronousQueue$TransferStack@5a8ff0f
          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
          at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
          at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          RequestHandlerThread[#5]

          "RequestHandlerThread[#5]" Id=26 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5a8ff0f
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.SynchronousQueue$TransferStack@5a8ff0f
          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
          at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
          at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          RequestHandlerThread[#6]

          "RequestHandlerThread[#6]" Id=128 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@5a8ff0f
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.SynchronousQueue$TransferStack@5a8ff0f
          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
          at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
          at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          SCMTrigger [#3]

          "SCMTrigger [#3]" Id=129 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@605b3ba5
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.SynchronousQueue$TransferStack@605b3ba5
          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
          at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
          at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          SocketListener(ip-172-31-50-95.local.)

          "SocketListener(ip-172-31-50-95.local.)" Id=100 Group=main RUNNABLE (in native)
          at java.net.PlainDatagramSocketImpl.receive0(Native Method)
          - locked java.net.PlainDatagramSocketImpl@4f9e2643
          at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:146)
          - locked java.net.PlainDatagramSocketImpl@4f9e2643
          at java.net.DatagramSocket.receive(DatagramSocket.java:817)
          - locked java.net.DatagramPacket@59a0b187
          - locked java.net.MulticastSocket@44e74409
          at javax.jmdns.impl.SocketListener.run(SocketListener.java:41)

          TCP slave agent listener port=0

          "TCP slave agent listener port=0" Id=97 Group=main RUNNABLE (in native)
          at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
          at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
          - locked java.lang.Object@5d819b07
          at hudson.TcpSlaveAgentListener.run(TcpSlaveAgentListener.java:95)

          Thread-5

          "Thread-5" Id=44 Group=main RUNNABLE (in native)
          at java.net.SocketInputStream.socketRead0(Native Method)
          at java.net.SocketInputStream.read(SocketInputStream.java:152)
          at java.net.SocketInputStream.read(SocketInputStream.java:122)
          at com.trilead.ssh2.crypto.cipher.CipherInputStream.fill_buffer(CipherInputStream.java:41)
          at com.trilead.ssh2.crypto.cipher.CipherInputStream.internal_read(CipherInputStream.java:52)
          at com.trilead.ssh2.crypto.cipher.CipherInputStream.getBlock(CipherInputStream.java:79)
          at com.trilead.ssh2.crypto.cipher.CipherInputStream.read(CipherInputStream.java:108)
          at com.trilead.ssh2.transport.TransportConnection.receiveMessage(TransportConnection.java:232)
          at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:693)
          at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
          at java.lang.Thread.run(Thread.java:745)

          Thread-6

          "Thread-6" Id=45 Group=main RUNNABLE (in native)
          at java.net.SocketInputStream.socketRead0(Native Method)
          at java.net.SocketInputStream.read(SocketInputStream.java:152)
          at java.net.SocketInputStream.read(SocketInputStream.java:122)
          at com.trilead.ssh2.crypto.cipher.CipherInputStream.fill_buffer(CipherInputStream.java:41)
          at com.trilead.ssh2.crypto.cipher.CipherInputStream.internal_read(CipherInputStream.java:52)
          at com.trilead.ssh2.crypto.cipher.CipherInputStream.getBlock(CipherInputStream.java:79)
          at com.trilead.ssh2.crypto.cipher.CipherInputStream.read(CipherInputStream.java:108)
          at com.trilead.ssh2.transport.TransportConnection.receiveMessage(TransportConnection.java:232)
          at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:693)
          at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
          at java.lang.Thread.run(Thread.java:745)

          Finalizer

          "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@646be43
          at java.lang.Object.wait(Native Method)
          - waiting on java.lang.ref.ReferenceQueue$Lock@646be43
          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
          at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

          Java2D Disposer

          "Java2D Disposer" Id=11 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@f1cddfb
          at java.lang.Object.wait(Native Method)
          - waiting on java.lang.ref.ReferenceQueue$Lock@f1cddfb
          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
          at sun.java2d.Disposer.run(Disposer.java:145)
          at java.lang.Thread.run(Thread.java:745)

          Reference Handler

          "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@e7523ed
          at java.lang.Object.wait(Native Method)
          - waiting on java.lang.ref.Reference$Lock@e7523ed
          at java.lang.Object.wait(Object.java:503)
          at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)

          Signal Dispatcher

          "Signal Dispatcher" Id=4 Group=system RUNNABLE

          trusted-agent-1

          Channel reader thread: channel

          "Channel reader thread: channel" Id=10 Group=main RUNNABLE (in native)
          at java.io.FileInputStream.readBytes(Native Method)
          at java.io.FileInputStream.read(FileInputStream.java:272)
          at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
          at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
          - locked java.io.BufferedInputStream@35df85a1
          at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82)
          at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72)
          at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103)
          at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
          at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
          at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

          main

          "main" Id=1 Group=main TIMED_WAITING on hudson.remoting.Channel@a54a40c
          at java.lang.Object.wait(Native Method)
          - waiting on hudson.remoting.Channel@a54a40c
          at hudson.remoting.Channel.join(Channel.java:948)
          at hudson.remoting.Launcher.main(Launcher.java:534)
          at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:459)
          at hudson.remoting.Launcher.run(Launcher.java:239)
          at hudson.remoting.Launcher.main(Launcher.java:192)

          Ping thread for channel hudson.remoting.Channel@a54a40c:channel

          "Ping thread for channel hudson.remoting.Channel@a54a40c:channel" Id=11 Group=main TIMED_WAITING
          at java.lang.Thread.sleep(Native Method)
          at hudson.remoting.PingThread.run(PingThread.java:90)

          pool-1-thread-6 for channel

          "pool-1-thread-6 for channel" Id=19 Group=main RUNNABLE
          at sun.management.ThreadImpl.dumpThreads0(Native Method)
          at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446)
          at hudson.Functions.getThreadInfos(Functions.java:1200)
          at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:98)
          at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:95)
          at hudson.remoting.UserRequest.perform(UserRequest.java:120)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:326)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
          at java.util.concurrent.FutureTask.run(FutureTask.java:262)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          Number of locked synchronizers = 1
          - java.util.concurrent.ThreadPoolExecutor$Worker@33cc093f

          RemoteInvocationHandler [#1]

          "RemoteInvocationHandler [#1]" Id=9 Group=main TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@65be28a8
          at java.lang.Object.wait(Native Method)
          - waiting on java.lang.ref.ReferenceQueue$Lock@65be28a8
          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
          at hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:415)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          at java.util.concurrent.FutureTask.run(FutureTask.java:262)
          at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
          at java.lang.Thread.run(Thread.java:745)

          Finalizer

          "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@275f61c7
          at java.lang.Object.wait(Native Method)
          - waiting on java.lang.ref.ReferenceQueue$Lock@275f61c7
          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
          at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

          process reaper

          "process reaper" Id=20 Group=system TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@1af6a16b
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.SynchronousQueue$TransferStack@1af6a16b
          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
          at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
          at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          Reference Handler

          "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@670c6402
          at java.lang.Object.wait(Native Method)
          - waiting on java.lang.ref.Reference$Lock@670c6402
          at java.lang.Object.wait(Object.java:503)
          at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)

          Signal Dispatcher

          "Signal Dispatcher" Id=4 Group=system RUNNABLE

          trusted-agent-2

          Channel reader thread: channel

          "Channel reader thread: channel" Id=10 Group=main RUNNABLE (in native)
          at java.io.FileInputStream.readBytes(Native Method)
          at java.io.FileInputStream.read(FileInputStream.java:272)
          at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
          at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
          - locked java.io.BufferedInputStream@2c17b55
          at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82)
          at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72)
          at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103)
          at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
          at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
          at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

          main

          "main" Id=1 Group=main TIMED_WAITING on hudson.remoting.Channel@a54a40c
          at java.lang.Object.wait(Native Method)
          - waiting on hudson.remoting.Channel@a54a40c
          at hudson.remoting.Channel.join(Channel.java:948)
          at hudson.remoting.Launcher.main(Launcher.java:534)
          at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:459)
          at hudson.remoting.Launcher.run(Launcher.java:239)
          at hudson.remoting.Launcher.main(Launcher.java:192)

          Ping thread for channel hudson.remoting.Channel@a54a40c:channel

          "Ping thread for channel hudson.remoting.Channel@a54a40c:channel" Id=11 Group=main TIMED_WAITING
          at java.lang.Thread.sleep(Native Method)
          at hudson.remoting.PingThread.run(PingThread.java:90)

          pool-1-thread-6 for channel

          "pool-1-thread-6 for channel" Id=19 Group=main RUNNABLE
          at sun.management.ThreadImpl.dumpThreads0(Native Method)
          at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446)
          at hudson.Functions.getThreadInfos(Functions.java:1200)
          at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:98)
          at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:95)
          at hudson.remoting.UserRequest.perform(UserRequest.java:120)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:326)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
          at java.util.concurrent.FutureTask.run(FutureTask.java:262)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)

          Number of locked synchronizers = 1
          - java.util.concurrent.ThreadPoolExecutor$Worker@189a6c56

          RemoteInvocationHandler [#1]

          "RemoteInvocationHandler [#1]" Id=9 Group=main TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@6c880adc
          at java.lang.Object.wait(Native Method)
          - waiting on java.lang.ref.ReferenceQueue$Lock@6c880adc
          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
          at hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:415)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          at java.util.concurrent.FutureTask.run(FutureTask.java:262)
          at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
          at java.lang.Thread.run(Thread.java:745)

          Finalizer

          "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@5781292e
          at java.lang.Object.wait(Native Method)
          - waiting on java.lang.ref.ReferenceQueue$Lock@5781292e
          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
          at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

          Reference Handler

          "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@60b6b7e3
          at java.lang.Object.wait(Native Method)
          - waiting on java.lang.ref.Reference$Lock@60b6b7e3
          at java.lang.Object.wait(Object.java:503)
          at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)

          Signal Dispatcher

          "Signal Dispatcher" Id=4 Group=system RUNNABLE
          {code}
          New: Our own Jenkins Pipeline projects seem to be getting stuck in this state of "master is offline" when attempting to run on our clusters which have zero executors assigned to the master node.

          It's unclear what, past a service restart, will clear this up :(

          Steps to reproduce:

          # forcing the master to run out of storage
          # shutdown master, clear up storage
          # restart master, confirm it's up
          # observe that it is still marked as offline in CJOC for a long time. 30+ minutes
          Ryan Campbell made changes -
          Description Original: Our own Jenkins Pipeline projects seem to be getting stuck in this state of "master is offline" when attempting to run on our clusters which have zero executors assigned to the master node.

          It's unclear what, past a service restart, will clear this up :(

          Steps to reproduce:

          # forcing the master to run out of storage
          # shutdown master, clear up storage
          # restart master, confirm it's up
          # observe that it is still marked as offline in CJOC for a long time. 30+ minutes
          New: Our own Jenkins Pipeline projects seem to be getting stuck in this state of "master is offline" when attempting to run on our clusters which have zero executors assigned to the master node.

          It's unclear what, past a service restart, will clear this up :(

          Steps to reproduce:

          # forcing the master to run out of storage
          # shutdown master, clear up storage
          # restart master, confirm it's up
          # observe that it is still marked as offline for a long time. 30+ minutes
          Ryan Campbell made changes -
          Attachment New: threaddump.txt [ 35645 ]
          Ryan Campbell made changes -
          Labels Original: pipeline New: pipeline robustness
          Ryan Campbell made changes -
          Description Original: Our own Jenkins Pipeline projects seem to be getting stuck in this state of "master is offline" when attempting to run on our clusters which have zero executors assigned to the master node.

          It's unclear what, past a service restart, will clear this up :(

          Steps to reproduce:

          # forcing the master to run out of storage
          # shutdown master, clear up storage
          # restart master, confirm it's up
          # observe that it is still marked as offline for a long time. 30+ minutes
          New: Our own Jenkins Pipeline projects seem to be getting stuck in this state of "master is offline" when attempting to run on our clusters which have zero executors assigned to the master node.

          It's unclear what, past a service restart, will clear this up :(

          Steps to reproduce:

          # Start a pipeline job
          # forcing the master to run out of storage
          # shutdown master, clear up storage
          # restart master, confirm it's up
          # observe that it is still marked as offline for a long time. 30+ minutes
          Ryan Campbell made changes -
          Labels Original: pipeline robustness New: pipeline pipeline-hangs robustness

            dnusbaum Devin Nusbaum
            rtyler R. Tyler Croy
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: