Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52362

Jenkins hangs due to "Running CpsFlowExecution unresponsive"

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Blocker
    • Resolution: Unresolved
    • Environment:
    • Similar Issues:

      Description

      Three times in the last two weeks, we've had our Jenkins server stop responding to requests. When I check syslog, I see errors like this:

      Jun 30 16:07:18 jenkins [jenkins]: Jun 30, 2018 4:07:18 PM org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0
      Jun 30 16:07:18 jenkins [jenkins]: INFO: Running CpsFlowExecutionOwner[project/263:project #263] unresponsive for 5 sec
      Jun 30 16:07:18 jenkins [jenkins]: Jun 30, 2018 4:07:18 PM org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0
      Jun 30 16:07:18 jenkins [jenkins]: INFO: Running CpsFlowExecutionOwner[project/368:project #368] unresponsive for 5 sec
      Jun 30 16:07:18 jenkins [jenkins]: Jun 30, 2018 4:07:18 PM org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0
      Jun 30 16:07:18 jenkins [jenkins]: INFO: Running CpsFlowExecutionOwner[project/318:project #318] unresponsive for 5 sec

      These seem to persist indefinitely and there don't seem to be any other relevant messages in the log. The Web UI just hangs until nginx times out.

      The Java process will then refuse to stop when I try to restart the service and I have to kill it with kill -9.

       

        Attachments

          Activity

          Hide
          girishkg Girish KG added a comment - - edited

          Unfortunately, this issue exists in 2.3.0.3 as well.
          We are now relying on restarting the Jenkins between 4-8 hours.

          Killing the threads will be quickly loaded with another deadlock, and this goes on until it is restarted.
          We have also observed that this deadlock will not appear for FREE-STYLE jobs.

          This appears only when we have PIPELINES running at around 25 to 40 concurrent JOBS.

          We have 72 Gigs of RAM, 32-CORE CPU, and fast lane disk at 15K RPM, but nothing is saving us other than just a restart.

          It's been 15-days and we are struggling to keep the Jenkins up for at least 12 continuous hours to facilitate long-run needed builds but hitting deadlock.

          So, extremely painful, badly hurt  and now we are migrating long-run jobs to individual Jenkins.

          These are the the documentation and guide I tried.

          2021-11-30 16:44:06.415+0000 [id=87538] INFO o.j.p.s.s.g.SandboxResolvingClassLoader#lambda$load$2: took 5,736ms to load/not load java.io.Serializable$groovy$lang$com$cloudbees$groovy$cps$env$HTTPS_PROXY from classLoader hudson.PluginManager$UberClassLoader
          2021-11-30 16:44:06.413+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecutionOwner[SEPG/VTY/VTY~vm-provisioning~git/216:SEPG/VTY/VTY~vm-provisioning~git #216]] loading java.io.Serializable$com$cloudbees$groovy$cps$hudson$model$Hudson unresponsive for 13 min
          2021-11-30 16:45:27.977+0000 [id=87816] INFO o.j.p.s.s.g.SandboxResolvingClassLoader#lambda$load$2: took 57,825ms to load/not load java/util/groovy/json/env$SEPG_OVERRIDE_NO_PROXY.groovy from classLoader hudson.PluginManager$UberClassLoader
          2021-11-30 16:44:59.173+0000 [id=87338] INFO o.j.p.s.s.g.SandboxResolvingClassLoader#lambda$load$2: took 5,805ms to load/not load java/io/Serializable$com$cloudbees$groovy$cps$hudson$model$Hudson.groovy from classLoader hudson.PluginManager$UberClassLoader
          2021-11-30 16:45:33.645+0000 [id=87801] INFO o.j.p.s.s.g.SandboxResolvingClassLoader#lambda$load$2: took 5,664ms to load/not load groovy.lang.GroovyObject$java$net$jenkins$env$JOB_NAME from classLoader hudson.PluginManager$UberClassLoader
          2021-11-30 16:45:27.980+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecutionOwner[SEPG/IDSolutions/IDSolutions~eurus-ui~ISOL-883~CI/1:SEPG/IDLSolutions/IDLSolutions~eurus-ui~ISOL-883~CI #1]] unresponsive for 12 min
          2021-11-30 16:45:39.405+0000 [id=87338] INFO o.j.p.s.s.g.SandboxResolvingClassLoader#lambda$load$2: took 11,423ms to load/not load groovy.lang.GroovyObject$com$cloudbees$groovy$cps$hudson$model$Hudson from classLoader hudson.PluginManager$UberClassLoader
          2021-11-30 16:45:39.405+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecutionOwner[SEPG/VTY/VTY~ftp~master~CI/1012:SEPG/VTY/VTY~ftp~master~CI #1012]] unresponsive for 13 min
          2021-11-30 16:45:39.405+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep 447: checking /tmp/ws on sepg-nix03-DH unresponsive for 13 min
          2021-11-30 16:45:39.405+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecutionOwner[SEPG/sepg/sepg~sample-dotnet-6~git/5:SEPG/sepg/sepg~sample-dotnet-6~git #5]] unresponsive for 14 min
          2021-11-30 16:46:02.449+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecutionOwner[SEPG/sepg/sepg~SEPGrelBasic~git/99:SEPG/sepg/sepg~SEPGrelBasic~git #99]] unresponsive for 19 min
          2021-11-30 16:46:02.449+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecutionOwner[SEPG/VTY/VTY~deploy-helm~git/651:SEPG/VTY/VTY~deploy-helm~git #651]] loading groovy/lang/GroovyObject$com$cloudbees$groovy$cps$hudson$model$Hudson.groovy unresponsive for 15 min
          2021-11-30 16:46:02.449+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecutionOwner[SEPG/VTY/VTY~vm-provisioning~git/216:SEPG/VTY/VTY~vm-provisioning~git #216]] unresponsive for 15 min
          2021-11-30 16:46:02.450+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecutionOwner[SEPG/IDLSolutions/IDLSolutions~eurus-ui~ISOL-883~CI/1:SEPG/IDLSolutions/IDLSolutions~eurus-ui~ISOL-883~CI #1]] unresponsive for 12 min
          2021-11-30 16:46:02.450+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecutionOwner[SEPG/VTY/VTY~ftp~master~CI/1012:SEPG/VTY/VTY~ftp~master~CI #1012]] unresponsive for 14 min
          2021-11-30 16:46:02.450+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep 447: checking /tmp/ws on sepg-nix03-DH unresponsive for 14 min

           

          JENKINS_JAVA_OPTIONS="-Xms1024m \
          -Xmx30720m \
          -XX:+UseG1GC \
          -XX:MaxMetaspaceSize=2048m \
          -XX:MaxGCPauseMillis=200 \
          -XX:+ParallelRefProcEnabled \
          -XX:+AlwaysPreTouch \
          -XX:+UseStringDeduplication \
          -XX:+DisableExplicitGC \
          -XX:+ExplicitGCInvokesConcurrent \
          -Djava.awt.headless=true \"

           

           

          Show
          girishkg Girish KG added a comment - - edited Unfortunately, this issue exists in 2.3.0.3 as well. We are now relying on restarting the Jenkins between 4-8 hours. Killing the threads will be quickly loaded with another deadlock, and this goes on until it is restarted. We have also observed that this deadlock will not appear for FREE-STYLE jobs. This appears only when we have PIPELINES running at around 25 to 40 concurrent JOBS. We have 72 Gigs of RAM, 32-CORE CPU, and fast lane disk at 15K RPM, but nothing is saving us other than just a restart. It's been 15-days and we are struggling to keep the Jenkins up for at least 12 continuous hours to facilitate long-run needed builds but hitting deadlock. So, extremely painful, badly hurt  and now we are migrating long-run jobs to individual Jenkins. These are the the documentation and guide I tried. https://www.jenkins.io/blog/2016/11/21/gc-tuning/ https://www.cloudbees.com/blog/joining-big-leagues-tuning-jenkins-gc-responsiveness-and-stability https://www.oracle.com/java/technologies/javase/vmoptions-jsp.html https://wiki.jenkins.io/JENKINS/Consideration-for-Large-Scale-Jenkins-Deployment.html https://www.jenkins.io/blog/2016/11/21/gc-tuning/ https://issues.jenkins.io/browse/JENKINS-29664 https://issues.jenkins.io/browse/JENKINS-59850 https://issues.jenkins.io/browse/JENKINS-62222 2021-11-30 16:44:06.415+0000 [id=87538] INFO o.j.p.s.s.g.SandboxResolvingClassLoader#lambda$load$2: took 5,736ms to load/not load java.io.Serializable$groovy$lang$com$cloudbees$groovy$cps$env$HTTPS_PROXY from classLoader hudson.PluginManager$UberClassLoader 2021-11-30 16:44:06.413+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecution Owner[SEPG/VTY/VTY~vm-provisioning~git/216:SEPG/VTY/VTY~vm-provisioning~git #216] ] loading java.io.Serializable$com$cloudbees$groovy$cps$hudson$model$Hudson unresponsive for 13 min 2021-11-30 16:45:27.977+0000 [id=87816] INFO o.j.p.s.s.g.SandboxResolvingClassLoader#lambda$load$2: took 57,825ms to load/not load java/util/groovy/json/env$SEPG_OVERRIDE_NO_PROXY.groovy from classLoader hudson.PluginManager$UberClassLoader 2021-11-30 16:44:59.173+0000 [id=87338] INFO o.j.p.s.s.g.SandboxResolvingClassLoader#lambda$load$2: took 5,805ms to load/not load java/io /Serializable$com$cloudbees$groovy$cps$hudson$model$Hudson.groovy from classLoader hudson.PluginManager$UberClassLoader 2021-11-30 16:45:33.645+0000 [id=87801] INFO o.j.p.s.s.g.SandboxResolvingClassLoader#lambda$load$2: took 5,664ms to load/not load groovy.lang.GroovyObject$java$net$jenkins$env$JOB_NAME from classLoader hudson.PluginManager$UberClassLoader 2021-11-30 16:45:27.980+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout #lambda$ping$0: Running CpsFlowExecution Owner[SEPG/IDSolutions/IDSolutions~eurus-ui~ISOL-883~CI/1:SEPG/IDLSolutions/IDLSolutions~eurus-ui~ISOL-883~CI #1] ] unresponsive for 12 min 2021-11-30 16:45:39.405+0000 [id=87338] INFO o.j.p.s.s.g.SandboxResolvingClassLoader#lambda$load$2: took 11,423ms to load/not load groovy.lang.GroovyObject$com$cloudbees$groovy$cps$hudson$model$Hudson from classLoader hudson.PluginManager$UberClassLoader 2021-11-30 16:45:39.405+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout #lambda$ping$0: Running CpsFlowExecution Owner[SEPG/VTY/VTY~ftp~master~CI/1012:SEPG/VTY/VTY~ftp~master~CI #1012] ] unresponsive for 13 min 2021-11-30 16:45:39.405+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout #lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep 447 : checking /tmp/ws on sepg-nix03-DH unresponsive for 13 min 2021-11-30 16:45:39.405+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecution Owner[SEPG/sepg/sepg~sample-dotnet-6~git/5:SEPG/sepg/sepg~sample-dotnet-6~git #5] ] unresponsive for 14 min 2021-11-30 16:46:02.449+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout #lambda$ping$0: Running CpsFlowExecution Owner[SEPG/sepg/sepg~SEPGrelBasic~git/99:SEPG/sepg/sepg~SEPGrelBasic~git #99] ] unresponsive for 19 min 2021-11-30 16:46:02.449+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout #lambda$ping$0: Running CpsFlowExecution Owner[SEPG/VTY/VTY~deploy-helm~git/651:SEPG/VTY/VTY~deploy-helm~git #651] ] loading groovy/lang/GroovyObject$com$cloudbees$groovy$cps$hudson$model$Hudson.groovy unresponsive for 15 min 2021-11-30 16:46:02.449+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecution Owner[SEPG/VTY/VTY~vm-provisioning~git/216:SEPG/VTY/VTY~vm-provisioning~git #216] ] unresponsive for 15 min 2021-11-30 16:46:02.450+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout #lambda$ping$0: Running CpsFlowExecution Owner[SEPG/IDLSolutions/IDLSolutions~eurus-ui~ISOL-883~CI/1:SEPG/IDLSolutions/IDLSolutions~eurus-ui~ISOL-883~CI #1] ] unresponsive for 12 min 2021-11-30 16:46:02.450+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout #lambda$ping$0: Running CpsFlowExecution Owner[SEPG/VTY/VTY~ftp~master~CI/1012:SEPG/VTY/VTY~ftp~master~CI #1012] ] unresponsive for 14 min 2021-11-30 16:46:02.450+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout #lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep 447 : checking /tmp/ws on sepg-nix03-DH unresponsive for 14 mi n   JENKINS_JAVA_OPTIONS="-Xms1024m \ -Xmx30720m \ -XX:+UseG1GC \ -XX:MaxMetaspaceSize=2048m \ -XX:MaxGCPauseMillis=200 \ -XX:+ParallelRefProcEnabled \ -XX:+AlwaysPreTouch \ -XX:+UseStringDeduplication \ -XX:+DisableExplicitGC \ -XX:+ExplicitGCInvokesConcurrent \ -Djava.awt.headless=true \"    
          Hide
          bpleines Branden added a comment -

          Also running into this issue. As others have noted restarting Jenkins resolves the issue short term, but after a couple days uptime the problem resurfaces. Adding the versions of our affected environment.

           

          Jenkins Core: 2.289.1

          workflow-aggregator: 2.6
          workflow-api: 2.45
          workflow-basic-steps: 2.23
          workflow-cps: 2.92
          workflow-cps-global-lib: 2.19
          workflow-durable-task-step: 2.39
          workflow-job: 2.41
          workflow-multibranch: 2.26
          workflow-scm-step: 2.13
          workflow-step-api: 2.23
          workflow-support: 3.8
          pipeline-build-step: 2.13
          pipeline-graph-analysis: 1.11
          pipeline-input-step: 2.12
          pipeline-milestone-step: 1.3.2
          pipeline-model-api: 1.8.5
          pipeline-model-declarative-agent: N/A
          pipeline-model-definition: 1.8.5
          pipeline-model-extensions: 1.8.5
          pipeline-rest-api: 2.19
          pipeline-stage-step: 2.5
          pipeline-stage-tags-metadata: 1.8.5
          pipeline-stage-view: 2.19
          pipeline-utility-steps: 2.8.0

           

          Show
          bpleines Branden added a comment - Also running into this issue. As others have noted restarting Jenkins resolves the issue short term, but after a couple days uptime the problem resurfaces. Adding the versions of our affected environment.   Jenkins Core: 2.289.1 workflow-aggregator: 2.6 workflow-api: 2.45 workflow-basic-steps: 2.23 workflow-cps: 2.92 workflow-cps-global-lib: 2.19 workflow-durable-task-step: 2.39 workflow-job: 2.41 workflow-multibranch: 2.26 workflow-scm-step: 2.13 workflow-step-api: 2.23 workflow-support: 3.8 pipeline-build-step: 2.13 pipeline-graph-analysis: 1.11 pipeline-input-step: 2.12 pipeline-milestone-step: 1.3.2 pipeline-model-api: 1.8.5 pipeline-model-declarative-agent: N/A pipeline-model-definition: 1.8.5 pipeline-model-extensions: 1.8.5 pipeline-rest-api: 2.19 pipeline-stage-step: 2.5 pipeline-stage-tags-metadata: 1.8.5 pipeline-stage-view: 2.19 pipeline-utility-steps: 2.8.0  
          Hide
          daxroc Damien Roche added a comment -

          Our timeouts are set quite high, I'll reduce those today. It looks like this is known issue in the Ec2 Plugin - JENKINS-53858 

          Show
          daxroc Damien Roche added a comment - Our timeouts are set quite high, I'll reduce those today. It looks like this is known issue in the Ec2 Plugin -  JENKINS-53858  
          Hide
          eagle_rainbow Nico Schmoigl added a comment -

          Damien Roche, we had similar issues before - and I fear that this is only loosely related to this issue here. In my case, we are not using EC2 cloud but the docker cloud plugin (YAD). Root cause for us was that the plugin did not have a connection timeout configured. If then a connection to the cloud manager fails, the thread is waiting eternally for an answer. For the sake of consistency, however, the thread aquired a lock, which then is never released... and all the blues started...

          That is why I would suggest you to have a look at your timeout values (I don't know whether they are configurable in the case of EC2) - and if applicable post them here for further cross-checking. If they are too high, you should fix that first.

          Show
          eagle_rainbow Nico Schmoigl added a comment - Damien Roche , we had similar issues before - and I fear that this is only loosely related to this issue here. In my case, we are not using EC2 cloud but the docker cloud plugin (YAD). Root cause for us was that the plugin did not have a connection timeout configured. If then a connection to the cloud manager fails, the thread is waiting eternally for an answer. For the sake of consistency, however, the thread aquired a lock, which then is never released... and all the blues started... That is why I would suggest you to have a look at your timeout values (I don't know whether they are configurable in the case of EC2) - and if applicable post them here for further cross-checking. If they are too high, you should fix that first.
          Hide
          daxroc Damien Roche added a comment -

          I took a thread dump during our encounter of this issue in the log today. We seem to encounter a Deadlock with Ec2 provisioning - possibly related to the changes introduced to spawn instances without delay. Have others enabled this feature?

          Our symptoms: Jenkins queue will grow (not substantially) - jobs don't seem to execute tho we have available executors. New jobs fail to even launch - nothing in logs either. All intensive purposes we don't get any lightweight executors. We see the following log entry multiple times for jobs that were in progress.

          Jan 02, 2019 10:02:36 PM org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0 INFO: Running CpsFlowExecution[Owner[<Omitted>/master #59]] unresponsive for 55 min
          

          hudson.plugins.ec2.AmazonEC2Cloud is blocking 3 threads.

           jenkins.util.Timer [#7] thread obtained hudson.plugins.ec2.AmazonEC2Cloud's lock & did not release it. Due to that 3 threads are BLOCKED as shown in the below graph. If threads are BLOCKED for prolonged period, application will become unresponsive. Examine 'jenkins.util.Timer [#7]' stacktrace to see why lock is not released.
          

          jenkins.util.Timer [#7] - priority:5 - threadId:0x00007f21a00f9800 - nativeId:0x1e3ac - nativeId (decimal):123820 - state:WAITING stackTrace: java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c06ca768> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at hudson.model.Queue._withLock(Queue.java:1438) at hudson.model.Queue.withLock(Queue.java:1301) at jenkins.model.Nodes.updateNode(Nodes.java:193) at jenkins.model.Jenkins.updateNode(Jenkins.java:2095) at hudson.model.Node.save(Node.java:140) at hudson.util.PersistedList.onModified(PersistedList.java:173) at hudson.util.PersistedList.replaceBy(PersistedList.java:85) at hudson.model.Slave.<init>(Slave.java:198) at hudson.plugins.ec2.EC2AbstractSlave.<init>(EC2AbstractSlave.java:138) at hudson.plugins.ec2.EC2OndemandSlave.<init>(EC2OndemandSlave.java:49) at hudson.plugins.ec2.EC2OndemandSlave.<init>(EC2OndemandSlave.java:42) at hudson.plugins.ec2.SlaveTemplate.newOndemandSlave(SlaveTemplate.java:963) at hudson.plugins.ec2.SlaveTemplate.toSlaves(SlaveTemplate.java:660) at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:632) at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:463) at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:587) - locked <0x00000000d5800000> (a hudson.plugins.ec2.AmazonEC2Cloud) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:602) at hudson.plugins.ec2.NoDelayProvisionerStrategy.apply(NoDelayProvisionerStrategy.java:48) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
          
          Handling POST /cloud/ec2-us-east-1/provision from <omitted> : <omitted>  Stack Trace is:  java.lang.Thread.State: BLOCKED (on object monitor) at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:568) - waiting to lock <0x00000000d5800000> (a hudson.plugins.ec2.AmazonEC2Cloud) at hudson.plugins.ec2.EC2Cloud.doProvision(EC2Cloud.java:358) at java.lang.invoke.LambdaForm$DMH/565760380.invokeVirtual_LL_L(LambdaForm$DMH) at java.lang.invoke.LambdaForm$BMH/222162113.reinvoke(LambdaForm$BMH) at java.lang.invoke.LambdaForm$MH/775458576.invoker(LambdaForm$MH) at java.lang.invoke.LambdaForm$MH/2100864823.invokeExact_MT(LambdaForm$MH) at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145) at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:537) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:282) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:64) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:56) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:503) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.lang.Thread.run(Thread.java:748)
          
          EC2 alive slaves monitor thread  Stack Trace is:  java.lang.Thread.State: BLOCKED (on object monitor) at hudson.plugins.ec2.EC2Cloud.connect(EC2Cloud.java:748) - waiting to lock <0x00000000d5800000> (a hudson.plugins.ec2.AmazonEC2Cloud) at hudson.plugins.ec2.CloudHelper.getInstance(CloudHelper.java:47) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:475) at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:443) at hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:43) at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:101) at java.lang.Thread.run(Thread.java:748)
          
          jenkins.util.Timer [#10] 
          Stack Trace is: 
          java.lang.Thread.State: BLOCKED (on object monitor)
          at hudson.plugins.ec2.EC2Cloud.connect(EC2Cloud.java:748)
          - waiting to lock <0x00000000d5800000> (a hudson.plugins.ec2.AmazonEC2Cloud)
          at hudson.plugins.ec2.CloudHelper.getInstance(CloudHelper.java:47)
          at hudson.plugins.ec2.CloudHelper.getInstanceWithRetry(CloudHelper.java:25)
          at hudson.plugins.ec2.EC2Computer.getState(EC2Computer.java:127)
          at hudson.plugins.ec2.EC2RetentionStrategy.internalCheck(EC2RetentionStrategy.java:112)
          at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:90)
          at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:48)
          at hudson.slaves.ComputerRetentionWork$1.run(ComputerRetentionWork.java:72)
          at hudson.model.Queue._withLock(Queue.java:1381)
          at hudson.model.Queue.withLock(Queue.java:1258)
          at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:63)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
          at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)
          Show
          daxroc Damien Roche added a comment - I took a thread dump during our encounter of this issue in the log today. We seem to encounter a Deadlock with Ec2 provisioning - possibly related to the changes introduced to spawn instances without delay. Have others enabled this feature? Our symptoms: Jenkins queue will grow (not substantially) - jobs don't seem to execute tho we have available executors. New jobs fail to even launch - nothing in logs either. All intensive purposes we don't get any lightweight executors. We see the following log entry multiple times for jobs that were in progress. Jan 02, 2019 10:02:36 PM org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0 INFO: Running CpsFlowExecution[Owner[<Omitted>/master #59]] unresponsive for 55 min hudson.plugins.ec2.AmazonEC2Cloud  is blocking 3 threads.  jenkins.util.Timer [#7] thread obtained hudson.plugins.ec2.AmazonEC2Cloud 's lock & did not release it. Due to that 3 threads are BLOCKED as shown in the below graph. If threads are BLOCKED for prolonged period, application will become unresponsive. Examine ' jenkins.util.Timer [#7]' stacktrace to see why lock is not released. jenkins.util.Timer [#7] - priority:5 - threadId:0x00007f21a00f9800 - nativeId:0x1e3ac - nativeId (decimal):123820 - state:WAITING stackTrace: java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c06ca768> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at hudson.model.Queue._withLock(Queue.java:1438) at hudson.model.Queue.withLock(Queue.java:1301) at jenkins.model.Nodes.updateNode(Nodes.java:193) at jenkins.model.Jenkins.updateNode(Jenkins.java:2095) at hudson.model.Node.save(Node.java:140) at hudson.util.PersistedList.onModified(PersistedList.java:173) at hudson.util.PersistedList.replaceBy(PersistedList.java:85) at hudson.model.Slave.<init>(Slave.java:198) at hudson.plugins.ec2.EC2AbstractSlave.<init>(EC2AbstractSlave.java:138) at hudson.plugins.ec2.EC2OndemandSlave.<init>(EC2OndemandSlave.java:49) at hudson.plugins.ec2.EC2OndemandSlave.<init>(EC2OndemandSlave.java:42) at hudson.plugins.ec2.SlaveTemplate.newOndemandSlave(SlaveTemplate.java:963) at hudson.plugins.ec2.SlaveTemplate.toSlaves(SlaveTemplate.java:660) at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:632) at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:463) at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:587) - locked <0x00000000d5800000> (a hudson.plugins.ec2.AmazonEC2Cloud) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:602) at hudson.plugins.ec2.NoDelayProvisionerStrategy.apply(NoDelayProvisionerStrategy.java:48) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Handling POST /cloud/ec2-us-east-1/provision from <omitted> : <omitted>  Stack Trace is:  java.lang.Thread.State: BLOCKED (on object monitor) at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:568) - waiting to lock <0x00000000d5800000> (a hudson.plugins.ec2.AmazonEC2Cloud) at hudson.plugins.ec2.EC2Cloud.doProvision(EC2Cloud.java:358) at java.lang.invoke.LambdaForm$DMH/565760380.invokeVirtual_LL_L(LambdaForm$DMH) at java.lang.invoke.LambdaForm$BMH/222162113.reinvoke(LambdaForm$BMH) at java.lang.invoke.LambdaForm$MH/775458576.invoker(LambdaForm$MH) at java.lang.invoke.LambdaForm$MH/2100864823.invokeExact_MT(LambdaForm$MH) at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145) at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:537) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:282) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:64) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:56) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:503) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.lang.Thread.run(Thread.java:748) EC2 alive slaves monitor thread  Stack Trace is:  java.lang.Thread.State: BLOCKED (on object monitor) at hudson.plugins.ec2.EC2Cloud.connect(EC2Cloud.java:748) - waiting to lock <0x00000000d5800000> (a hudson.plugins.ec2.AmazonEC2Cloud) at hudson.plugins.ec2.CloudHelper.getInstance(CloudHelper.java:47) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:475) at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:443) at hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:43) at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:101) at java.lang.Thread.run(Thread.java:748) jenkins.util.Timer [#10]  Stack Trace is:  java.lang.Thread.State: BLOCKED (on object monitor) at hudson.plugins.ec2.EC2Cloud.connect(EC2Cloud.java:748) - waiting to lock <0x00000000d5800000> (a hudson.plugins.ec2.AmazonEC2Cloud) at hudson.plugins.ec2.CloudHelper.getInstance(CloudHelper.java:47) at hudson.plugins.ec2.CloudHelper.getInstanceWithRetry(CloudHelper.java:25) at hudson.plugins.ec2.EC2Computer.getState(EC2Computer.java:127) at hudson.plugins.ec2.EC2RetentionStrategy.internalCheck(EC2RetentionStrategy.java:112) at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:90) at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:48) at hudson.slaves.ComputerRetentionWork$1.run(ComputerRetentionWork.java:72) at hudson.model.Queue._withLock(Queue.java:1381) at hudson.model.Queue.withLock(Queue.java:1258) at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:63) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            pdouglas Philip Douglas
            Votes:
            31 Vote for this issue
            Watchers:
            45 Start watching this issue

              Dates

              Created:
              Updated: