-
Bug
-
Resolution: Fixed
-
Critical
-
Jenkins v2.144, ubuntu 14.04, ec2-plugin 1.38, 1.39, 1.40-SNAPSHOT (private-160d794a-masondonahue), 1.40.1
We seem to be running into an issue about once per day where multiple threads deadlock trying to access and update resources within the EC2 plugin.
We have several jobs that add substantial numbers of subjobs (~40) to the build queue, and they thus invoke the Pipeline step `ec2 cloud: 'AWS Cloud', template: 'Micro'` several times to preallocate enough EC2 nodes to run them all (though it looks like this behavior will no longer be necessary in ec2-plugin 1.40).
In addition, it seems that manually provisioning a node through the UI or manually deleting a node has a chance of deadlocking if it runs at the same time as the provisioning or unprovisioning process happens.
The following stacktrace shows the three threads running in 1.40-SNAPSHOT (master as of Friday afternoon).
Warning, the following threads are deadlocked : Handling POST /job/Selenium%20Tests/job/PAID-1256%252Fenable-paid-tests/build from 172.26.3.39 : qtp125130493-18700, jenkins.util.Timer [#3], jenkins.util.Timer [#6] "jenkins.util.Timer [#3]" daemon prio=5 WAITING sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) hudson.model.Queue._withLock(Queue.java:1437) hudson.model.Queue.withLock(Queue.java:1300) jenkins.model.Nodes.updateNode(Nodes.java:193) jenkins.model.Jenkins.updateNode(Jenkins.java:2077) hudson.model.Node.save(Node.java:140) hudson.util.PersistedList.onModified(PersistedList.java:173) hudson.util.PersistedList.replaceBy(PersistedList.java:85) hudson.model.Slave.<init>(Slave.java:198) hudson.plugins.ec2.EC2AbstractSlave.<init>(EC2AbstractSlave.java:134) hudson.plugins.ec2.EC2OndemandSlave.<init>(EC2OndemandSlave.java:49) hudson.plugins.ec2.EC2OndemandSlave.<init>(EC2OndemandSlave.java:42) hudson.plugins.ec2.SlaveTemplate.newOndemandSlave(SlaveTemplate.java:899) hudson.plugins.ec2.SlaveTemplate.toSlaves(SlaveTemplate.java:606) hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:578) hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:415) hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:542) hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:557) hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715) hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320) hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61) hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809) hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) "jenkins.util.Timer [#6]" daemon prio=5 BLOCKED hudson.plugins.ec2.EC2Cloud.connect(EC2Cloud.java:671) hudson.plugins.ec2.CloudHelper.getInstance(CloudHelper.java:47) hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:452) hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:420) hudson.plugins.ec2.EC2OndemandSlave.terminate(EC2OndemandSlave.java:68) hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:360) hudson.plugins.ec2.EC2RetentionStrategy.internalCheck(EC2RetentionStrategy.java:126) hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:88) hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:46) hudson.slaves.ComputerRetentionWork$1.run(ComputerRetentionWork.java:72) hudson.model.Queue._withLock(Queue.java:1380) hudson.model.Queue.withLock(Queue.java:1257) hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:63) hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) "Handling POST /job/Selenium%20Tests/job/PAID-1256%252Fenable-paid-tests/build from 172.26.3.39 : qtp125130493-18700" prio=5 WAITING sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) hudson.model.Queue.schedule2(Queue.java:587) hudson.model.Queue.schedule2(Queue.java:713) jenkins.model.ParameterizedJobMixIn.doBuild(ParameterizedJobMixIn.java:217) jenkins.model.ParameterizedJobMixIn$ParameterizedJob.doBuild(ParameterizedJobMixIn.java:408) java.lang.invoke.LambdaForm$DMH/227306521.invokeInterface_L4_V(LambdaForm$DMH) java.lang.invoke.LambdaForm$BMH/1196970080.reinvoke(LambdaForm$BMH) java.lang.invoke.LambdaForm$MH/457755914.invoker(LambdaForm$MH) java.lang.invoke.LambdaForm$MH/145876426.invokeExact_MT(LambdaForm$MH) java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343) org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184) org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117) org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:129) org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:734) org.kohsuke.stapler.Stapler.invoke(Stapler.java:864) org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:248) org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:734) org.kohsuke.stapler.Stapler.invoke(Stapler.java:864) org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:248) org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:734) org.kohsuke.stapler.Stapler.invoke(Stapler.java:864) org.kohsuke.stapler.Stapler.invoke(Stapler.java:668) org.kohsuke.stapler.Stapler.service(Stapler.java:238) javax.servlet.http.HttpServlet.service(HttpServlet.java:790) org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655) hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243) hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61) hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239) net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215) net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88) org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114) hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:99) org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
We upgraded to 1.40-SNAPSHOT after running into similar global deadlocks in 1.38 and 1.39, which I can attach stack dumps for, but since current master has a lot of reworking of the locking code, I'm not sure if they'll be useful.
- is duplicated by
-
-
- Closed
-
- is related to
-
-
- Closed
-
-
-
- Closed
-
[JENKINS-53858] Deadlock on EC2 resources
Mason Donahue
created issue -
FABRIZIO MANFREDI
made changes -
| Status | Original: Open [ 1 ] | New: In Progress [ 3 ] |
Mason Donahue
added a comment - Just an update with a stackdump from 1.40.1:
10/4/18 2:34 PM ===== Threads on dev-jenkins-master-useast1b-01@172.25.33.234 ===== Warning, the following threads are deadlocked : GitHubPushTrigger [#4], jenkins.util.Timer [#2], jenkins.util.Timer [#9] "GitHubPushTrigger [#4]" prio=5 WAITING sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) hudson.model.Queue.schedule2(Queue.java:587) jenkins.model.ParameterizedJobMixIn.scheduleBuild2(ParameterizedJobMixIn.java:156) jenkins.model.ParameterizedJobMixIn.scheduleBuild(ParameterizedJobMixIn.java:116) jenkins.model.ParameterizedJobMixIn.scheduleBuild(ParameterizedJobMixIn.java:105) com.cloudbees.jenkins.GitHubPushTrigger$1.run(GitHubPushTrigger.java:143) hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:119) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) "jenkins.util.Timer [#2]" daemon prio=5 BLOCKED hudson.plugins.ec2.EC2Cloud.connect(EC2Cloud.java:673) hudson.plugins.ec2.EC2AbstractSlave.stop(EC2AbstractSlave.java:314) hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:362) hudson.plugins.ec2.EC2RetentionStrategy.internalCheck(EC2RetentionStrategy.java:126) hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:88) hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:46) hudson.slaves.ComputerRetentionWork$1.run(ComputerRetentionWork.java:72) hudson.model.Queue._withLock(Queue.java:1380) hudson.model.Queue.withLock(Queue.java:1257) hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:63) hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) "jenkins.util.Timer [#9]" daemon prio=5 WAITING sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) hudson.model.Queue._withLock(Queue.java:1437) hudson.model.Queue.withLock(Queue.java:1300) jenkins.model.Nodes.updateNode(Nodes.java:193) jenkins.model.Jenkins.updateNode(Jenkins.java:2077) hudson.model.Node.save(Node.java:140) hudson.util.PersistedList.onModified(PersistedList.java:173) hudson.util.PersistedList.replaceBy(PersistedList.java:85) hudson.model.Slave.<init>(Slave.java:198) hudson.plugins.ec2.EC2AbstractSlave.<init>(EC2AbstractSlave.java:134) hudson.plugins.ec2.EC2OndemandSlave.<init>(EC2OndemandSlave.java:49) hudson.plugins.ec2.EC2OndemandSlave.<init>(EC2OndemandSlave.java:42) hudson.plugins.ec2.SlaveTemplate.newOndemandSlave(SlaveTemplate.java:918) hudson.plugins.ec2.SlaveTemplate.toSlaves(SlaveTemplate.java:624) hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:572) hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:432) hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:544) hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:559) hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715) hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320) hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61) hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809) hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748)
Mason Donahue
added a comment - Just an update with a stackdump from 1.40.1:
10/4/18 2:34 PM
===== Threads on dev-jenkins-master-useast1b-01@172.25.33.234 =====
Warning, the following threads are deadlocked : GitHubPushTrigger [#4], jenkins.util.Timer [#2], jenkins.util.Timer [#9]
"GitHubPushTrigger [#4]" prio=5 WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
hudson.model.Queue.schedule2(Queue.java:587)
jenkins.model.ParameterizedJobMixIn.scheduleBuild2(ParameterizedJobMixIn.java:156)
jenkins.model.ParameterizedJobMixIn.scheduleBuild(ParameterizedJobMixIn.java:116)
jenkins.model.ParameterizedJobMixIn.scheduleBuild(ParameterizedJobMixIn.java:105)
com.cloudbees.jenkins.GitHubPushTrigger$1.run(GitHubPushTrigger.java:143)
hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:119)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang. Thread .run( Thread .java:748)
"jenkins.util.Timer [#2]" daemon prio=5 BLOCKED
hudson.plugins.ec2.EC2Cloud.connect(EC2Cloud.java:673)
hudson.plugins.ec2.EC2AbstractSlave.stop(EC2AbstractSlave.java:314)
hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:362)
hudson.plugins.ec2.EC2RetentionStrategy.internalCheck(EC2RetentionStrategy.java:126)
hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:88)
hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:46)
hudson.slaves.ComputerRetentionWork$1.run(ComputerRetentionWork.java:72)
hudson.model.Queue._withLock(Queue.java:1380)
hudson.model.Queue.withLock(Queue.java:1257)
hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:63)
hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang. Thread .run( Thread .java:748)
"jenkins.util.Timer [#9]" daemon prio=5 WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
hudson.model.Queue._withLock(Queue.java:1437)
hudson.model.Queue.withLock(Queue.java:1300)
jenkins.model.Nodes.updateNode(Nodes.java:193)
jenkins.model.Jenkins.updateNode(Jenkins.java:2077)
hudson.model.Node.save(Node.java:140)
hudson.util.PersistedList.onModified(PersistedList.java:173)
hudson.util.PersistedList.replaceBy(PersistedList.java:85)
hudson.model.Slave.<init>(Slave.java:198)
hudson.plugins.ec2.EC2AbstractSlave.<init>(EC2AbstractSlave.java:134)
hudson.plugins.ec2.EC2OndemandSlave.<init>(EC2OndemandSlave.java:49)
hudson.plugins.ec2.EC2OndemandSlave.<init>(EC2OndemandSlave.java:42)
hudson.plugins.ec2.SlaveTemplate.newOndemandSlave(SlaveTemplate.java:918)
hudson.plugins.ec2.SlaveTemplate.toSlaves(SlaveTemplate.java:624)
hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:572)
hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:432)
hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:544)
hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:559)
hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715)
hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320)
hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61)
hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809)
hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang. Thread .run( Thread .java:748)
Mason Donahue
added a comment - And as for the actual objects being held:
Oct 04, 2018 4:34:47 PM jenkins.metrics.api.Metrics$HealthChecker execute
WARNING: Some health checks are reporting as unhealthy: [thread-deadlock : [jenkins.util.Timer [#8] locked on hudson.plugins.ec2.AmazonEC2Cloud@61f05bdb (owned by jenkins.util.Timer [#9]):
at hudson.plugins.ec2.EC2Cloud.connect(EC2Cloud.java:673)
at hudson.plugins.ec2.EC2AbstractSlave.stop(EC2AbstractSlave.java:314)
at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:362)
at hudson.plugins.ec2.EC2RetentionStrategy.internalCheck(EC2RetentionStrategy.java:126)
at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:88)
at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:46)
at hudson.slaves.ComputerRetentionWork$1.run(ComputerRetentionWork.java:72)
at hudson.model.Queue._withLock(Queue.java:1380)
at hudson.model.Queue.withLock(Queue.java:1257)
at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:63)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
, jenkins.util.Timer [#9] locked on java.util.concurrent.locks.ReentrantLock$NonfairSync@731b93cc (owned by jenkins.util.Timer [#8]):
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at hudson.model.Queue._withLock(Queue.java:1437)
at hudson.model.Queue.withLock(Queue.java:1300)
at jenkins.model.Nodes.updateNode(Nodes.java:193)
at jenkins.model.Jenkins.updateNode(Jenkins.java:2077)
at hudson.model.Node.save(Node.java:140)
at hudson.util.PersistedList.onModified(PersistedList.java:173)
at hudson.util.PersistedList.replaceBy(PersistedList.java:85)
at hudson.model.Slave.<init>(Slave.java:198)
at hudson.plugins.ec2.EC2AbstractSlave.<init>(EC2AbstractSlave.java:134)
at hudson.plugins.ec2.EC2OndemandSlave.<init>(EC2OndemandSlave.java:49)
at hudson.plugins.ec2.EC2OndemandSlave.<init>(EC2OndemandSlave.java:42)
at hudson.plugins.ec2.SlaveTemplate.newOndemandSlave(SlaveTemplate.java:918)
at hudson.plugins.ec2.SlaveTemplate.toSlaves(SlaveTemplate.java:624)
at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:596)
at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:432)
at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:544)
at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:559)
at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715)
at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320)
at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61)
at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
]]
Mason Donahue
added a comment - And as for the actual objects being held:
Oct 04, 2018 4:34:47 PM jenkins.metrics.api.Metrics$HealthChecker execute
WARNING: Some health checks are reporting as unhealthy: [thread-deadlock : [jenkins.util.Timer [#8] locked on hudson.plugins.ec2.AmazonEC2Cloud@61f05bdb (owned by jenkins.util.Timer [#9]):
at hudson.plugins.ec2.EC2Cloud.connect(EC2Cloud.java:673)
at hudson.plugins.ec2.EC2AbstractSlave.stop(EC2AbstractSlave.java:314)
at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:362)
at hudson.plugins.ec2.EC2RetentionStrategy.internalCheck(EC2RetentionStrategy.java:126)
at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:88)
at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:46)
at hudson.slaves.ComputerRetentionWork$1.run(ComputerRetentionWork.java:72)
at hudson.model.Queue._withLock(Queue.java:1380)
at hudson.model.Queue.withLock(Queue.java:1257)
at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:63)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang. Thread .run( Thread .java:748)
, jenkins.util.Timer [#9] locked on java.util.concurrent.locks.ReentrantLock$NonfairSync@731b93cc (owned by jenkins.util.Timer [#8]):
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at hudson.model.Queue._withLock(Queue.java:1437)
at hudson.model.Queue.withLock(Queue.java:1300)
at jenkins.model.Nodes.updateNode(Nodes.java:193)
at jenkins.model.Jenkins.updateNode(Jenkins.java:2077)
at hudson.model.Node.save(Node.java:140)
at hudson.util.PersistedList.onModified(PersistedList.java:173)
at hudson.util.PersistedList.replaceBy(PersistedList.java:85)
at hudson.model.Slave.<init>(Slave.java:198)
at hudson.plugins.ec2.EC2AbstractSlave.<init>(EC2AbstractSlave.java:134)
at hudson.plugins.ec2.EC2OndemandSlave.<init>(EC2OndemandSlave.java:49)
at hudson.plugins.ec2.EC2OndemandSlave.<init>(EC2OndemandSlave.java:42)
at hudson.plugins.ec2.SlaveTemplate.newOndemandSlave(SlaveTemplate.java:918)
at hudson.plugins.ec2.SlaveTemplate.toSlaves(SlaveTemplate.java:624)
at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:596)
at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:432)
at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:544)
at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:559)
at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715)
at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320)
at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61)
at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang. Thread .run( Thread .java:748)
]]
FABRIZIO MANFREDI
added a comment - You using multiple cloud configuration ? I found a potential race condition .
From the stack trace seems more a core deadlock
oleg_nenashev What do you think of the deadlock reported by the user ?
FABRIZIO MANFREDI
added a comment - You using multiple cloud configuration ? I found a potential race condition .
From the stack trace seems more a core deadlock
oleg_nenashev What do you think of the deadlock reported by the user ?
Mason Donahue
added a comment - We are currently using one cloud, but multiple configurations within that (same AMI, but machine types and labels differ).
Mason Donahue
added a comment - We are currently using one cloud, but multiple configurations within that (same AMI, but machine types and labels differ). Mason Donahue
made changes -
| Environment | Original: Jenkins v2.144, ubuntu 14.04, ec2-plugin 1.38, 1.39, 1.40-SNAPSHOT (private-160d794a-masondonahue) | New: Jenkins v2.144, ubuntu 14.04, ec2-plugin 1.38, 1.39, 1.40-SNAPSHOT (private-160d794a-masondonahue), 1.40.1 |
Perrin Morrow
added a comment - We started using our new Jenkins instance on Monday, which is running the very latest of everything. I was seeing this exact same deadlock several times an hour, and having to kill the threads via the JavaMelody monitoring UI when it happened.
It's happened only two or three times in the last two days, but I think that's because I drastically increased the idle timeouts for all the agents, so it is terminating and launching them far less often.
It looks like two timer tasks are acquiring the queue lock and EC2Cloud object monitor, but in different order?
- the idle check timer task locks the queue while checking for idle agents, then tries to synchronize on the EC2Cloud object when it needs to terminate one.
- the node provisioner timer task synchronizes on the EC2Cloud object while it launches an agent, but in the construction of hudson.util.Slave, it tries to acquire the queue lock.
Perrin Morrow
added a comment - We started using our new Jenkins instance on Monday, which is running the very latest of everything. I was seeing this exact same deadlock several times an hour, and having to kill the threads via the JavaMelody monitoring UI when it happened.
It's happened only two or three times in the last two days, but I think that's because I drastically increased the idle timeouts for all the agents, so it is terminating and launching them far less often.
It looks like two timer tasks are acquiring the queue lock and EC2Cloud object monitor, but in different order?
the idle check timer task locks the queue while checking for idle agents, then tries to synchronize on the EC2Cloud object when it needs to terminate one.
the node provisioner timer task synchronizes on the EC2Cloud object while it launches an agent, but in the construction of hudson.util.Slave, it tries to acquire the queue lock.
Seeing this issue also on ec2-plugin 1.40, with a single cloud configuration, on Jenkins 2.138.2.
"EC2 alive slaves monitor thread" daemon prio=5 BLOCKED"EC2 alive slaves monitor thread" daemon prio=5 BLOCKED hudson.plugins.ec2.EC2Cloud.connect(EC2Cloud.java:638) hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:279) hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:438) hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:406) hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:43) hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:101) java.lang.Thread.run(Thread.java:748)
"jenkins.util.Timer [#1]" daemon prio=5 BLOCKED hudson.plugins.ec2.EC2Cloud.connect(EC2Cloud.java:638) hudson.plugins.ec2.EC2AbstractSlave.stop(EC2AbstractSlave.java:300) hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:348) hudson.plugins.ec2.EC2RetentionStrategy.internalCheck(EC2RetentionStrategy.java:123) hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:85) hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:43) hudson.slaves.ComputerRetentionWork$1.run(ComputerRetentionWork.java:72) hudson.model.Queue._withLock(Queue.java:1380) hudson.model.Queue.withLock(Queue.java:1257) hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:63) hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748)
Yes, with the 1.40 should not needed anymore the pre warming the instances, the plugin is able to raise 40 nodes per minutes (if you don't reach the AWS api limits). Anyway I started to investigate the deadlock.