[JENKINS-55189] Thread deadlock when spinning up hosts with ec2-plugin

Type: Bug
Resolution: Fixed
Priority: Blocker
Component/s: ec2-plugin
Labels:
None
Environment:

Hide
Jenkins version: 2.150.1 - yum install, running natively on host not in container

ec2-plugin version: 1.41

OS/kernel: Linux version 4.14.77-70.59.amzn1.x86_64 (mockbuild@gobi-build-64001) (gcc version 7.2.1 20170915 (Red Hat 7.2.1-2) (GCC)) #1 SMP Mon Nov 12 22:02:45 UTC 2018

JDK: OpenJDK Runtime Environment (build 1.8.0_191-b12)

Other Installed plugins:
Active Directory plugin (active-directory): 2.10
Hudson global-build-stats plugin (global-build-stats): 1.5
External Monitor Job Type Plugin (external-monitor-job): 1.7
Docker Commons Plugin (docker-commons): 1.13
Ant Plugin (ant): 1.9
HTML Publisher plugin (htmlpublisher): 1.17
SSH Credentials Plugin (ssh-credentials): 1.14
embeddable-build-status (embeddable-build-status): 1.9
Purge Job History Plugin (purge-job-history): 1.1
CloudBees Credentials Plugin (cloudbees-credentials): 3.3
Role-based Authorization Strategy (role-strategy): 2.9.0
Pipeline: Input Step (pipeline-input-step): 2.8
Git plugin (git): 3.9.1
Multijob plugin (jenkins-multijob-plugin): 1.31
Matrix Authorization Strategy Plugin (matrix-auth): 2.3
Token Macro Plugin (token-macro): 2.5
Windows Slaves Plugin (windows-slaves): 1.3.1
Javadoc Plugin (javadoc): 1.4
Publish Over SSH (publish-over-ssh): 1.20.1
Job Configuration History Plugin (jobConfigHistory): 2.19
Conditional BuildStep (conditional-buildstep): 1.3.6
JUnit Plugin (junit): 1.26.1
Pipeline: Job (workflow-job): 2.31
Jackson 2 API Plugin (jackson2-api): 2.9.7.1
Gradle Plugin (gradle): 1.29
built-on-column (built-on-column): 1.1
Pipeline: GitHub Groovy Libraries (pipeline-github-lib): 1.0
Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.12
Workspace Cleanup Plugin (ws-cleanup): 0.36
bouncycastle API Plugin (bouncycastle-api): 2.17
Parameterized Trigger plugin (parameterized-trigger): 2.35.2
Pipeline: Milestone Step (pipeline-milestone-step): 1.3.1
Apache HttpComponents Client 4.x API Plugin (apache-httpcomponents-client-4-api): 4.5.5-3.0
Purge Build Queue Plugin (purge-build-queue-plugin): 1.0
JSch dependency plugin (jsch): 0.1.54.2
JDK Tool Plugin (jdk-tool): 1.2
Environment Injector Plugin (envinject): 2.1.6
Groovy (groovy): 2.0
Ivy Plugin (ivy): 1.28
Pipeline: Stage Step (pipeline-stage-step): 2.3
Folders Plugin (cloudbees-folder): 6.7
Maven Integration plugin (maven-plugin): 3.2
Pipeline: Basic Steps (workflow-basic-steps): 2.13
Artifactory Plugin (artifactory): 2.16.2
Infrastructure plugin for Publish Over X (publish-over): 0.22
Pipeline: Stage View Plugin (pipeline-stage-view): 2.10
Dashboard View (dashboard-view): 2.10
JavaScript GUI Lib: Handlebars bundle plugin (handlebars): 1.1.1
Display URL API (display-url-api): 2.3.0
Plain Credentials Plugin (plain-credentials): 1.5
Prometheus metrics plugin (prometheus): 2.0.0
JavaScript GUI Lib: Moment.js bundle plugin (momentjs): 1.1.1
Git Parameter Plug-In (git-parameter): 0.9.6
Pipeline: Model API (pipeline-model-api): 1.3.4
Simple Theme Plugin (simple-theme-plugin): 0.5.1
JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached): 1.2.1
Pipeline: Build Step (pipeline-build-step): 2.7
Docker Pipeline (docker-workflow): 1.17
Credentials Plugin (credentials): 2.1.18
user build vars plugin (build-user-vars-plugin): 1.5
Command Agent Launcher Plugin (command-launcher): 1.2
Resource Disposer Plugin (resource-disposer): 0.12
Pipeline: Step API (workflow-step-api): 2.17
GitLab Plugin (gitlab-plugin): 1.5.11
SSH plugin (ssh): 2.6.1
JavaScript GUI Lib: ACE Editor bundle plugin (ace-editor): 1.1
Git client plugin (git-client): 2.7.4
Build Pipeline Plugin (build-pipeline-plugin): 1.5.8
LDAP Plugin (ldap): 1.20
Build Environment Plugin (build-environment): 1.6
GIT server Plugin (git-server): 1.7
Structs Plugin (structs): 1.17
Script Security Plugin (script-security): 1.49
Pipeline: Multibranch (workflow-multibranch): 2.20
Rebuilder (rebuild): 1.29
Authentication Tokens API Plugin (authentication-tokens): 1.3
Pipeline: Supporting APIs (workflow-support): 2.24
Pipeline: Declarative Agent API (pipeline-model-declarative-agent): 1.1.1
Pipeline: REST API Plugin (pipeline-rest-api): 2.10
Slack Notification Plugin (slack): 2.6
Matrix Project Plugin (matrix-project): 1.13
Trilead API Plugin (trilead-api): 1.0.1
Pipeline: Nodes and Processes (workflow-durable-task-step): 2.26
CloudBees AWS Credentials Plugin (aws-credentials): 1.24
Run Condition Plugin (run-condition): 1.2
Timestamper (timestamper): 1.8.10
GitHub Branch Source Plugin (github-branch-source): 2.4.1
Amazon EC2 plugin (ec2): 1.41
Pipeline: Groovy (workflow-cps): 2.61
Build Timeout (build-timeout): 1.19
Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 1.9
Webhook Step Plugin (webhook-step): 1.3
jQuery plugin (jquery): 1.12.4-0
Pipeline: Declarative Extension Points API (pipeline-model-extensions): 1.3.4
OWASP Markup Formatter Plugin (antisamy-markup-formatter): 1.5
Amazon Web Services SDK (aws-java-sdk): 1.11.457
Copy To Slave Plugin (copy-to-slave): 1.4.4
Pipeline: SCM Step (workflow-scm-step): 2.7
Throttle Concurrent Builds Plug-in (throttle-concurrents): 2.0.1
Global Slack Notifier Plugin (global-slack-notifier): 1.3
Build History Metrics plugin (build-history-metrics-plugin): 1.2
SCM API Plugin (scm-api): 2.3.0
Mailer Plugin (mailer): 1.22
build-metrics (build-metrics): 1.3
JIRA plugin (jira): 3.0.5
Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata): 1.3.4
Lockable Resources plugin (lockable-resources): 2.3
Pipeline: Declarative (pipeline-model-definition): 1.3.4
SSH Agent Plugin (ssh-agent): 1.17
Durable Task Plugin (durable-task): 1.28
Pipeline: API (workflow-api): 2.33
Credentials Binding Plugin (credentials-binding): 1.17
GitHub plugin (github): 1.29.3
Variant Plugin (variant): 1.1
Icon Shim Plugin (icon-shim): 2.0.3
EnvInject API Plugin (envinject-api): 1.5
SSH Slaves plugin (ssh-slaves): 1.29.1
GitHub API Plugin (github-api): 1.95
Mask Passwords Plugin (mask-passwords): 2.12.0
Pipeline (workflow-aggregator): 2.6
Node Iterator API Plugin (node-iterator-api): 1.5.0
Metrics Plugin (metrics): 4.0.2.2
Deployment Dashboard Plugin for Jenkins (ec2-deployment-dashboard): 1.0.10
Branch API Plugin (branch-api): 2.1.2
PAM Authentication plugin (pam-auth): 1.4
Mission Control Plugin (mission-control-view): 0.9.14
HTML5 Notifier Plugin (html5-notifier-plugin): 1.5
Config File Provider Plugin (config-file-provider): 3.4.1
Build With Parameters (build-with-parameters): 1.4
Email Extension Plugin (email-ext): 2.63
Git Changelog (git-changelog): 2.14

Show
Jenkins version: 2.150.1 - yum install, running natively on host not in container ec2-plugin version: 1.41 OS/kernel: Linux version 4.14.77-70.59.amzn1.x86_64 ( mockbuild@gobi-build-64001 ) (gcc version 7.2.1 20170915 (Red Hat 7.2.1-2) (GCC)) #1 SMP Mon Nov 12 22:02:45 UTC 2018 JDK: OpenJDK Runtime Environment (build 1.8.0_191-b12) Other Installed plugins: Active Directory plugin (active-directory): 2.10 Hudson global-build-stats plugin (global-build-stats): 1.5 External Monitor Job Type Plugin (external-monitor-job): 1.7 Docker Commons Plugin (docker-commons): 1.13 Ant Plugin (ant): 1.9 HTML Publisher plugin (htmlpublisher): 1.17 SSH Credentials Plugin (ssh-credentials): 1.14 embeddable-build-status (embeddable-build-status): 1.9 Purge Job History Plugin (purge-job-history): 1.1 CloudBees Credentials Plugin (cloudbees-credentials): 3.3 Role-based Authorization Strategy (role-strategy): 2.9.0 Pipeline: Input Step (pipeline-input-step): 2.8 Git plugin (git): 3.9.1 Multijob plugin (jenkins-multijob-plugin): 1.31 Matrix Authorization Strategy Plugin (matrix-auth): 2.3 Token Macro Plugin (token-macro): 2.5 Windows Slaves Plugin (windows-slaves): 1.3.1 Javadoc Plugin (javadoc): 1.4 Publish Over SSH (publish-over-ssh): 1.20.1 Job Configuration History Plugin (jobConfigHistory): 2.19 Conditional BuildStep (conditional-buildstep): 1.3.6 JUnit Plugin (junit): 1.26.1 Pipeline: Job (workflow-job): 2.31 Jackson 2 API Plugin (jackson2-api): 2.9.7.1 Gradle Plugin (gradle): 1.29 built-on-column (built-on-column): 1.1 Pipeline: GitHub Groovy Libraries (pipeline-github-lib): 1.0 Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.12 Workspace Cleanup Plugin (ws-cleanup): 0.36 bouncycastle API Plugin (bouncycastle-api): 2.17 Parameterized Trigger plugin (parameterized-trigger): 2.35.2 Pipeline: Milestone Step (pipeline-milestone-step): 1.3.1 Apache HttpComponents Client 4.x API Plugin (apache-httpcomponents-client-4-api): 4.5.5-3.0 Purge Build Queue Plugin (purge-build-queue-plugin): 1.0 JSch dependency plugin (jsch): 0.1.54.2 JDK Tool Plugin (jdk-tool): 1.2 Environment Injector Plugin (envinject): 2.1.6 Groovy (groovy): 2.0 Ivy Plugin (ivy): 1.28 Pipeline: Stage Step (pipeline-stage-step): 2.3 Folders Plugin (cloudbees-folder): 6.7 Maven Integration plugin (maven-plugin): 3.2 Pipeline: Basic Steps (workflow-basic-steps): 2.13 Artifactory Plugin (artifactory): 2.16.2 Infrastructure plugin for Publish Over X (publish-over): 0.22 Pipeline: Stage View Plugin (pipeline-stage-view): 2.10 Dashboard View (dashboard-view): 2.10 JavaScript GUI Lib: Handlebars bundle plugin (handlebars): 1.1.1 Display URL API (display-url-api): 2.3.0 Plain Credentials Plugin (plain-credentials): 1.5 Prometheus metrics plugin (prometheus): 2.0.0 JavaScript GUI Lib: Moment.js bundle plugin (momentjs): 1.1.1 Git Parameter Plug-In (git-parameter): 0.9.6 Pipeline: Model API (pipeline-model-api): 1.3.4 Simple Theme Plugin (simple-theme-plugin): 0.5.1 JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached): 1.2.1 Pipeline: Build Step (pipeline-build-step): 2.7 Docker Pipeline (docker-workflow): 1.17 Credentials Plugin (credentials): 2.1.18 user build vars plugin (build-user-vars-plugin): 1.5 Command Agent Launcher Plugin (command-launcher): 1.2 Resource Disposer Plugin (resource-disposer): 0.12 Pipeline: Step API (workflow-step-api): 2.17 GitLab Plugin (gitlab-plugin): 1.5.11 SSH plugin (ssh): 2.6.1 JavaScript GUI Lib: ACE Editor bundle plugin (ace-editor): 1.1 Git client plugin (git-client): 2.7.4 Build Pipeline Plugin (build-pipeline-plugin): 1.5.8 LDAP Plugin (ldap): 1.20 Build Environment Plugin (build-environment): 1.6 GIT server Plugin (git-server): 1.7 Structs Plugin (structs): 1.17 Script Security Plugin (script-security): 1.49 Pipeline: Multibranch (workflow-multibranch): 2.20 Rebuilder (rebuild): 1.29 Authentication Tokens API Plugin (authentication-tokens): 1.3 Pipeline: Supporting APIs (workflow-support): 2.24 Pipeline: Declarative Agent API (pipeline-model-declarative-agent): 1.1.1 Pipeline: REST API Plugin (pipeline-rest-api): 2.10 Slack Notification Plugin (slack): 2.6 Matrix Project Plugin (matrix-project): 1.13 Trilead API Plugin (trilead-api): 1.0.1 Pipeline: Nodes and Processes (workflow-durable-task-step): 2.26 CloudBees AWS Credentials Plugin (aws-credentials): 1.24 Run Condition Plugin (run-condition): 1.2 Timestamper (timestamper): 1.8.10 GitHub Branch Source Plugin (github-branch-source): 2.4.1 Amazon EC2 plugin (ec2): 1.41 Pipeline: Groovy (workflow-cps): 2.61 Build Timeout (build-timeout): 1.19 Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 1.9 Webhook Step Plugin (webhook-step): 1.3 jQuery plugin (jquery): 1.12.4-0 Pipeline: Declarative Extension Points API (pipeline-model-extensions): 1.3.4 OWASP Markup Formatter Plugin (antisamy-markup-formatter): 1.5 Amazon Web Services SDK (aws-java-sdk): 1.11.457 Copy To Slave Plugin (copy-to-slave): 1.4.4 Pipeline: SCM Step (workflow-scm-step): 2.7 Throttle Concurrent Builds Plug-in (throttle-concurrents): 2.0.1 Global Slack Notifier Plugin (global-slack-notifier): 1.3 Build History Metrics plugin (build-history-metrics-plugin): 1.2 SCM API Plugin (scm-api): 2.3.0 Mailer Plugin (mailer): 1.22 build-metrics (build-metrics): 1.3 JIRA plugin (jira): 3.0.5 Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata): 1.3.4 Lockable Resources plugin (lockable-resources): 2.3 Pipeline: Declarative (pipeline-model-definition): 1.3.4 SSH Agent Plugin (ssh-agent): 1.17 Durable Task Plugin (durable-task): 1.28 Pipeline: API (workflow-api): 2.33 Credentials Binding Plugin (credentials-binding): 1.17 GitHub plugin (github): 1.29.3 Variant Plugin (variant): 1.1 Icon Shim Plugin (icon-shim): 2.0.3 EnvInject API Plugin (envinject-api): 1.5 SSH Slaves plugin (ssh-slaves): 1.29.1 GitHub API Plugin (github-api): 1.95 Mask Passwords Plugin (mask-passwords): 2.12.0 Pipeline (workflow-aggregator): 2.6 Node Iterator API Plugin (node-iterator-api): 1.5.0 Metrics Plugin (metrics): 4.0.2.2 Deployment Dashboard Plugin for Jenkins (ec2-deployment-dashboard): 1.0.10 Branch API Plugin (branch-api): 2.1.2 PAM Authentication plugin (pam-auth): 1.4 Mission Control Plugin (mission-control-view): 0.9.14 HTML5 Notifier Plugin (html5-notifier-plugin): 1.5 Config File Provider Plugin (config-file-provider): 3.4.1 Build With Parameters (build-with-parameters): 1.4 Email Extension Plugin (email-ext): 2.63 Git Changelog (git-changelog): 2.14

Similar Issues:
Powered by SuggestiMate

Show

Every morning, our jenkins instance provisions a set of nodes with the ec2-plugin. The plugin is able to spin up a handful of nodes however, ends up locking and then the node provisioning job eventually times out. This thread lock also prevents the jenkins machine from successfully executing any future job, not just jobs associated with the ec2-plugin. The UI remains responsive.

The following stack trace is visible in our logs. I have listed our environment details above.

Dec 13, 2018 9:50:36 AM jenkins.metrics.api.Metrics$HealthChecker executeDec 13, 2018 9:50:36 AM jenkins.metrics.api.Metrics$HealthChecker executeWARNING: Some health checks are reporting as unhealthy: [thread-deadlock : [Handling POST /cloud/ec2-ephemeral-dev-instances/provision from 10.123.1.107 : qtp1785210046-147 locked on java.util.concurrent.locks.ReentrantLock$NonfairSync@e7ba29 (owned by jenkins.util.Timer [#3]): at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at hudson.model.Queue._withLock(Queue.java:1438) at hudson.model.Queue.withLock(Queue.java:1301) at jenkins.model.Nodes.updateNode(Nodes.java:193) at jenkins.model.Jenkins.updateNode(Jenkins.java:2095) at hudson.model.Node.save(Node.java:140) at hudson.util.PersistedList.onModified(PersistedList.java:173) at hudson.util.PersistedList.replaceBy(PersistedList.java:85) at hudson.model.Slave.<init>(Slave.java:198) at hudson.plugins.ec2.EC2AbstractSlave.<init>(EC2AbstractSlave.java:138) at hudson.plugins.ec2.EC2OndemandSlave.<init>(EC2OndemandSlave.java:49) at hudson.plugins.ec2.EC2OndemandSlave.<init>(EC2OndemandSlave.java:42) at hudson.plugins.ec2.SlaveTemplate.newOndemandSlave(SlaveTemplate.java:963) at hudson.plugins.ec2.SlaveTemplate.toSlaves(SlaveTemplate.java:660) at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:632) at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:463) at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:587) at hudson.plugins.ec2.EC2Cloud.doProvision(EC2Cloud.java:358) at java.lang.invoke.LambdaForm$DMH/1342443276.invokeVirtual_LL_L(LambdaForm$DMH) at java.lang.invoke.LambdaForm$BMH/1627769036.reinvoke(LambdaForm$BMH) at java.lang.invoke.LambdaForm$MH/182693765.invoker(LambdaForm$MH) at java.lang.invoke.LambdaForm$MH/1016551825.invokeExact_MT(LambdaForm$MH) at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145) at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:537) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:282) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:99) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.success(BasicHeaderProcessor.java:140) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:82) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:503) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305), jenkins.util.Timer [#3] locked on hudson.plugins.ec2.AmazonEC2Cloud@6490d7c0 (owned by Handling POST /cloud/ec2-ephemeral-dev-instances/provision from 10.123.1.107 : qtp1785210046-147): at hudson.plugins.ec2.EC2Cloud.connect(EC2Cloud.java:748) at hudson.plugins.ec2.CloudHelper.getInstance(CloudHelper.java:47) at hudson.plugins.ec2.CloudHelper.getInstanceWithRetry(CloudHelper.java:25) at hudson.plugins.ec2.EC2Computer.getState(EC2Computer.java:127) at hudson.plugins.ec2.EC2RetentionStrategy.internalCheck(EC2RetentionStrategy.java:112) at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:90) at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:48) at hudson.slaves.ComputerRetentionWork$1.run(ComputerRetentionWork.java:72) at hudson.model.Queue._withLock(Queue.java:1381) at hudson.model.Queue.withLock(Queue.java:1258) at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:63) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

relates to

JENKINS-54187 EC2 Plugin deadlock leaving Jenkins unresponsive

Closed

JENKINS-54298 Jenkins crashes with Deadlock with EC2 Plugin

Closed

dashvvv created issue - 2018-12-13 18:56

dashvvv made changes - 2018-12-19 17:50

Link

New: This issue relates to ~~JENKINS-54298~~ [ ~~JENKINS-54298~~ ]

dashvvv made changes - 2018-12-19 17:51

Link

New: This issue relates to ~~JENKINS-54187~~ [ ~~JENKINS-54187~~ ]

FABRIZIO MANFREDI added a comment - 2019-02-11 16:55

Solved in 1.42

FABRIZIO MANFREDI added a comment - 2019-02-11 16:55 Solved in 1.42

FABRIZIO MANFREDI made changes - 2019-02-11 16:56

Resolution		New: Fixed [ 1 ]
Status	Original: Open [ 1 ]	New: Resolved [ 5 ]

Assignee:: FABRIZIO MANFREDI

Reporter:: dashvvv

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Created:: 2018-12-13 18:56

Updated:: 2019-02-11 16:56

Resolved:: 2019-02-11 16:56

Jenkins

Details

Description

Attachments

Issue Links

Activity

Collapse comment: FABRIZIO MANFREDI added a comment - 2019-02-11 16:55

Expand comment: FABRIZIO MANFREDI added a comment - 2019-02-11 16:55

People

Dates