[JENKINS-41551] Deadlock between support core plugin and workflow-step-api-plugin

Type: Bug
Resolution: Fixed
Priority: Critical
Component/s: workflow-step-api-plugin
Labels:

Similar Issues:
Powered by SuggestiMate

Show

It seems like there is a deadlock between support core plugin and workflow-step-api-plugin.

Because of this issue, the UI becomes unresponsive which makes Jenkins admins to think the instance is down.

Found one Java-level deadlock:
=============================
"Computer.threadPoolForRemoting [#25]":
  waiting to lock monitor 0x00007fa024cfffd8 (object 0x0000000142c07458, a java.util.logging.StreamHandler),
  which is held by "Running CpsFlowExecution[Owner[SDOM/Builds/P1208/P1208__5.5.0/9:SDOM/Builds/P1208/P1208__5.5.0 #9]]"
"Running CpsFlowExecution[Owner[SDOM/Builds/P1208/P1208__5.5.0/9:SDOM/Builds/P1208/P1208__5.5.0 #9]]":
  waiting to lock monitor 0x00007f9ff0031568 (object 0x00000001407bbe98, a hudson.util.MaskingClassLoader),
  which is held by "Executor #4 for SI-Z0S4D : executing PlaceholderExecutable:ExecutorStepExecution.PlaceholderTask{runId=SDOM/Builds/P1208/P1208__5.5.0#9,context=CpsStepContext[71]:Owner[SDOM/Builds/P1208/P1208__5.5.0/9:SDOM/Builds/P1208/P1208__5.5.0 #9],cookie=6ef58131-2e13-4185-b552-c9be42e7949f}"
"Executor #4 for SI-Z0S4D : executing PlaceholderExecutable:ExecutorStepExecution.PlaceholderTask{runId=SDOM/Builds/P1208/P1208__5.5.0#9,context=CpsStepContext[71]:Owner[SDOM/Builds/P1208/P1208__5.5.0/9:SDOM/Builds/P1208/P1208__5.5.0 #9],cookie=6ef58131-2e13-4185-b552-c9be42e7949f}":
  waiting to lock monitor 0x00007fa024cfffd8 (object 0x0000000142c07458, a java.util.logging.StreamHandler),
  which is held by "Running CpsFlowExecution[Owner[SDOM/Builds/P1208/P1208__5.5.0/9:SDOM/Builds/P1208/P1208__5.5.0 #9]]"

Java stack information for the threads listed above:
===================================================
"Computer.threadPoolForRemoting [#25]":
	at java.util.logging.StreamHandler.publish(Unknown Source)
	- waiting to lock <0x0000000142c07458> (a java.util.logging.StreamHandler)
	at com.cloudbees.jenkins.support.impl.JenkinsLogs$LogFile.publish(JenkinsLogs.java:213)
	at com.cloudbees.jenkins.support.impl.JenkinsLogs$CustomHandler.publish(JenkinsLogs.java:252)
	at java.util.logging.Logger.log(Unknown Source)
	at java.util.logging.Logger.doLog(Unknown Source)
	at java.util.logging.Logger.log(Unknown Source)
	at java.util.logging.Logger.fine(Unknown Source)
	at hudson.remoting.Channel$1.handle(Channel.java:499)
	at hudson.remoting.AbstractByteArrayCommandTransport$1.handle(AbstractByteArrayCommandTransport.java:61)
	at org.jenkinsci.remoting.nio.NioChannelHub$2.run(NioChannelHub.java:594)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
"Running CpsFlowExecution[Owner[SDOM/Builds/P1208/P1208__5.5.0/9:SDOM/Builds/P1208/P1208__5.5.0 #9]]":
	at hudson.util.MaskingClassLoader.loadClass(MaskingClassLoader.java:70)
	- waiting to lock <0x00000001407bbe98> (a hudson.util.MaskingClassLoader)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	- locked <0x00000001409dda60> (a hudson.ClassicPluginStrategy$DependencyClassLoader)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	at jenkins.util.AntClassLoader.findBaseClass(AntClassLoader.java:1398)
	at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1075)
	- locked <0x00000001409dd9f0> (a hudson.ClassicPluginStrategy$AntClassLoader2)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	at org.jenkinsci.plugins.workflow.steps.StepExecution.getStatusBounded(StepExecution.java:132)
	at org.jenkinsci.plugins.workflow.steps.StepExecution.toString(StepExecution.java:107)
	at java.lang.String.valueOf(Unknown Source)
	at java.lang.StringBuilder.append(Unknown Source)
	at java.util.AbstractCollection.toString(Unknown Source)
	at java.util.Collections$UnmodifiableCollection.toString(Unknown Source)
	at java.text.MessageFormat.subformat(Unknown Source)
	at java.text.MessageFormat.format(Unknown Source)
	at java.text.Format.format(Unknown Source)
	at java.text.MessageFormat.format(Unknown Source)
	at java.util.logging.Formatter.formatMessage(Unknown Source)
	- locked <0x0000000142c07490> (a com.cloudbees.jenkins.support.SupportLogFormatter)
	at com.cloudbees.jenkins.support.SupportLogFormatter.format(SupportLogFormatter.java:101)
	at java.util.logging.StreamHandler.publish(Unknown Source)
	- locked <0x0000000142c07458> (a java.util.logging.StreamHandler)
	at com.cloudbees.jenkins.support.impl.JenkinsLogs$LogFile.publish(JenkinsLogs.java:213)
	at com.cloudbees.jenkins.support.impl.JenkinsLogs$CustomHandler.publish(JenkinsLogs.java:252)
	at java.util.logging.Logger.log(Unknown Source)
	at java.util.logging.Logger.doLog(Unknown Source)
	at java.util.logging.Logger.log(Unknown Source)
	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl$1.onSuccess(FlowExecutionList.java:177)
	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl$1.onSuccess(FlowExecutionList.java:174)
	at com.google.common.util.concurrent.Futures$6.run(Futures.java:975)
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
	at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:170)
	at com.google.common.util.concurrent.SettableFuture.set(SettableFuture.java:53)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$5.onSuccess(CpsFlowExecution.java:712)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$5.onSuccess(CpsFlowExecution.java:690)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4$1.run(CpsFlowExecution.java:627)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
"Executor #4 for SI-Z0S4D : executing PlaceholderExecutable:ExecutorStepExecution.PlaceholderTask{runId=SDOM/Builds/P1208/P1208__5.5.0#9,context=CpsStepContext[71]:Owner[SDOM/Builds/P1208/P1208__5.5.0/9:SDOM/Builds/P1208/P1208__5.5.0 #9],cookie=6ef58131-2e13-4185-b552-c9be42e7949f}":
	at java.util.logging.StreamHandler.publish(Unknown Source)
	- waiting to lock <0x0000000142c07458> (a java.util.logging.StreamHandler)
	at com.cloudbees.jenkins.support.impl.JenkinsLogs$LogFile.publish(JenkinsLogs.java:213)
	at com.cloudbees.jenkins.support.impl.JenkinsLogs$CustomHandler.publish(JenkinsLogs.java:252)
	at java.util.logging.Logger.log(Unknown Source)
	at java.util.logging.Logger.doLog(Unknown Source)
	at java.util.logging.Logger.log(Unknown Source)
	at org.eclipse.jetty.util.log.JavaUtilLog.debug(JavaUtilLog.java:111)
	at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:459)
	- locked <0x000000014001e980> (a org.eclipse.jetty.webapp.WebAppClassLoader)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	- locked <0x00000001407bbe98> (a hudson.util.MaskingClassLoader)
	at hudson.util.MaskingClassLoader.loadClass(MaskingClassLoader.java:75)
	- locked <0x00000001407bbe98> (a hudson.util.MaskingClassLoader)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	- locked <0x0000000140c70fc0> (a hudson.ClassicPluginStrategy$DependencyClassLoader)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	at jenkins.util.AntClassLoader.findBaseClass(AntClassLoader.java:1398)
	at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1075)
	- locked <0x0000000140c70f50> (a hudson.ClassicPluginStrategy$AntClassLoader2)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	at org.jenkinsci.plugins.workflow.support.actions.LogActionImpl.stream(LogActionImpl.java:88)
	at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:73)
	at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable.run(ExecutorStepExecution.java:513)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:410)

Found 1 deadlock.

I was also able to capture the same issue on a different time windows:

Found one Java-level deadlock:
=============================
"com.cloudbees.jenkins.plugin.metrics.views.Alerter thread":
  waiting to lock monitor 0x00007fbfc01452d8 (object 0x00000001450406d8, a java.util.logging.StreamHandler),
  which is held by "Running CpsFlowExecution[Owner[CI_MAINTENANCE/RESTART_MASTER/40:CI_MAINTENANCE/RESTART_MASTER #40]]"
"Running CpsFlowExecution[Owner[CI_MAINTENANCE/RESTART_MASTER/40:CI_MAINTENANCE/RESTART_MASTER #40]]":
  waiting to lock monitor 0x00007fbfc0143f98 (object 0x0000000142400060, a org.eclipse.jetty.webapp.WebAppClassLoader),
  which is held by "jenkins.util.Timer [#1]"
"jenkins.util.Timer [#1]":
  waiting to lock monitor 0x00007fbfc01452d8 (object 0x00000001450406d8, a java.util.logging.StreamHandler),
  which is held by "Running CpsFlowExecution[Owner[CI_MAINTENANCE/RESTART_MASTER/40:CI_MAINTENANCE/RESTART_MASTER #40]]"

Java stack information for the threads listed above:
===================================================
"com.cloudbees.jenkins.plugin.metrics.views.Alerter thread":
	at java.util.logging.StreamHandler.publish(Unknown Source)
	- waiting to lock <0x00000001450406d8> (a java.util.logging.StreamHandler)
	at com.cloudbees.jenkins.support.impl.JenkinsLogs$LogFile.publish(JenkinsLogs.java:213)
	at com.cloudbees.jenkins.support.impl.JenkinsLogs$CustomHandler.publish(JenkinsLogs.java:252)
	at java.util.logging.Logger.log(Unknown Source)
	at java.util.logging.Logger.doLog(Unknown Source)
	at java.util.logging.Logger.log(Unknown Source)
	at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:91)
	at java.lang.Thread.run(Unknown Source)
"Running CpsFlowExecution[Owner[CI_MAINTENANCE/RESTART_MASTER/40:CI_MAINTENANCE/RESTART_MASTER #40]]":
	at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:410)
	- waiting to lock <0x0000000142400060> (a org.eclipse.jetty.webapp.WebAppClassLoader)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	- locked <0x00000001436497b0> (a hudson.util.MaskingClassLoader)
	at hudson.util.MaskingClassLoader.loadClass(MaskingClassLoader.java:75)
	- locked <0x00000001436497b0> (a hudson.util.MaskingClassLoader)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	- locked <0x00000001462ae218> (a hudson.ClassicPluginStrategy$DependencyClassLoader)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	at jenkins.util.AntClassLoader.findBaseClass(AntClassLoader.java:1398)
	at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1075)
	- locked <0x00000001439c6988> (a hudson.ClassicPluginStrategy$AntClassLoader2)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	at org.jenkinsci.plugins.workflow.steps.StepExecution.getStatusBounded(StepExecution.java:142)
	at org.jenkinsci.plugins.workflow.steps.StepExecution.toString(StepExecution.java:107)
	at java.lang.String.valueOf(Unknown Source)
	at java.lang.StringBuilder.append(Unknown Source)
	at java.util.AbstractCollection.toString(Unknown Source)
	at java.util.Collections$UnmodifiableCollection.toString(Unknown Source)
	at java.text.MessageFormat.subformat(Unknown Source)
	at java.text.MessageFormat.format(Unknown Source)
	at java.text.Format.format(Unknown Source)
	at java.text.MessageFormat.format(Unknown Source)
	at java.util.logging.Formatter.formatMessage(Unknown Source)
	- locked <0x0000000145040710> (a com.cloudbees.jenkins.support.SupportLogFormatter)
	at com.cloudbees.jenkins.support.SupportLogFormatter.format(SupportLogFormatter.java:101)
	at java.util.logging.StreamHandler.publish(Unknown Source)
	- locked <0x00000001450406d8> (a java.util.logging.StreamHandler)
	at com.cloudbees.jenkins.support.impl.JenkinsLogs$LogFile.publish(JenkinsLogs.java:213)
	at com.cloudbees.jenkins.support.impl.JenkinsLogs$CustomHandler.publish(JenkinsLogs.java:252)
	at java.util.logging.Logger.log(Unknown Source)
	at java.util.logging.Logger.doLog(Unknown Source)
	at java.util.logging.Logger.log(Unknown Source)
	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl$1.onSuccess(FlowExecutionList.java:177)
	at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl$1.onSuccess(FlowExecutionList.java:174)
	at com.google.common.util.concurrent.Futures$6.run(Futures.java:975)
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
	at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:170)
	at com.google.common.util.concurrent.SettableFuture.set(SettableFuture.java:53)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$5.onSuccess(CpsFlowExecution.java:712)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$5.onSuccess(CpsFlowExecution.java:690)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4$1.run(CpsFlowExecution.java:627)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
"jenkins.util.Timer [#1]":
	at java.util.logging.StreamHandler.publish(Unknown Source)
	- waiting to lock <0x00000001450406d8> (a java.util.logging.StreamHandler)
	at com.cloudbees.jenkins.support.impl.JenkinsLogs$LogFile.publish(JenkinsLogs.java:213)
	at com.cloudbees.jenkins.support.impl.JenkinsLogs$CustomHandler.publish(JenkinsLogs.java:252)
	at java.util.logging.Logger.log(Unknown Source)
	at java.util.logging.Logger.doLog(Unknown Source)
	at java.util.logging.Logger.log(Unknown Source)
	at org.eclipse.jetty.util.log.JavaUtilLog.debug(JavaUtilLog.java:111)
	at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:459)
	- locked <0x0000000142400060> (a org.eclipse.jetty.webapp.WebAppClassLoader)
	at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:403)
	at hudson.util.Iterators.reverse(Iterators.java:168)
	at hudson.model.Queue.getItems(Queue.java:757)
	at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution.getStatus(ExecutorStepExecution.java:175)
	at org.jenkinsci.plugins.workflow.steps.StepExecution$1.call(StepExecution.java:134)
	at org.jenkinsci.plugins.workflow.steps.StepExecution$1.call(StepExecution.java:132)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

Found 1 deadlock.

blocks

JENKINS-39275 DurableTaskStep$Execution.getStatus can hang

Resolved

links to

CloudBees Internal OSS-1917

PR 18

Félix Belzunce Arcos created issue - 2017-01-30 07:12

Ryan Campbell made changes - 2017-01-30 17:02

Issue Type

Original: Improvement [ 4 ]

New: Bug [ 1 ]

Jesse Glick made changes - 2017-01-30 17:45

Link

New: This issue blocks ~~JENKINS-39275~~ [ ~~JENKINS-39275~~ ]

Jesse Glick made changes - 2017-01-30 17:49

Component/s		New: workflow-step-api-plugin [ 21718 ]
Component/s	Original: workflow-basic-steps-plugin [ 21712 ]

Jesse Glick made changes - 2017-01-30 17:49

Status

Original: Open [ 1 ]

New: In Progress [ 3 ]

Jesse Glick made changes - 2017-01-30 18:03

Remote Link

New: This issue links to "PR 18 (Web Link)" [ 15362 ]

Jesse Glick made changes - 2017-01-30 18:04

Status

Original: In Progress [ 3 ]

New: In Review [ 10005 ]

Jesse Glick made changes - 2017-01-30 18:04

Labels

Original: support-core

New: deadlock support-core

Carlos Rodríguez López made changes - 2017-01-30 18:51

Labels

Original: deadlock support-core

New: deadlock robustness support-core

Jesse Glick made changes - 2017-01-31 17:16

Resolution		New: Fixed [ 1 ]
Status	Original: In Review [ 10005 ]	New: Resolved [ 5 ]

CloudBees Inc. made changes - 2017-12-05 05:32

Remote Link

New: This issue links to "CloudBees Internal OSS-1917 (Web Link)" [ 18512 ]

Assignee:: Jesse Glick

Reporter:: Félix Belzunce Arcos

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Created:: 2017-01-30 07:12

Updated:: 2017-12-05 05:32

Resolved:: 2017-01-31 17:16

Jenkins

Details

Description

Attachments

Issue Links

Activity

People

Dates