[JENKINS-39275] DurableTaskStep$Execution.getStatus can hang

Type: Bug
Resolution: Fixed
Priority: Major
Component/s: workflow-durable-task-step-plugin
Labels:
- diagnostics

Similar Issues:
Powered by SuggestiMate

Show

There is some improvements to be made in the support core plugin (will create a different jira), but there is a case where the pipeline step executions toString method can hang forever if it is unable to obtain the durable task steps workspace:

"Running CpsFlowExecution[Owner[JOB_NAME:JOB_NAME#BUILD_NUMBER]] / waiting for hudson.remoting.Channel@111111111:NODE_NAME" #1 daemon prio=5 os_prio=0 tid=0x000000002e00d800 nid=0x4a8 in Object.wait() [0x000000002f5de000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at hudson.remoting.Request.call(Request.java:147)
    - locked <0x00000007b795ff98> (a hudson.remoting.UserRequest)
    at hudson.remoting.Channel.call(Channel.java:780)
    at hudson.FilePath.act(FilePath.java:979)
    at hudson.FilePath.act(FilePath.java:968)
    at hudson.FilePath.isDirectory(FilePath.java:1504)
    at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.getWorkspace(DurableTaskStep.java:108)
    at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.getStatus(DurableTaskStep.java:153)
    at org.jenkinsci.plugins.workflow.steps.StepExecution.toString(StepExecution.java:98)
    at java.lang.String.valueOf(String.java:2994)
    at java.lang.StringBuilder.append(StringBuilder.java:131)
    at java.util.AbstractCollection.toString(AbstractCollection.java:462)
    at java.util.Collections$UnmodifiableCollection.toString(Collections.java:1035)
    at java.text.MessageFormat.subformat(MessageFormat.java:1280)
    at java.text.MessageFormat.format(MessageFormat.java:865)
    at java.text.Format.format(Format.java:157)
    at java.text.MessageFormat.format(MessageFormat.java:841)
    at java.util.logging.Formatter.formatMessage(Formatter.java:138)
    - locked <0x00000006c394e168> (a com.cloudbees.jenkins.support.SupportLogFormatter)
    at com.cloudbees.jenkins.support.SupportLogFormatter.format(SupportLogFormatter.java:89)
    at java.util.logging.StreamHandler.publish(StreamHandler.java:211)
    - locked <0x00000006c394e130> (a java.util.logging.StreamHandler)
    at com.cloudbees.jenkins.support.impl.JenkinsLogs$LogFile.publish(JenkinsLogs.java:544)
    at com.cloudbees.jenkins.support.impl.JenkinsLogs$CustomHandler.publish(JenkinsLogs.java:583)
    at java.util.logging.Logger.log(Logger.java:738)
    at java.util.logging.Logger.doLog(Logger.java:765)
    at java.util.logging.Logger.log(Logger.java:830)
    at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl$1.onSuccess(FlowExecutionList.java:177)

I believe the issue is that the StepExecution.toString make a call to getStatus which if it is currently in a DurableTaskStep.getStatus method call it will try to get the workspace which could potentially hang if something has happened to the channel.

is blocked by

JENKINS-41551 Deadlock between support core plugin and workflow-step-api-plugin

Resolved

is blocking

JENKINS-31842 Allow steps to decorate thread dumps

Resolved

relates to

JENKINS-37720 Virtual thread dump hangs waiting for ProcessLiveness

Resolved

links to

workflow-cps PR 81

workflow-step-api PR 12

Steven Christou created issue - 2016-10-26 19:05

Steven Christou made changes - 2016-10-26 19:05

Component/s

Original: support-core-plugin [ 18146 ]

Steven Christou made changes - 2016-10-26 19:05

Assignee

Original: Steven Christou [ schristou ]

Steven Christou made changes - 2016-10-26 19:06

Description

Original: There is some improvements to be made in the support core plugin (will create a different ticket), but there is a case where the pipeline step executions toString method can hang forever if it is unable to obtain the durable task steps workspace:

{noformat}
"Running CpsFlowExecution[Owner[JOB_NAME:JOB_NAME#BUILD_NUMBER]] / waiting for hudson.remoting.Channel@111111111:NODE_NAME" #1 daemon prio=5 os_prio=0 tid=0x000000002e00d800 nid=0x4a8 in Object.wait() [0x000000002f5de000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at hudson.remoting.Request.call(Request.java:147)
    - locked <0x00000007b795ff98> (a hudson.remoting.UserRequest)
    at hudson.remoting.Channel.call(Channel.java:780)
    at hudson.FilePath.act(FilePath.java:979)
    at hudson.FilePath.act(FilePath.java:968)
    at hudson.FilePath.isDirectory(FilePath.java:1504)
    at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.getWorkspace(DurableTaskStep.java:108)
    at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.getStatus(DurableTaskStep.java:153)
    at org.jenkinsci.plugins.workflow.steps.StepExecution.toString(StepExecution.java:98)
    at java.lang.String.valueOf(String.java:2994)
    at java.lang.StringBuilder.append(StringBuilder.java:131)
    at java.util.AbstractCollection.toString(AbstractCollection.java:462)
    at java.util.Collections$UnmodifiableCollection.toString(Collections.java:1035)
    at java.text.MessageFormat.subformat(MessageFormat.java:1280)
    at java.text.MessageFormat.format(MessageFormat.java:865)
    at java.text.Format.format(Format.java:157)
    at java.text.MessageFormat.format(MessageFormat.java:841)
    at java.util.logging.Formatter.formatMessage(Formatter.java:138)
    - locked <0x00000006c394e168> (a com.cloudbees.jenkins.support.SupportLogFormatter)
    at com.cloudbees.jenkins.support.SupportLogFormatter.format(SupportLogFormatter.java:89)
    at java.util.logging.StreamHandler.publish(StreamHandler.java:211)
    - locked <0x00000006c394e130> (a java.util.logging.StreamHandler)
    at com.cloudbees.jenkins.support.impl.JenkinsLogs$LogFile.publish(JenkinsLogs.java:544)
    at com.cloudbees.jenkins.support.impl.JenkinsLogs$CustomHandler.publish(JenkinsLogs.java:583)
    at java.util.logging.Logger.log(Logger.java:738)
    at java.util.logging.Logger.doLog(Logger.java:765)
    at java.util.logging.Logger.log(Logger.java:830)
    at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl$1.onSuccess(FlowExecutionList.java:177)
{noformat}

I believe the issue is that the [StepExecution.toString|https://github.com/jenkinsci/workflow-step-api-plugin/blob/99a0704aa7b2fed962b4a434d122bcf8dc7158af/src/main/java/org/jenkinsci/plugins/workflow/steps/StepExecution.java#L98] make a call to {{getStatus}} which if it is currently in a [DurableTaskStep.getStatus|https://github.com/jenkinsci/workflow-durable-task-step-plugin/blob/8f7dac968c97e2a123befc2419f16e24234ca785/src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/DurableTaskStep.java#L220] method call it will try to get the workspace which could potentially hang _if_ something has happened to the channel.

New: There is some improvements to be made in the support core plugin (will create a different jira), but there is a case where the pipeline step executions toString method can hang forever if it is unable to obtain the durable task steps workspace:

{noformat}
"Running CpsFlowExecution[Owner[JOB_NAME:JOB_NAME#BUILD_NUMBER]] / waiting for hudson.remoting.Channel@111111111:NODE_NAME" #1 daemon prio=5 os_prio=0 tid=0x000000002e00d800 nid=0x4a8 in Object.wait() [0x000000002f5de000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at hudson.remoting.Request.call(Request.java:147)
    - locked <0x00000007b795ff98> (a hudson.remoting.UserRequest)
    at hudson.remoting.Channel.call(Channel.java:780)
    at hudson.FilePath.act(FilePath.java:979)
    at hudson.FilePath.act(FilePath.java:968)
    at hudson.FilePath.isDirectory(FilePath.java:1504)
    at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.getWorkspace(DurableTaskStep.java:108)
    at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.getStatus(DurableTaskStep.java:153)
    at org.jenkinsci.plugins.workflow.steps.StepExecution.toString(StepExecution.java:98)
    at java.lang.String.valueOf(String.java:2994)
    at java.lang.StringBuilder.append(StringBuilder.java:131)
    at java.util.AbstractCollection.toString(AbstractCollection.java:462)
    at java.util.Collections$UnmodifiableCollection.toString(Collections.java:1035)
    at java.text.MessageFormat.subformat(MessageFormat.java:1280)
    at java.text.MessageFormat.format(MessageFormat.java:865)
    at java.text.Format.format(Format.java:157)
    at java.text.MessageFormat.format(MessageFormat.java:841)
    at java.util.logging.Formatter.formatMessage(Formatter.java:138)
    - locked <0x00000006c394e168> (a com.cloudbees.jenkins.support.SupportLogFormatter)
    at com.cloudbees.jenkins.support.SupportLogFormatter.format(SupportLogFormatter.java:89)
    at java.util.logging.StreamHandler.publish(StreamHandler.java:211)
    - locked <0x00000006c394e130> (a java.util.logging.StreamHandler)
    at com.cloudbees.jenkins.support.impl.JenkinsLogs$LogFile.publish(JenkinsLogs.java:544)
    at com.cloudbees.jenkins.support.impl.JenkinsLogs$CustomHandler.publish(JenkinsLogs.java:583)
    at java.util.logging.Logger.log(Logger.java:738)
    at java.util.logging.Logger.doLog(Logger.java:765)
    at java.util.logging.Logger.log(Logger.java:830)
    at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl$1.onSuccess(FlowExecutionList.java:177)
{noformat}

I believe the issue is that the [StepExecution.toString|https://github.com/jenkinsci/workflow-step-api-plugin/blob/99a0704aa7b2fed962b4a434d122bcf8dc7158af/src/main/java/org/jenkinsci/plugins/workflow/steps/StepExecution.java#L98] make a call to {{getStatus}} which if it is currently in a [DurableTaskStep.getStatus|https://github.com/jenkinsci/workflow-durable-task-step-plugin/blob/8f7dac968c97e2a123befc2419f16e24234ca785/src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/DurableTaskStep.java#L220] method call it will try to get the workspace which could potentially hang _if_ something has happened to the channel.

Steven Christou made changes - 2016-10-26 22:12

Assignee

New: Jesse Glick [ jglick ]

Jesse Glick added a comment - 2016-10-26 23:01

getStatus is expected to return quickly and reliably, so nothing to be done in support-core.

Jesse Glick added a comment - 2016-10-26 23:01 getStatus is expected to return quickly and reliably, so nothing to be done in support-core .

Jesse Glick made changes - 2016-10-26 23:01

Component/s		New: workflow-durable-task-step-plugin [ 21715 ]
Component/s	Original: workflow-step-api-plugin [ 21718 ]
Assignee	Original: Jesse Glick [ jglick ]
Labels		New: diagnostics
Summary	Original: StepExecution tostring hangs	New: DurableTaskStep$Execution.getStatus can hang

Ryan Campbell made changes - 2016-10-31 15:46

Priority

Original: Minor [ 4 ]

New: Major [ 3 ]

Jesse Glick added a comment - 2016-10-31 17:49

Note that the stack trace quoted above should only happen if you have enabled a FINE logger.

Jesse Glick added a comment - 2016-10-31 17:49 Note that the stack trace quoted above should only happen if you have enabled a FINE logger.

Jesse Glick made changes - 2016-10-31 17:54

Status

Original: Open [ 1 ]

New: In Progress [ 3 ]

Assignee:: Unassigned

Reporter:: Steven Christou

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Created:: 2016-10-26 19:05

Updated:: 2017-01-30 17:45

Resolved:: 2016-10-31 20:14

Jenkins

Details

Description

Attachments

Issue Links

Activity

Collapse comment: Jesse Glick added a comment - 2016-10-26 23:01

Expand comment: Jesse Glick added a comment - 2016-10-26 23:01

Collapse comment: Jesse Glick added a comment - 2016-10-31 17:49

Expand comment: Jesse Glick added a comment - 2016-10-31 17:49

People

Dates