Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39275

DurableTaskStep$Execution.getStatus can hang

    XMLWordPrintable

Details

    Description

      There is some improvements to be made in the support core plugin (will create a different jira), but there is a case where the pipeline step executions toString method can hang forever if it is unable to obtain the durable task steps workspace:

      "Running CpsFlowExecution[Owner[JOB_NAME:JOB_NAME#BUILD_NUMBER]] / waiting for hudson.remoting.Channel@111111111:NODE_NAME" #1 daemon prio=5 os_prio=0 tid=0x000000002e00d800 nid=0x4a8 in Object.wait() [0x000000002f5de000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
          at java.lang.Object.wait(Native Method)
          at hudson.remoting.Request.call(Request.java:147)
          - locked <0x00000007b795ff98> (a hudson.remoting.UserRequest)
          at hudson.remoting.Channel.call(Channel.java:780)
          at hudson.FilePath.act(FilePath.java:979)
          at hudson.FilePath.act(FilePath.java:968)
          at hudson.FilePath.isDirectory(FilePath.java:1504)
          at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.getWorkspace(DurableTaskStep.java:108)
          at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.getStatus(DurableTaskStep.java:153)
          at org.jenkinsci.plugins.workflow.steps.StepExecution.toString(StepExecution.java:98)
          at java.lang.String.valueOf(String.java:2994)
          at java.lang.StringBuilder.append(StringBuilder.java:131)
          at java.util.AbstractCollection.toString(AbstractCollection.java:462)
          at java.util.Collections$UnmodifiableCollection.toString(Collections.java:1035)
          at java.text.MessageFormat.subformat(MessageFormat.java:1280)
          at java.text.MessageFormat.format(MessageFormat.java:865)
          at java.text.Format.format(Format.java:157)
          at java.text.MessageFormat.format(MessageFormat.java:841)
          at java.util.logging.Formatter.formatMessage(Formatter.java:138)
          - locked <0x00000006c394e168> (a com.cloudbees.jenkins.support.SupportLogFormatter)
          at com.cloudbees.jenkins.support.SupportLogFormatter.format(SupportLogFormatter.java:89)
          at java.util.logging.StreamHandler.publish(StreamHandler.java:211)
          - locked <0x00000006c394e130> (a java.util.logging.StreamHandler)
          at com.cloudbees.jenkins.support.impl.JenkinsLogs$LogFile.publish(JenkinsLogs.java:544)
          at com.cloudbees.jenkins.support.impl.JenkinsLogs$CustomHandler.publish(JenkinsLogs.java:583)
          at java.util.logging.Logger.log(Logger.java:738)
          at java.util.logging.Logger.doLog(Logger.java:765)
          at java.util.logging.Logger.log(Logger.java:830)
          at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl$1.onSuccess(FlowExecutionList.java:177)
      

      I believe the issue is that the StepExecution.toString make a call to getStatus which if it is currently in a DurableTaskStep.getStatus method call it will try to get the workspace which could potentially hang if something has happened to the channel.

      Attachments

        Issue Links

          Activity

            schristou Steven Christou created issue -
            schristou Steven Christou made changes -
            Field Original Value New Value
            Component/s support-core-plugin [ 18146 ]
            schristou Steven Christou made changes -
            Assignee Steven Christou [ schristou ]
            schristou Steven Christou made changes -
            Description There is some improvements to be made in the support core plugin (will create a different ticket), but there is a case where the pipeline step executions toString method can hang forever if it is unable to obtain the durable task steps workspace:

            {noformat}
            "Running CpsFlowExecution[Owner[JOB_NAME:JOB_NAME#BUILD_NUMBER]] / waiting for hudson.remoting.Channel@111111111:NODE_NAME" #1 daemon prio=5 os_prio=0 tid=0x000000002e00d800 nid=0x4a8 in Object.wait() [0x000000002f5de000]
               java.lang.Thread.State: TIMED_WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at hudson.remoting.Request.call(Request.java:147)
                - locked <0x00000007b795ff98> (a hudson.remoting.UserRequest)
                at hudson.remoting.Channel.call(Channel.java:780)
                at hudson.FilePath.act(FilePath.java:979)
                at hudson.FilePath.act(FilePath.java:968)
                at hudson.FilePath.isDirectory(FilePath.java:1504)
                at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.getWorkspace(DurableTaskStep.java:108)
                at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.getStatus(DurableTaskStep.java:153)
                at org.jenkinsci.plugins.workflow.steps.StepExecution.toString(StepExecution.java:98)
                at java.lang.String.valueOf(String.java:2994)
                at java.lang.StringBuilder.append(StringBuilder.java:131)
                at java.util.AbstractCollection.toString(AbstractCollection.java:462)
                at java.util.Collections$UnmodifiableCollection.toString(Collections.java:1035)
                at java.text.MessageFormat.subformat(MessageFormat.java:1280)
                at java.text.MessageFormat.format(MessageFormat.java:865)
                at java.text.Format.format(Format.java:157)
                at java.text.MessageFormat.format(MessageFormat.java:841)
                at java.util.logging.Formatter.formatMessage(Formatter.java:138)
                - locked <0x00000006c394e168> (a com.cloudbees.jenkins.support.SupportLogFormatter)
                at com.cloudbees.jenkins.support.SupportLogFormatter.format(SupportLogFormatter.java:89)
                at java.util.logging.StreamHandler.publish(StreamHandler.java:211)
                - locked <0x00000006c394e130> (a java.util.logging.StreamHandler)
                at com.cloudbees.jenkins.support.impl.JenkinsLogs$LogFile.publish(JenkinsLogs.java:544)
                at com.cloudbees.jenkins.support.impl.JenkinsLogs$CustomHandler.publish(JenkinsLogs.java:583)
                at java.util.logging.Logger.log(Logger.java:738)
                at java.util.logging.Logger.doLog(Logger.java:765)
                at java.util.logging.Logger.log(Logger.java:830)
                at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl$1.onSuccess(FlowExecutionList.java:177)
            {noformat}

            I believe the issue is that the [StepExecution.toString|https://github.com/jenkinsci/workflow-step-api-plugin/blob/99a0704aa7b2fed962b4a434d122bcf8dc7158af/src/main/java/org/jenkinsci/plugins/workflow/steps/StepExecution.java#L98] make a call to {{getStatus}} which if it is currently in a [DurableTaskStep.getStatus|https://github.com/jenkinsci/workflow-durable-task-step-plugin/blob/8f7dac968c97e2a123befc2419f16e24234ca785/src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/DurableTaskStep.java#L220] method call it will try to get the workspace which could potentially hang _if_ something has happened to the channel.
            There is some improvements to be made in the support core plugin (will create a different jira), but there is a case where the pipeline step executions toString method can hang forever if it is unable to obtain the durable task steps workspace:

            {noformat}
            "Running CpsFlowExecution[Owner[JOB_NAME:JOB_NAME#BUILD_NUMBER]] / waiting for hudson.remoting.Channel@111111111:NODE_NAME" #1 daemon prio=5 os_prio=0 tid=0x000000002e00d800 nid=0x4a8 in Object.wait() [0x000000002f5de000]
               java.lang.Thread.State: TIMED_WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at hudson.remoting.Request.call(Request.java:147)
                - locked <0x00000007b795ff98> (a hudson.remoting.UserRequest)
                at hudson.remoting.Channel.call(Channel.java:780)
                at hudson.FilePath.act(FilePath.java:979)
                at hudson.FilePath.act(FilePath.java:968)
                at hudson.FilePath.isDirectory(FilePath.java:1504)
                at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.getWorkspace(DurableTaskStep.java:108)
                at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.getStatus(DurableTaskStep.java:153)
                at org.jenkinsci.plugins.workflow.steps.StepExecution.toString(StepExecution.java:98)
                at java.lang.String.valueOf(String.java:2994)
                at java.lang.StringBuilder.append(StringBuilder.java:131)
                at java.util.AbstractCollection.toString(AbstractCollection.java:462)
                at java.util.Collections$UnmodifiableCollection.toString(Collections.java:1035)
                at java.text.MessageFormat.subformat(MessageFormat.java:1280)
                at java.text.MessageFormat.format(MessageFormat.java:865)
                at java.text.Format.format(Format.java:157)
                at java.text.MessageFormat.format(MessageFormat.java:841)
                at java.util.logging.Formatter.formatMessage(Formatter.java:138)
                - locked <0x00000006c394e168> (a com.cloudbees.jenkins.support.SupportLogFormatter)
                at com.cloudbees.jenkins.support.SupportLogFormatter.format(SupportLogFormatter.java:89)
                at java.util.logging.StreamHandler.publish(StreamHandler.java:211)
                - locked <0x00000006c394e130> (a java.util.logging.StreamHandler)
                at com.cloudbees.jenkins.support.impl.JenkinsLogs$LogFile.publish(JenkinsLogs.java:544)
                at com.cloudbees.jenkins.support.impl.JenkinsLogs$CustomHandler.publish(JenkinsLogs.java:583)
                at java.util.logging.Logger.log(Logger.java:738)
                at java.util.logging.Logger.doLog(Logger.java:765)
                at java.util.logging.Logger.log(Logger.java:830)
                at org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ItemListenerImpl$1.onSuccess(FlowExecutionList.java:177)
            {noformat}

            I believe the issue is that the [StepExecution.toString|https://github.com/jenkinsci/workflow-step-api-plugin/blob/99a0704aa7b2fed962b4a434d122bcf8dc7158af/src/main/java/org/jenkinsci/plugins/workflow/steps/StepExecution.java#L98] make a call to {{getStatus}} which if it is currently in a [DurableTaskStep.getStatus|https://github.com/jenkinsci/workflow-durable-task-step-plugin/blob/8f7dac968c97e2a123befc2419f16e24234ca785/src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/DurableTaskStep.java#L220] method call it will try to get the workspace which could potentially hang _if_ something has happened to the channel.
            schristou Steven Christou made changes -
            Assignee Jesse Glick [ jglick ]
            jglick Jesse Glick made changes -
            Component/s workflow-durable-task-step-plugin [ 21715 ]
            Component/s workflow-step-api-plugin [ 21718 ]
            Assignee Jesse Glick [ jglick ]
            Labels diagnostics
            Summary StepExecution tostring hangs DurableTaskStep$Execution.getStatus can hang
            recampbell Ryan Campbell made changes -
            Priority Minor [ 4 ] Major [ 3 ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "workflow-step-api PR 12 (Web Link)" [ 15002 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "workflow-cps PR 81 (Web Link)" [ 15003 ]
            jglick Jesse Glick made changes -
            Link This issue is blocking JENKINS-31842 [ JENKINS-31842 ]
            jglick Jesse Glick made changes -
            Resolution Fixed [ 1 ]
            Status In Review [ 10005 ] Resolved [ 5 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-37720 [ JENKINS-37720 ]
            jglick Jesse Glick made changes -
            Link This issue is blocked by JENKINS-41551 [ JENKINS-41551 ]

            People

              Unassigned Unassigned
              schristou Steven Christou
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: