Details
-
Bug
-
Status: Open (View Workflow)
-
Blocker
-
Resolution: Unresolved
-
Jenkins core 2.107.3
workflow-aggregator: 2.5
workflow-api: 2.27
workflow-basic-steps: 2.7
workflow-cps: 2.53
workflow-cps-global-lib: 2.9
workflow-durable-task-step: 2.19
workflow-job: 2.21
workflow-multibranch: 2.19
workflow-scm-step: 2.6
workflow-step-api: 2.15
workflow-support: 2.18
pipeline-build-step: 2.7
pipeline-graph-analysis: 1.6
pipeline-input-step: 2.8
pipeline-milestone-step: 1.3.1
pipeline-model-api: 1.2.9
pipeline-model-declarative-agent: 1.1.1
pipeline-model-definition: 1.2.9
pipeline-model-extensions: 1.2.9
pipeline-rest-api: 2.10
pipeline-stage-step: 2.3
pipeline-stage-tags-metadata: 1.2.9
pipeline-stage-view: 2.10
pipeline-utility-steps: 2.1.0Jenkins core 2.107.3 workflow-aggregator: 2.5 workflow-api: 2.27 workflow-basic-steps: 2.7 workflow-cps: 2.53 workflow-cps-global-lib: 2.9 workflow-durable-task-step: 2.19 workflow-job: 2.21 workflow-multibranch: 2.19 workflow-scm-step: 2.6 workflow-step-api: 2.15 workflow-support: 2.18 pipeline-build-step: 2.7 pipeline-graph-analysis: 1.6 pipeline-input-step: 2.8 pipeline-milestone-step: 1.3.1 pipeline-model-api: 1.2.9 pipeline-model-declarative-agent: 1.1.1 pipeline-model-definition: 1.2.9 pipeline-model-extensions: 1.2.9 pipeline-rest-api: 2.10 pipeline-stage-step: 2.3 pipeline-stage-tags-metadata: 1.2.9 pipeline-stage-view: 2.10 pipeline-utility-steps: 2.1.0
Description
Three times in the last two weeks, we've had our Jenkins server stop responding to requests. When I check syslog, I see errors like this:
Jun 30 16:07:18 jenkins [jenkins]: Jun 30, 2018 4:07:18 PM org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0
Jun 30 16:07:18 jenkins [jenkins]: INFO: Running CpsFlowExecutionOwner[project/263:project #263] unresponsive for 5 sec
Jun 30 16:07:18 jenkins [jenkins]: Jun 30, 2018 4:07:18 PM org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0
Jun 30 16:07:18 jenkins [jenkins]: INFO: Running CpsFlowExecutionOwner[project/368:project #368] unresponsive for 5 sec
Jun 30 16:07:18 jenkins [jenkins]: Jun 30, 2018 4:07:18 PM org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0
Jun 30 16:07:18 jenkins [jenkins]: INFO: Running CpsFlowExecutionOwner[project/318:project #318] unresponsive for 5 sec
These seem to persist indefinitely and there don't seem to be any other relevant messages in the log. The Web UI just hangs until nginx times out.
The Java process will then refuse to stop when I try to restart the service and I have to kill it with kill -9.
Attachments
Activity
Philip Douglas
created issue -
Sam Van Oort
made changes -
| Field | Original Value | New Value |
|---|---|---|
| Component/s | durable-task-plugin [ 18622 ] | |
| Component/s | workflow-cps-plugin [ 21713 ] | |
| Component/s | workflow-durable-task-step-plugin [ 21715 ] | |
| Component/s | core [ 15593 ] |
Sam Van Oort
made changes -
| Labels | pipeline threads |
Nico Schmoigl
made changes -
| Assignee | Nico Schmoigl [ eagle_rainbow ] |
Nico Schmoigl
made changes -
| Assignee | Nico Schmoigl [ eagle_rainbow ] |
Nico Schmoigl
made changes -
| Attachment | 20180919-hangingjenkinsthreads-logs.txt [ 44595 ] |
John Wu
made changes -
| Attachment | jenkins-thread.dump.xml [ 44700 ] |
John Wu
made changes -
| Attachment | jenkins-thread.dump.xml [ 44700 ] |
John Wu
made changes -
| Attachment | jenkins-thread-dump.txt [ 44701 ] |
Nico Schmoigl
made changes -
| Attachment | 20181012-statebefore.txt [ 44769 ] |
| Labels | pipeline threads | pipeline threads triaged-2018-11 |
Ben Herfurth
made changes -
| Priority | Major [ 3 ] | Blocker [ 1 ] |
Damien Roche
made changes -
| Attachment | Screen Shot 2019-01-02 at 22.46.07.png [ 45566 ] |
Girish KG
made changes -
| Attachment | image-2021-11-30-23-13-44-342.png [ 56875 ] |
| Attachment | ganthore-threads.dump [ 57910 ] |