Blocker ref) https://wiki.jenkins.io/display/JENKINS/Remoting+issue
description:
we setup jenkins master and several agents. one of agents periodically deadlocked and cpu spiked. I found It because SynchronousCommandTransport$ReaderThread.run from remoting.jar. (ref: https://wiki.jenkins.io/display/JENKINS/Remoting+issue)
I added some screenshots. I really appreciate if you take a look and help us to find root cause.
agent server stats:
cpu usage:
tracking PID 15699: ( It was remoting.jar )
agentthread dump:
Thanks!
[JENKINS-61023] Jenkins agent threads deadlock due to remoting.jar
Kideok Kim
created issue -
Kideok Kim
made changes -
| Attachment | New: channel thread.png [ 50164 ] | |
| Attachment | New: cpu usage of remoting_jar.png [ 50165 ] | |
| Attachment | New: remoting_jar ps.png [ 50166 ] | |
| Attachment | New: stats.png [ 50167 ] |
Kideok Kim
made changes -
| Description |
Original:
*descriptiion:* we setup jenkins master and several slaves. one of slaves periodically deadlocked and cpu spiked. I found It because *remoting.jar*. Here is a screenshot. *slave server stats:* !image-2020-02-09-23-46-42-173.png|width=925,height=312! *cpu usage:* *!image-2020-02-09-23-48-03-663.png!* *tracking PID 15699: (* It was remoting.jar ) *!image-2020-02-09-23-48-19-132.png!* *slave thread dump:* !image-2020-02-09-23-47-26-493.png|width=849,height=291! I really appreciate if you take a look and help us to find root cause. Thanks! |
New:
*descriptiion:* we setup jenkins master and several slaves. one of slaves periodically deadlocked and cpu spiked. I found It because *remoting.jar*. Here is a screenshot. *slave server stats:* !stats.png|thumbnail! *cpu usage:* !cpu usage of remoting_jar.png|thumbnail! *tracking PID 15699: (* It was remoting.jar ) !remoting_jar ps.png|thumbnail! *slave thread dump:* !channel thread.png|thumbnail! I really appreciate if you take a look and help us to find root cause. Thanks! |
Kideok Kim
made changes -
| Description |
Original:
*descriptiion:* we setup jenkins master and several slaves. one of slaves periodically deadlocked and cpu spiked. I found It because *remoting.jar*. Here is a screenshot. *slave server stats:* !stats.png|thumbnail! *cpu usage:* !cpu usage of remoting_jar.png|thumbnail! *tracking PID 15699: (* It was remoting.jar ) !remoting_jar ps.png|thumbnail! *slave thread dump:* !channel thread.png|thumbnail! I really appreciate if you take a look and help us to find root cause. Thanks! |
New:
*description:* we setup jenkins master and several slaves. one of slaves periodically deadlocked and cpu spiked. I found It because *remoting.jar*. Here is a screenshot. *slave server stats:* !stats.png|thumbnail! *cpu usage:* !cpu usage of remoting_jar.png|thumbnail! *tracking PID 15699: (* It was remoting.jar ) !remoting_jar ps.png|thumbnail! *slave thread dump:* !channel thread.png|thumbnail! I really appreciate if you take a look and help us to find root cause. Thanks! |
Kideok Kim
made changes -
| Description |
Original:
*description:* we setup jenkins master and several slaves. one of slaves periodically deadlocked and cpu spiked. I found It because *remoting.jar*. Here is a screenshot. *slave server stats:* !stats.png|thumbnail! *cpu usage:* !cpu usage of remoting_jar.png|thumbnail! *tracking PID 15699: (* It was remoting.jar ) !remoting_jar ps.png|thumbnail! *slave thread dump:* !channel thread.png|thumbnail! I really appreciate if you take a look and help us to find root cause. Thanks! |
New:
ref) [https://wiki.jenkins.io/display/JENKINS/Remoting+issue] *description:* we setup jenkins master and several slaves. one of slaves periodically deadlocked and cpu spiked. I found It because *SynchronousCommandTransport$ReaderThread.run* from *remoting.jar*. (ref: [https://wiki.jenkins.io/display/JENKINS/Remoting+issue]) I added some screenshots. I really appreciate if you take a look and help us to find root cause. *slave server stats:* !stats.png|thumbnail! *cpu usage:* !cpu usage of remoting_jar.png|thumbnail! *tracking PID 15699: (* It was remoting.jar ) !remoting_jar ps.png|thumbnail! *slave thread dump:* !channel thread.png|thumbnail! Thanks! |
| Summary | Original: Jenkins slave threads deadlock due to remoting.jar | New: Jenkins agent threads deadlock due to remoting.jar |
| Description |
Original:
ref) [https://wiki.jenkins.io/display/JENKINS/Remoting+issue] *description:* we setup jenkins master and several slaves. one of slaves periodically deadlocked and cpu spiked. I found It because *SynchronousCommandTransport$ReaderThread.run* from *remoting.jar*. (ref: [https://wiki.jenkins.io/display/JENKINS/Remoting+issue]) I added some screenshots. I really appreciate if you take a look and help us to find root cause. *slave server stats:* !stats.png|thumbnail! *cpu usage:* !cpu usage of remoting_jar.png|thumbnail! *tracking PID 15699: (* It was remoting.jar ) !remoting_jar ps.png|thumbnail! *slave thread dump:* !channel thread.png|thumbnail! Thanks! |
New:
ref) [https://wiki.jenkins.io/display/JENKINS/Remoting+issue] *description:* we setup jenkins master and several agents. one of agents periodically deadlocked and cpu spiked. I found It because *SynchronousCommandTransport$ReaderThread.run* from *remoting.jar*. (ref: [https://wiki.jenkins.io/display/JENKINS/Remoting+issue]) I added some screenshots. I really appreciate if you take a look and help us to find root cause. *agent server stats:* !stats.png|thumbnail! *cpu usage:* !cpu usage of remoting_jar.png|thumbnail! *tracking PID 15699: (* It was remoting.jar ) !remoting_jar ps.png|thumbnail! *agentthread dump:* !channel thread.png|thumbnail! Thanks! |
| Assignee | Original: Jeff Thompson [ jthompson ] |
| Resolution | New: Cannot Reproduce [ 5 ] | |
| Status | Original: Open [ 1 ] | New: Closed [ 6 ] |
As described on the page you link to, you'll need to provide further information before anything can be done with this report. There is insufficient here for any action.
Things you will need to investigate include: versions used, types of agents used, operations of the different threads, what the agents are doing, etc.
Most of the time, issues like this are caused by something external to the Remoting library. These things can be system, network, or configuration. It might be a misbehaving plugin or interaction between multiple plugins.
Issues like this usually cannot be addressed without providing a reproducible scenario or sufficient diagnostics. Best of luck in troubleshooting and isolating the problem.