-
Bug
-
Resolution: Unresolved
-
Minor
-
Linux
I find a way to trigger a remoting problem using tcp fault injection with netem. I'm able to trigger this wait call at hudson.remoting.Request.call(Request.java:146):
while(response==null && !channel.isInClosed()) // I don't know exactly when this can happen, as pendingCalls are cleaned up by Channel, // but in production I've observed that in rare occasion it can block forever, even after a channel // is gone. So be defensive against that. wait(30*1000);
When this wait is triggered, the running build is stuck and consumes a executor. It loops over and over on the wait.
To reproduce, setup a SSH slave using the attached Dockerfile, and setup netem on the docker0 bridge like this:
tc qdisc add dev docker0 root netem tc qdisc change dev docker0 root netem corrupt 1
Testing requires to run the job one time before configuring netem, as netem settings are applied to all network streams, it could fail while downloading Maven dependencies. I just launched a Maven build of a example project to trigger the problem. It might be a Maven specific problem...
To remove netem settings, just run tc qdisc del dev docker0 root.
I've attached the Dockerfile, the command I used to launch it and a threaddump of a Jenkins stuck master.
- depends on
-
JENKINS-22252 Maven 3.2.1: IllegalAccessError on AbstractMapBasedMultimap
-
- Closed
-
- is related to
-
JENKINS-10840 Maven "module" shows as running after build is aborted.
-
- Open
-
- links to
[JENKINS-26947] Maven job stuck when slave channel get disconnected
Description |
Original:
I find a way to trigger a remoting problem using tcp fault injection with netem. I'm able to trigger this wait call at hudson.remoting.Request.call(Request.java:146): {{ while(response==null && !channel.isInClosed()) // I don't know exactly when this can happen, as pendingCalls are cleaned up by Channel, // but in production I've observed that in rare occasion it can block forever, even after a channel // is gone. So be defensive against that. wait(30*1000); }} When this wait is triggered, the running build is stuck and consumes a executor. It loops over and over on the wait. To reproduce, setup a SSH slave using the attached Dockerfile, and setup netem on the docker0 bridge like this: tc qdisc add dev docker0 root netem tc qdisc change dev docker0 root netem corrupt 1 Testing requires to run the job one time before configuring netem, as netem settings are applied to all network streams, it could fail while downloading Maven dependencies. I just launched a Maven build of a example project to trigger the problem. It might be a Maven specific problem... To remove netem settings, just run tc qdisc del dev docker0 root. I've attached the Dockerfile, the command I used to launch it and a threaddump of a Jenkins stuck master. |
New:
I find a way to trigger a remoting problem using tcp fault injection with netem. I'm able to trigger this wait call at hudson.remoting.Request.call(Request.java:146): {code} while(response==null && !channel.isInClosed()) // I don't know exactly when this can happen, as pendingCalls are cleaned up by Channel, // but in production I've observed that in rare occasion it can block forever, even after a channel // is gone. So be defensive against that. wait(30*1000); {code} When this wait is triggered, the running build is stuck and consumes a executor. It loops over and over on the wait. To reproduce, setup a SSH slave using the attached Dockerfile, and setup netem on the docker0 bridge like this: {code} tc qdisc add dev docker0 root netem tc qdisc change dev docker0 root netem corrupt 1 {code} Testing requires to run the job one time before configuring netem, as netem settings are applied to all network streams, it could fail while downloading Maven dependencies. I just launched a Maven build of a example project to trigger the problem. It might be a Maven specific problem... To remove netem settings, just run tc qdisc del dev docker0 root. I've attached the Dockerfile, the command I used to launch it and a threaddump of a Jenkins stuck master. |
Component/s | New: maven-plugin [ 16033 ] | |
Component/s | Original: remoting [ 15489 ] | |
Summary | Original: Unattended wait in the remoting code | New: Maven job stuck when slave channel get disconnected |
Remote Link | New: This issue links to "PR 39 (Web Link)" [ 12176 ] |
Assignee | New: Yoann Dubreuil [ ydubreuil ] |
Labels | New: remoting robustness slave |
Link | New: This issue is related to JENKINS-10840 [ JENKINS-10840 ] |
Resolution | New: Fixed [ 1 ] | |
Status | Original: Open [ 1 ] | New: Resolved [ 5 ] |
Link |
New:
This issue depends on |
Resolution | Original: Fixed [ 1 ] | |
Status | Original: Resolved [ 5 ] | New: Reopened [ 4 ] |
Workflow | Original: JNJira [ 161132 ] | New: JNJira + In-Review [ 186267 ] |