Jesse Glick added a comment - 2021-02-16 17:40 That, and jenkins.agents.WebSocketAgents I suppose.

Jesse Glick added a comment - 2021-02-19 19:18 Can you check whether adding -Djenkins.websocket.pingInterval=5 (or some other value less than the default of 30 ) to controller launch options helps?

Samuel Beaulieu added a comment - 2021-02-23 18:33 - edited Thank you. I have turned it on in our test instance, currently monitoring. Will put that in production as soon as I can get a maintenance window in. I notice there is nothing around triggering disconnects in the code for jenkins.websocket.pingInterval So from this perspective its more of a keep-alive to make sure nothing on the network layer tears the connection down.   I have more information from the strace logs I have been running in our test instance: This is the process space before we sleep for an hour and wait for the disconnect to happen (it does not always happen, but we capture when it does) *13:05:03* UID PID PPID C STIME TTY TIME CMD *13:05:03* root 1 0 31 19:04 pts/0 00:00:04 strace -ff -tt -o /home/jenkins/agent/strace.txt entrypoint *13:05:03* root 7 1 66 19:04 pts/0 00:00:09 java -javaagent:/ var /lib/jenkins/jSSLKeyLog.jar=/home/jenkins/agent/FOO.txt -Dorg.jenkinsci.remoting.engine.JnlpProtocol3.disabled= true -cp /usr/share/jenkins/slave.jar hudson.remoting.jnlp.Main -headless -url FOO -workDir /home/jenkins/agent 4d54c5f996f9da7c3ebad35ad60617c1e489e49f0d735ddc2f27189a0ed7623f jnlp-f64z8 *13:05:03* root 118 7 0 19:04 pts/0 00:00:00 tcpdump -s0 -i eth0 -w capture_jnlp-f64z8.cap *13:05:03* root 153 7 0 19:05 pts/0 00:00:00 /bin/bash -ex /tmp/jenkins11227328797915352496.sh *13:05:03* root 165 153 0 19:05 pts/0 00:00:00 ps -ef   The strace -ff follows the forks and saves each file with the pid as suffix eg strace.txt.7 for the java agent process. The entrypoint is based on https://github.com/jenkinsci/docker-inbound-agent/blob/ad12874fe5567e9c9f197144a767515b44683d9c/11/debian/Dockerfile and runs a source /usr/local/bin/jenkins-agent Other home made custom images also suffer from the disconnect, so I do not think at this point it is related to the image itself. Here are the relevant last lines of each strace: strace.txt.7 19:40:07.317915 --- SIGHUP {si_signo=SIGHUP, si_code=SI_USER, si_pid=0, si_uid=0} --- 19:40:07.318007 futex(0x7fc7b4008b00, FUTEX_WAKE_PRIVATE, 1) = 0 19:40:07.318110 rt_sigreturn({mask=[]}) = 202 19:40:07.318201 futex(0x7fc7bd8399d0, FUTEX_WAIT, 119, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set) 19:40:07.364432 futex(0x7fc7bd8399d0, FUTEX_WAIT, 119, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set) 19:40:07.497333 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=118, si_uid=0, si_status=0, si_utime=0, si_stime=2} --- 19:40:07.497381 futex(0x7fc7bd8399d0, FUTEX_WAIT, 119, NULL) = ? 19:40:07.812948 +++ exited with 129 +++ strace.txt.118 19:40:07.462836 --- SIGHUP {si_signo=SIGHUP, si_code=SI_USER, si_pid=0, si_uid=0} --- 19:40:07.462897 --- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=0, si_uid=0} --- 19:40:07.462966 --- SIGCONT {si_signo=SIGCONT, si_code=SI_USER, si_pid=0, si_uid=0} --- 19:40:07.463013 alarm(0) = 0 19:40:07.463072 rt_sigreturn({mask=[HUP]}) = 0 19:40:07.463134 alarm(0) = 0 19:40:07.463370 rt_sigreturn({mask=[]}) = -1 EINTR (Interrupted system call) 19:40:07.464347 open( "/proc/net/dev" , O_RDONLY) = 5 19:40:07.466061 fstat(5, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 19:40:07.466147 read(5, "Inter-| Receive " ..., 1024) = 446 19:40:07.466432 close(5) = 0 19:40:07.467197 getsockopt(3, SOL_PACKET, PACKET_STATISTICS, {packets=5167, drops=0}, [8]) = 0 19:40:07.467306 write(2, "5166 packets captured" , 21) = -1 EIO (Input/output error) 19:40:07.467435 write(2, "\n" , 1) = -1 EIO (Input/output error) 19:40:07.467661 write(2, "5167 packets received by filter" , 31) = -1 EIO (Input/output error) 19:40:07.467723 write(2, "\n" , 1) = -1 EIO (Input/output error) 19:40:07.467790 write(2, "0 packets dropped by kernel" , 27) = -1 EIO (Input/output error) 19:40:07.467857 write(2, "\n" , 1) = -1 EIO (Input/output error) 19:40:07.467920 setsockopt(3, SOL_PACKET, PACKET_RX_RING, {block_size=0, block_nr=0, frame_size=0, frame_nr=0}, 16) = -1 EINVAL (Invalid argument) 19:40:07.467985 munmap(0x7f50a7ec8000, 2097152) = 0 19:40:07.468287 munmap(0x7f50a8ee5000, 266240) = 0 19:40:07.468739 close(3) = 0 19:40:07.496796 write(4, "\314\205@\4#\222\237\257\366\344\370hX\357$A\35\350\242\311\232-\35\27\334\20Q\377\314\346\233\1" ..., 2442) = 2442 19:40:07.496951 exit_group(0) = ? 19:40:07.497192 +++ exited with 0 +++ strace.txt.153 19:40:07.361553 --- SIGHUP {si_signo=SIGHUP, si_code=SI_USER, si_pid=0, si_uid=0} --- 19:40:07.364313 +++ killed by SIGHUP +++ The weird part is that SIGHUP is coming from pid 0 which is the scheduler, I don't think I've seen that before. Looking at docker documentation it seems that a docker stop would first send a SIGTERM, then a SIGKILL after a time period. I have not seen any documented process that sends a SIGHUP (apart from sending a custom signal via docker cli where you can force it to be SIGHUP) a) This aligns with the bahavior we have seen. Java exits with 129 which is 128+kernel signal. Signal 1 is SIGHUP b) The tcpdump received SIGHUP also from pid 0, outputs some of its information eg "5166 packets captured" on the console and the exits successfully c) The jenkins "Execute Shell" forked process gets a SIGHUP too and dies. I have another 30+ strace files from that session. A lot of them seem to be forked java processes, that also exit with 129. Let me know if any of them would be useful data for this ticket. I have bundled them all in a tar file but I don't want to post this bundle on a public ticket.

Jesse Glick added a comment - 2021-02-23 19:30 I cannot think of any reason offhand why you would get SIGHUP. Note that if you suspect problems with WebSocket, you can check the server behavior using variants of websocat -vv -t wss://user:apitoken@jenkins/wsecho/

Samuel Beaulieu added a comment - 2021-03-04 15:12 Thanks for the tip. I do not think at this point that it is a specific issue due to websockets, it just happen that it is our setup, but the issue presents itself when using JNLP port too. Our next step to investigate is to run jenkins servers in kubernetes so that they are close to each other and see if the issue is still present.