After upgrading remoting to 4.14 we could observe agent node Response time more than 3s on node monitoring board, agent node hangs up on huge pipeline and also monitoring timeout in agent logs.
Reproduced with Kubernretes agent provisioning.
Agent successfully connected and online
ERROR: Failed to monitor for Clock Difference
java.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
ERROR: Failed to monitor for JVM Version
java.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
ERROR: Failed to monitor for Response Time
java.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.ResponseTimeMonitor$1.monitor(ResponseTimeMonitor.java:57)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
ERROR: Failed to monitor for Free Temp Space
java.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
ERROR: Failed to monitor for Free Swap Space
java.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
ERROR: Failed to monitor for Free Disk Space
java.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
ERROR: Failed to monitor for Architecture
java.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
ERROR: Failed to monitor for Clock Difference
java.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
ERROR: Failed to monitor for Architecture
java.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
ERROR: Failed to monitor for Free Temp Space
java.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
ERROR: Failed to monitor for Free Disk Space
java.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
ERROR: Failed to monitor for Free Swap Space
java.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
ERROR: Failed to monitor for JVM Version
java.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
ERROR: Failed to monitor for Response Time
java.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.ResponseTimeMonitor$1.monitor(ResponseTimeMonitor.java:57)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
Introduced by Pull request https://github.com/jenkinsci/remoting/pull/523
After replacing https://github.com/jenkinsci/remoting/pull/523/files#diff-df5e02eb0e94358acfe92fb427f75ea298f7e8a58857efb4fcff7c9f602974d6R54 with
Response time reduced to 50-300ms