Since upgrading from 2.263.3 -> 2.277.2 we're seeing issues with slow requests on Jenkins when downloading artifacts, or querying job results (but honestly it could be more than this, this is just what we've noticed). It's intermittent, spread across jobs, artifacts, clients, etc. 

Slow is defined as 10+ seconds here. 

What we've determined is that the request is seemingly hanging in Jenkins for some time, 10s of seconds based on logging (See https://www.nginx.com/blog/using-nginx-logging-for-application-performance-monitoring/) on the nginx side. Nginx is indicating that for these slow requests its waiting ages for the first byte to come back from Jenkins, and eventually the client gives up (Read timeout). Sometimes we get nothing back even for headers. 

There is no pattern to these slow requests that we can find, they're just slow randomly throughout the day and night. Load on the system is very low (like <1 on an 8 core system), iowait is fine, we're not seeing drops on the network (but the nginx proxy is connecting over localhost to jenkins, just putting it out there to rule it out).  

Debugging this is proving difficult since the Jenkins access log doesn't actually log a line for these requests that I can see, though the nginx proxy does:

X.X.X.X - username [29/Apr/2021:20:23:47 +0000] "GET /fingerprint/redacted_fingerprint/api/python HTTP/1.1" 499 0 "" "python-requests/2.25.1" "" rt="30.003" uct="0.000" uht="-" urt="30.004"

Note the urt, this is all time spent waiting for jenkins to come back with something, but never does, which is why "uht" is empty, uct is very fast indicating the TCP connection was quick to jenkins. 

Beyond not having an access log line on the jenkins side for this, I'm not able to spot a smoking gun in custom LDAP logs that would indicate a problem with the LDAP servers. I'm also not having much luck sifting through the haystack that is jetty logs for an issue. 

Reproducing this is challenging, but not impossible. If someone can give us some hints as to what logging we should have enabled, or where to look next, that would be great. We never had this issue in 2.263.3 so we're lead to think this is something in Jenkins that has changed since we've checked for other changes on our side and there are none, including no new plugins installed.