+1 for the issue that was annoying for so long now, although most often seen on real branches (masters and releases of our project that are really providing a Jenkinsfile)
Yes we do use GitHub caching, advocated for it to appear, and hope it stays
with the poor internet uplink we have, and with github REST API quotas for uncached requests abound, and no possibility to get hooks thus requiring polling, - our farm couldn't really work without it.
So I set out digging in the data, and found that cached HTTP-404s in *.0 files correlate with very short *.1 files (the compact error message from Github REST API), so selecting those to look deeper:
:; find . -name '*.1' -size 1 | sed -e 's,^./,,' -e s',.1$,,' | while read F ; do egrep 'HTTP.*404' "$F.0" >&2 && echo "=== $F" && head -1 "$F.0" && ls -la "$F"* ; done
Due to reasons unknown however, the cached response for some of the URLs is HTTP/404 even with a valid JSON in the (gzipped) `hashstring.1` file:
{"message":"No commit found for the ref refs/heads/4.2.0-FTY","documentation_url":"https:}
and the corresponding `hashstring.0` file looks like:
:; cat fbe7227813e6f1a6bbb2f1e5202a84a2.0
https:GET
1
Authorization: Basic NDJpdHktY2k6NjA5MDk2YTVmNzNhNTc1YzE1OWYxZjI3NDJlZmI1YjhiMTQzZmIzMw==
HTTP/1.1 404 Not Found
31
X-OAuth-Scopes: admin:repo_hook, public_repo, repo:status, repo_deployment
X-Accepted-OAuth-Scopes:
X-GitHub-Media-Type: github.v3; format=json
Content-Encoding: gzip
Transfer-Encoding: chunked
Connection: keep-alive
Content-Type: application/octet-stream
X-Cache: MISS from thunderbolt.localdomain
X-Cache-Lookup: MISS from thunderbolt.localdomain:8080
Via: 1.1 thunderbolt.localdomain (squid/3.4.4)
Server: GitHub.com
Date: Thu, 28 Nov 2019 00:41:22 GMT
Status: 304 Not Modified
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 5000
X-RateLimit-Reset: 1574905280
Cache-Control: private, max-age=60, s-maxage=60
Vary: Accept, Authorization, Cookie, X-GitHub-OTP
ETag: "2513f4bbc2abb8b63adbec8336a82810a4fb5dc5"
Last-Modified: Wed, 05 Dec 2018 10:54:24 GMT
Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type
Access-Control-Allow-Origin: *
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: deny
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
Content-Security-Policy: default-src 'none'
X-GitHub-Request-Id: F066:2FC4:FE494:24E933:5DDF17B1
OkHttp-Sent-Millis: 1574901681913
OkHttp-Received-Millis: 1574901682110
TLS_RSA_WITH_AES_128_GCM_SHA256
2
MIIECDCCAvCgAwIBAgIUEG8XFkmTLxiL4iPSXqLddY7e6AswDQYJKoZIhvcNAQEFBQAwga0xCzAJBgNVBAYTAkNaMRcwFQYDVQQIDA5QcmFndWUgc3VidXJiczEQMA4GA1UEBwwHUm96dG9reTENMAsGA1UECgwERUVJQzERMA8GA1UECwwIQklPUyBMQUIxJDAiBgNVBAMMG3RodW5kZXJib2x0LnJvei5sYWIuZXRuLmNvbTErMCkGCSqGSIb3DQEJARYcRWF0b25JUENPcGVuc291cmNlQEVhdG9uLmNvbTAeFw0xOTA3MDgwMDAwMDBaFw0yMDA3MTYxMjAwMDBaMGgxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQHEw1TYW4gRnJhbmNpc2NvMRUwEwYDVQQKEwxHaXRIdWIsIEluYy4xFTATBgNVBAMMDCouZ2l0aHViLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKIFH+JTppW1pvbrqnLU1SCYOsFsI6vdoL66M/497v413h1TOEwGWEo1wvZq3YhD65VSlxrsEj7xGd+ZUy2/mzRh2XmGRolJUWd/XKCQ+lJukRLX3BYhRBXfGK9Njv/afR1OIs96A4dTZA7PpPwC5Gvk34iTcJe4giludMIIELzCCAxegAwIBAgIJAOz23xAU+F0TMA0GCSqGSIb3DQEBCwUAMIGtMQswCQYDVQQGEwJDWjEXMBUGA1UECAwOUHJhZ3VlIHN1YnVyYnMxEDAOBgNVBAcMB1JvenRva3kxDTALBgNVBAoMBEVFSUMxETAPBgNVBAsMCEJJT1MgTEFCMSQwIgYDVQQDDBt0aHVuZGVyYm9sdC5yb3oubGFiLmV0bi5jb20xKzApBgkqhkiG9w0BCQEWHEVhdG9uSVBDT3BlbnNvdXJjZUBFYXRvbi5jb20wHhcNMTgwNDAzMTIxNzU2WhcNMjgwMzMxMTIxNzU2WjCBrTELMAkGA1UEBhMCQ1oxFzAVBgNVBAgMDlByYWd1ZSBzdWJ1cmJzMRAwDgYDVQQHDAdSb3p0b2t5MQ0wCwYDVQQKDARFRUlDMREwDwYDVQQLDAhCSU9TIExBQjEkMCIGA1UEAwwbdGh1bmRlcmJvbHQucm96LmxhYi5ldG4uY29tMSswKQYJKoZIhvcNAQkBFhxFYXRvbklQQ09wZW5zb3VyY2VARWF0b24uY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAogUf4lOmlbWm9uuqctTVIJg6wWwjq92gvroz/j3u/jXeHVM4TAZYSjXC9mrdiEPrlVKXGuwSPvEZ35lTLb+bNGHZeYZGiUlRZ39coJD6Um6REtfcFiFEFd8Yr02O/9p9HU4iz3oDh1NkDs+k/ALka+TfiJNwl7iCKW530
It seems that a large portion of such files appeared Jul 22 between 15:45-16:30 UTC so maybe there was an outage of GitHub at that time... there were a few this year. The few other short files apparently point to scans/builds of recently merged PRs so the ephemeral branch is really not there.
UPDATE: https://t.co/cFs8GfdpVV marks it at 15:46 
For reasons unknown, the "Date:" timestamp in the .0 header file is fresh, probably from the last scan; the result and content on-disk remain unchanged. Manually submitted requests through same proxy do return expected contents of the Git branch (wrapped into GitHub's REST API JSON markup). Probably the client did submit the cached Etag, maybe with object timestamp, and Github confirmed the cached value is still valid (except due to that hiccup it isn't).
Possibly the sort-of-fix would be to set up an optional timeout for cached (negative only?) responses so eventually they are retried. Or making a forced option to Branch indexing/MBP rescan/SCM Polling/... so that the manually issued request is done not-cached (for all or negative cached replies) so updating the cache with real current replies as if from scratch.
Previously tried forcing the job configs to be not-disabled (via on-disk XMLs and reload of Jenkins configuration), this got the jobs not-marked with gray balls in the dashboard... but then they were re-marked probably due to this cache issue. For our OrgFolders making MultiBranch pipelines, the half-successful magic looked like this:
:; for D in /var/lib/jenkins/jobsbranches/*/config.xml ; do sed 's,<disabled>true</disabled>,<disabled>false</disabled>,' -i "$F" ; done ); done
jordanjennings - That is a completely separate issue.
jayache80 joncormier allir
Would you be willing to try the OkHttp3 update in
JENKINS-57411? It will not fix the issue while it is happening, but we are hoping it may prevent or reduce the occurrences, but we need more people that have actually seen the issue to try the fix.