Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54126

Jenkinsfile not found in PR on GitHub -- Does not meet criteria

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • None
    • Jenkins ver. 2.138.2

      Sometimes, when we create PR on GitHub - it is ignored by Jenkins. I've tried to re-scan organization - and see in logs:

        Getting remote pull requests...
      
          Checking pull request #128
            ‘Jenkinsfile’ not found
          Does not meet criteria
      

      But Jenkinsfile is here! I see other bugs in your Jira and here is some additional information:

      • Repository is added to Jenkins via GitHub Organization Folder
      • We don't use forks
      • Commiter is a member of our organization and has full rights on this repository
      • Problem PRs are also very simple and doesn't touch Jenkinsfile at all
      • Source PR branch has Jenkinsfile
      • Branch which we want to merge - was created from source branch and also has Jenkinsfile
      • Jenkinsfile was not changed between merge commits
      • If I merge branches manually - result has Jenkinsfile
      • If we do merge on GitHub UI - result will have Jenkinsfile (and will be built)
      • If we recreate same pull request several times - it will be built

      And more logs:

      Examining our-organization/our-repo:  
      
      Checking branches...  
      Getting remote branches...  
        
      Checking branch master
            ‘Jenkinsfile’ found
          Met criteria
      No changes detected: master (still at 923197f48be5cd8296b8ca95bd72a4a830a474f4)    
      
      Checking branch develop
            ‘Jenkinsfile’ found
          Met criteria
      No changes detected: develop (still at 43c2cce36623a4af90b28d886dfb28ea8d813ab8)    
      
      Checking branch feature/198-video-verification
            ‘Jenkinsfile’ found
          Met criteria
      No changes detected: feature/198-video-verification (still at 9126f0c6958db3c712078ed0d2587e96004d27c6)  
      
      3 branches were processed  
      
      Checking pull-requests...  
      Getting remote pull requests...    
      
      Checking pull request #128
            ‘Jenkinsfile’ not found
          Does not meet criteria
      

      PR #128 - is between develop and feature/198-video-verification - and, as you can see, both have Jenkinsfile and were built successfully.

          [JENKINS-54126] Jenkinsfile not found in PR on GitHub -- Does not meet criteria

          Liam Newman added a comment -

          jordanjennings - That is a completely separate issue. 

          jayache80 joncormier allir 

          Would you be willing to try the OkHttp3 update in JENKINS-57411 ?  It will not fix the issue while it is happening, but we are hoping it may prevent or reduce the occurrences, but we need more people that have actually seen the issue to try the fix.

          Liam Newman added a comment - jordanjennings - That is a completely separate issue.  jayache80 joncormier allir   Would you be willing to try the OkHttp3 update in  JENKINS-57411 ?  It will not fix the issue while it is happening, but we are hoping it may prevent or reduce the occurrences, but we need more people that have actually seen the issue to try the fix.

          Aðalsteinn Rúnarsson added a comment - - edited

          Hey, bitwiseman

          This is happening very rarely for us, but it is annoying when it happens
          I can see about running that OkHttp3 update, it seems to be a patch version behind though? Are there any drawbacks from installing the OkHttp3 version?

           

          As a side note, It might be related, sometimes when we add a github repo and the credentials provided in Jenkins don't have access we seem to run into a cache-ing issue as well.

          So we add the Jenkins user to the Github repo with Admin access, but rescanning the pipeline still fails with invalid credentials error. Also the "Validate" function in the Job configuration will output `Error: Credentials Ok`
          This can be worked around by changing the case of any letter in the repo url in the Job configuration. (for example change `https://github.com/user/repo` to `https://github.com/User/repo`)

          I haven't tried deleting the Github Branch Source cache but I'm guessing that would also solve the issue, I will try that next time instead of the above mentioned workaround.

          Aðalsteinn Rúnarsson added a comment - - edited Hey,  bitwiseman This is happening very rarely for us, but it is annoying when it happens I can see about running that OkHttp3 update, it seems to be a patch version behind though? Are there any drawbacks from installing the OkHttp3 version?   As a side note, It might be related, sometimes when we add a github repo and the credentials provided in Jenkins don't have access we seem to run into a cache-ing issue as well. So we add the Jenkins user to the Github repo with Admin access, but rescanning the pipeline still fails with invalid credentials error. Also the "Validate" function in the Job configuration will output `Error: Credentials Ok` This can be worked around by changing the case of any letter in the repo url in the Job configuration. (for example change ` https://github.com/user/repo ` to ` https://github.com/User/repo `) I haven't tried deleting the Github Branch Source cache but I'm guessing that would also solve the issue, I will try that next time instead of the above mentioned workaround.

          Jim Klimov added a comment - - edited

          +1 for the issue that was annoying for so long now, although most often seen on real branches (masters and releases of our project that are really providing a Jenkinsfile)

          Yes we do use GitHub caching, advocated for it to appear, and hope it stays with the poor internet uplink we have, and with github REST API quotas for uncached requests abound, and no possibility to get hooks thus requiring polling, - our farm couldn't really work without it.

          So I set out digging in the data, and found that cached HTTP-404s in *.0 files correlate with very short *.1 files (the compact error message from Github REST API), so selecting those to look deeper:

          :; find . -name '*.1' -size 1 | sed -e 's,^./,,' -e s',.1$,,' | while read F ; do egrep 'HTTP.*404' "$F.0" >&2 && echo "=== $F" && head -1 "$F.0" && ls -la "$F"* ; done
          

          Due to reasons unknown however, the cached response for some of the URLs is HTTP/404 even with a valid JSON in the (gzipped) `hashstring.1` file:

          {"message":"No commit found for the ref refs/heads/4.2.0-FTY","documentation_url":"https://developer.github.com/v3/repos/contents/"}
          

          and the corresponding `hashstring.0` file looks like:

          :; cat fbe7227813e6f1a6bbb2f1e5202a84a2.0
          
          https://api.github.com/repos/42ity/libzmq/contents/?ref=refs%2Fheads%2F4.2.0-FTY
          GET
          1
          Authorization: Basic NDJpdHktY2k6NjA5MDk2YTVmNzNhNTc1YzE1OWYxZjI3NDJlZmI1YjhiMTQzZmIzMw==
          HTTP/1.1 404 Not Found
          31
          X-OAuth-Scopes: admin:repo_hook, public_repo, repo:status, repo_deployment
          X-Accepted-OAuth-Scopes:
          X-GitHub-Media-Type: github.v3; format=json
          Content-Encoding: gzip
          Transfer-Encoding: chunked
          Connection: keep-alive
          Content-Type: application/octet-stream
          X-Cache: MISS from thunderbolt.localdomain
          X-Cache-Lookup: MISS from thunderbolt.localdomain:8080
          Via: 1.1 thunderbolt.localdomain (squid/3.4.4)
          Server: GitHub.com
          Date: Thu, 28 Nov 2019 00:41:22 GMT
          Status: 304 Not Modified
          X-RateLimit-Limit: 5000
          X-RateLimit-Remaining: 5000
          X-RateLimit-Reset: 1574905280
          Cache-Control: private, max-age=60, s-maxage=60
          Vary: Accept, Authorization, Cookie, X-GitHub-OTP
          ETag: "2513f4bbc2abb8b63adbec8336a82810a4fb5dc5"
          Last-Modified: Wed, 05 Dec 2018 10:54:24 GMT
          Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type
          Access-Control-Allow-Origin: *
          Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
          X-Frame-Options: deny
          X-Content-Type-Options: nosniff
          X-XSS-Protection: 1; mode=block
          Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
          Content-Security-Policy: default-src 'none'
          X-GitHub-Request-Id: F066:2FC4:FE494:24E933:5DDF17B1
          OkHttp-Sent-Millis: 1574901681913
          OkHttp-Received-Millis: 1574901682110
          
          TLS_RSA_WITH_AES_128_GCM_SHA256
          2
          MIIECDCCAvCgAwIBAgIUEG8XFkmTLxiL4iPSXqLddY7e6AswDQYJKoZIhvcNAQEFBQAwga0xCzAJBgNVBAYTAkNaMRcwFQYDVQQIDA5QcmFndWUgc3VidXJiczEQMA4GA1UEBwwHUm96dG9reTENMAsGA1UECgwERUVJQzERMA8GA1UECwwIQklPUyBMQUIxJDAiBgNVBAMMG3RodW5kZXJib2x0LnJvei5sYWIuZXRuLmNvbTErMCkGCSqGSIb3DQEJARYcRWF0b25JUENPcGVuc291cmNlQEVhdG9uLmNvbTAeFw0xOTA3MDgwMDAwMDBaFw0yMDA3MTYxMjAwMDBaMGgxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQHEw1TYW4gRnJhbmNpc2NvMRUwEwYDVQQKEwxHaXRIdWIsIEluYy4xFTATBgNVBAMMDCouZ2l0aHViLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKIFH+JTppW1pvbrqnLU1SCYOsFsI6vdoL66M/497v413h1TOEwGWEo1wvZq3YhD65VSlxrsEj7xGd+ZUy2/mzRh2XmGRolJUWd/XKCQ+lJukRLX3BYhRBXfGK9Njv/afR1OIs96A4dTZA7PpPwC5Gvk34iTcJe4gilud//3UqD55A0jk+uEwQqosAImeGQg4Ayqo3K5rR+NhF8NnR7kXT1Cijk6jySbgX5Lhu8FPu7LdiPntxjuvFNJNaRy+6t4PxHJ1iRRlDdsVHyZMcZGb8klafrKsr7kLBWSMKiVaXTdlNc26bUOctH+LySlZB6Q7LgSec3MBqXZBFk0AzfwxPcCAwEAAaNkMGIwIwYDVR0RBBwwGoIMKi5naXRodWIuY29tggpnaXRodWIuY29tMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQUFAAOCAQEAocIF+SVNlLFzWv0A/OUu4TG+aRBdzplMrF6Gy8JxwBSp22SB1PD2H71R5bi4U7UA3vgnpLbyg283XhZndNern1rIf49XXTqFbPC1xcZi85NcYc6xE18pnO0GQRaVgple2MOZXrn32FPgV2Zn/5XxGlQU1eL8leLc8tvMZkokmuBWRkuvCkx7xM5YMSAo4lRsL6zqzio/RLTOqWP1d6qSsGsf3Zc4HJ5RUTeA2QnyO1TRVvO+8bo5rQUHBOVmYhc006zs35LsjaUhG/6R1POZW2OS55U8ArQgLE/dZZV9mNJsTdd2hefv3v0+/whB+Y3stiO7zDMVFIOoHEd0+cUfGg==
          MIIELzCCAxegAwIBAgIJAOz23xAU+F0TMA0GCSqGSIb3DQEBCwUAMIGtMQswCQYDVQQGEwJDWjEXMBUGA1UECAwOUHJhZ3VlIHN1YnVyYnMxEDAOBgNVBAcMB1JvenRva3kxDTALBgNVBAoMBEVFSUMxETAPBgNVBAsMCEJJT1MgTEFCMSQwIgYDVQQDDBt0aHVuZGVyYm9sdC5yb3oubGFiLmV0bi5jb20xKzApBgkqhkiG9w0BCQEWHEVhdG9uSVBDT3BlbnNvdXJjZUBFYXRvbi5jb20wHhcNMTgwNDAzMTIxNzU2WhcNMjgwMzMxMTIxNzU2WjCBrTELMAkGA1UEBhMCQ1oxFzAVBgNVBAgMDlByYWd1ZSBzdWJ1cmJzMRAwDgYDVQQHDAdSb3p0b2t5MQ0wCwYDVQQKDARFRUlDMREwDwYDVQQLDAhCSU9TIExBQjEkMCIGA1UEAwwbdGh1bmRlcmJvbHQucm96LmxhYi5ldG4uY29tMSswKQYJKoZIhvcNAQkBFhxFYXRvbklQQ09wZW5zb3VyY2VARWF0b24uY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAogUf4lOmlbWm9uuqctTVIJg6wWwjq92gvroz/j3u/jXeHVM4TAZYSjXC9mrdiEPrlVKXGuwSPvEZ35lTLb+bNGHZeYZGiUlRZ39coJD6Um6REtfcFiFEFd8Yr02O/9p9HU4iz3oDh1NkDs+k/ALka+TfiJNwl7iCKW53//dSoPnkDSOT64TBCqiwAiZ4ZCDgDKqjcrmtH42EXw2dHuRdPUKKOTqPJJuBfkuG7wU+7st2I+e3GO68U0k1pHL7q3g/EcnWJFGUN2xUfJkxxkZvySVp+sqyvuQsFZIwqJVpdN2U1zbptQ5y0f4vJKVkHpDsuBJ5zcwGpdkEWTQDN/DE9wIDAQABo1AwTjAdBgNVHQ4EFgQUAf/vfDxEB9kv3Cfo9fb3ikvyWNswHwYDVR0jBBgwFoAUAf/vfDxEB9kv3Cfo9fb3ikvyWNswDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAlwBAM+b+mxtzgP+Q5AFWzLqj2TwSWXERGNnZQFDVeoZXb2y7UqaAf+Dz8WvTOrn51/fE5jsyqYHUCBXucbFJIuFx4G7vhsspcraIgenTGoP5N4L2UamrEkrqBl1CkYVhP2aykdA9G2Tu/61/rHMNycuLCf/CrZA54QlVQ8M8KtAQo+CEKcGeDBabP4TOtWvPO7ScM9kj5vRTiwy0DaVIL2VaNWLsdqT9tQ8e01wB1CRtjBFb1lhr3zMT0wXF8gAA9zcL6h1/1yiD5lNFKYUTKtsAuLpNb51lUq1k8eshyqiCHMrSm9/nj4L1WcWSiiR4MxvU2DTGUmwrKJ6Z3tf1Xw==
          0
          

          It seems that a large portion of such files appeared Jul 22 between 15:45-16:30 UTC so maybe there was an outage of GitHub at that time... there were a few this year. The few other short files apparently point to scans/builds of recently merged PRs so the ephemeral branch is really not there.

          UPDATE: https://t.co/cFs8GfdpVV marks it at 15:46

          For reasons unknown, the "Date:" timestamp in the .0 header file is fresh, probably from the last scan; the result and content on-disk remain unchanged. Manually submitted requests through same proxy do return expected contents of the Git branch (wrapped into GitHub's REST API JSON markup). Probably the client did submit the cached Etag, maybe with object timestamp, and Github confirmed the cached value is still valid (except due to that hiccup it isn't).

          Possibly the sort-of-fix would be to set up an optional timeout for cached (negative only?) responses so eventually they are retried. Or making a forced option to Branch indexing/MBP rescan/SCM Polling/... so that the manually issued request is done not-cached (for all or negative cached replies) so updating the cache with real current replies as if from scratch.

          Previously tried forcing the job configs to be not-disabled (via on-disk XMLs and reload of Jenkins configuration), this got the jobs not-marked with gray balls in the dashboard... but then they were re-marked probably due to this cache issue. For our OrgFolders making MultiBranch pipelines, the half-successful magic looked like this:

          :; for D in /var/lib/jenkins/jobs/*/jobs ; do ( cd "$D" && for F in */branches/*/config.xml ; do sed 's,<disabled>true</disabled>,<disabled>false</disabled>,' -i "$F" ; done ); done
          

          Jim Klimov added a comment - - edited +1 for the issue that was annoying for so long now, although most often seen on real branches (masters and releases of our project that are really providing a Jenkinsfile) Yes we do use GitHub caching, advocated for it to appear, and hope it stays with the poor internet uplink we have, and with github REST API quotas for uncached requests abound, and no possibility to get hooks thus requiring polling, - our farm couldn't really work without it. So I set out digging in the data, and found that cached HTTP-404s in *.0 files correlate with very short *.1 files (the compact error message from Github REST API), so selecting those to look deeper: :; find . -name '*.1' -size 1 | sed -e 's,^./,,' -e s ',.1$,,' | while read F ; do egrep 'HTTP.*404' "$F.0" >&2 && echo "=== $F" && head -1 "$F.0" && ls -la "$F" * ; done Due to reasons unknown however, the cached response for some of the URLs is HTTP/404 even with a valid JSON in the (gzipped) `hashstring.1` file: { "message" : "No commit found for the ref refs/heads/4.2.0-FTY" , "documentation_url" : "https: //developer.github.com/v3/repos/contents/" } and the corresponding `hashstring.0` file looks like: :; cat fbe7227813e6f1a6bbb2f1e5202a84a2.0 https: //api.github.com/repos/42ity/libzmq/contents/?ref=refs%2Fheads%2F4.2.0-FTY GET 1 Authorization: Basic NDJpdHktY2k6NjA5MDk2YTVmNzNhNTc1YzE1OWYxZjI3NDJlZmI1YjhiMTQzZmIzMw== HTTP/1.1 404 Not Found 31 X-OAuth-Scopes: admin:repo_hook, public_repo, repo:status, repo_deployment X-Accepted-OAuth-Scopes: X-GitHub-Media-Type: github.v3; format=json Content-Encoding: gzip Transfer-Encoding: chunked Connection: keep-alive Content-Type: application/octet-stream X-Cache: MISS from thunderbolt.localdomain X-Cache-Lookup: MISS from thunderbolt.localdomain:8080 Via: 1.1 thunderbolt.localdomain (squid/3.4.4) Server: GitHub.com Date: Thu, 28 Nov 2019 00:41:22 GMT Status: 304 Not Modified X-RateLimit-Limit: 5000 X-RateLimit-Remaining: 5000 X-RateLimit-Reset: 1574905280 Cache-Control: private , max-age=60, s-maxage=60 Vary: Accept, Authorization, Cookie, X-GitHub-OTP ETag: "2513f4bbc2abb8b63adbec8336a82810a4fb5dc5" Last-Modified: Wed, 05 Dec 2018 10:54:24 GMT Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type Access-Control-Allow-Origin: * Strict-Transport-Security: max-age=31536000; includeSubdomains; preload X-Frame-Options: deny X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin Content-Security-Policy: default -src 'none' X-GitHub-Request-Id: F066:2FC4:FE494:24E933:5DDF17B1 OkHttp-Sent-Millis: 1574901681913 OkHttp-Received-Millis: 1574901682110 TLS_RSA_WITH_AES_128_GCM_SHA256 2 MIIECDCCAvCgAwIBAgIUEG8XFkmTLxiL4iPSXqLddY7e6AswDQYJKoZIhvcNAQEFBQAwga0xCzAJBgNVBAYTAkNaMRcwFQYDVQQIDA5QcmFndWUgc3VidXJiczEQMA4GA1UEBwwHUm96dG9reTENMAsGA1UECgwERUVJQzERMA8GA1UECwwIQklPUyBMQUIxJDAiBgNVBAMMG3RodW5kZXJib2x0LnJvei5sYWIuZXRuLmNvbTErMCkGCSqGSIb3DQEJARYcRWF0b25JUENPcGVuc291cmNlQEVhdG9uLmNvbTAeFw0xOTA3MDgwMDAwMDBaFw0yMDA3MTYxMjAwMDBaMGgxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQHEw1TYW4gRnJhbmNpc2NvMRUwEwYDVQQKEwxHaXRIdWIsIEluYy4xFTATBgNVBAMMDCouZ2l0aHViLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKIFH+JTppW1pvbrqnLU1SCYOsFsI6vdoL66M/497v413h1TOEwGWEo1wvZq3YhD65VSlxrsEj7xGd+ZUy2/mzRh2XmGRolJUWd/XKCQ+lJukRLX3BYhRBXfGK9Njv/afR1OIs96A4dTZA7PpPwC5Gvk34iTcJe4gilud //3UqD55A0jk+uEwQqosAImeGQg4Ayqo3K5rR+NhF8NnR7kXT1Cijk6jySbgX5Lhu8FPu7LdiPntxjuvFNJNaRy+6t4PxHJ1iRRlDdsVHyZMcZGb8klafrKsr7kLBWSMKiVaXTdlNc26bUOctH+LySlZB6Q7LgSec3MBqXZBFk0AzfwxPcCAwEAAaNkMGIwIwYDVR0RBBwwGoIMKi5naXRodWIuY29tggpnaXRodWIuY29tMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQUFAAOCAQEAocIF+SVNlLFzWv0A/OUu4TG+aRBdzplMrF6Gy8JxwBSp22SB1PD2H71R5bi4U7UA3vgnpLbyg283XhZndNern1rIf49XXTqFbPC1xcZi85NcYc6xE18pnO0GQRaVgple2MOZXrn32FPgV2Zn/5XxGlQU1eL8leLc8tvMZkokmuBWRkuvCkx7xM5YMSAo4lRsL6zqzio/RLTOqWP1d6qSsGsf3Zc4HJ5RUTeA2QnyO1TRVvO+8bo5rQUHBOVmYhc006zs35LsjaUhG/6R1POZW2OS55U8ArQgLE/dZZV9mNJsTdd2hefv3v0+/whB+Y3stiO7zDMVFIOoHEd0+cUfGg== MIIELzCCAxegAwIBAgIJAOz23xAU+F0TMA0GCSqGSIb3DQEBCwUAMIGtMQswCQYDVQQGEwJDWjEXMBUGA1UECAwOUHJhZ3VlIHN1YnVyYnMxEDAOBgNVBAcMB1JvenRva3kxDTALBgNVBAoMBEVFSUMxETAPBgNVBAsMCEJJT1MgTEFCMSQwIgYDVQQDDBt0aHVuZGVyYm9sdC5yb3oubGFiLmV0bi5jb20xKzApBgkqhkiG9w0BCQEWHEVhdG9uSVBDT3BlbnNvdXJjZUBFYXRvbi5jb20wHhcNMTgwNDAzMTIxNzU2WhcNMjgwMzMxMTIxNzU2WjCBrTELMAkGA1UEBhMCQ1oxFzAVBgNVBAgMDlByYWd1ZSBzdWJ1cmJzMRAwDgYDVQQHDAdSb3p0b2t5MQ0wCwYDVQQKDARFRUlDMREwDwYDVQQLDAhCSU9TIExBQjEkMCIGA1UEAwwbdGh1bmRlcmJvbHQucm96LmxhYi5ldG4uY29tMSswKQYJKoZIhvcNAQkBFhxFYXRvbklQQ09wZW5zb3VyY2VARWF0b24uY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAogUf4lOmlbWm9uuqctTVIJg6wWwjq92gvroz/j3u/jXeHVM4TAZYSjXC9mrdiEPrlVKXGuwSPvEZ35lTLb+bNGHZeYZGiUlRZ39coJD6Um6REtfcFiFEFd8Yr02O/9p9HU4iz3oDh1NkDs+k/ALka+TfiJNwl7iCKW53 //dSoPnkDSOT64TBCqiwAiZ4ZCDgDKqjcrmtH42EXw2dHuRdPUKKOTqPJJuBfkuG7wU+7st2I+e3GO68U0k1pHL7q3g/EcnWJFGUN2xUfJkxxkZvySVp+sqyvuQsFZIwqJVpdN2U1zbptQ5y0f4vJKVkHpDsuBJ5zcwGpdkEWTQDN/DE9wIDAQABo1AwTjAdBgNVHQ4EFgQUAf/vfDxEB9kv3Cfo9fb3ikvyWNswHwYDVR0jBBgwFoAUAf/vfDxEB9kv3Cfo9fb3ikvyWNswDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAlwBAM+b+mxtzgP+Q5AFWzLqj2TwSWXERGNnZQFDVeoZXb2y7UqaAf+Dz8WvTOrn51/fE5jsyqYHUCBXucbFJIuFx4G7vhsspcraIgenTGoP5N4L2UamrEkrqBl1CkYVhP2aykdA9G2Tu/61/rHMNycuLCf/CrZA54QlVQ8M8KtAQo+CEKcGeDBabP4TOtWvPO7ScM9kj5vRTiwy0DaVIL2VaNWLsdqT9tQ8e01wB1CRtjBFb1lhr3zMT0wXF8gAA9zcL6h1/1yiD5lNFKYUTKtsAuLpNb51lUq1k8eshyqiCHMrSm9/nj4L1WcWSiiR4MxvU2DTGUmwrKJ6Z3tf1Xw== 0 It seems that a large portion of such files appeared Jul 22 between 15:45-16:30 UTC so maybe there was an outage of GitHub at that time... there were a few this year. The few other short files apparently point to scans/builds of recently merged PRs so the ephemeral branch is really not there. UPDATE: https://t.co/cFs8GfdpVV marks it at 15:46 For reasons unknown, the "Date:" timestamp in the .0 header file is fresh, probably from the last scan; the result and content on-disk remain unchanged. Manually submitted requests through same proxy do return expected contents of the Git branch (wrapped into GitHub's REST API JSON markup). Probably the client did submit the cached Etag, maybe with object timestamp, and Github confirmed the cached value is still valid (except due to that hiccup it isn't). Possibly the sort-of-fix would be to set up an optional timeout for cached (negative only?) responses so eventually they are retried. Or making a forced option to Branch indexing/MBP rescan/SCM Polling/... so that the manually issued request is done not-cached (for all or negative cached replies) so updating the cache with real current replies as if from scratch. Previously tried forcing the job configs to be not-disabled (via on-disk XMLs and reload of Jenkins configuration), this got the jobs not-marked with gray balls in the dashboard... but then they were re-marked probably due to this cache issue. For our OrgFolders making MultiBranch pipelines, the half-successful magic looked like this: :; for D in / var /lib/jenkins/jobs /*/jobs ; do ( cd "$D" && for F in */ branches/*/config.xml ; do sed 's,<disabled> true </disabled>,<disabled> false </disabled>,' -i "$F" ; done ); done

          Liam Newman added a comment -

          We have determined that issue is being caused by a bug in the GitHub API. The problem is described in https://github.com/github-api/github-api/pull/669 .

          The linked PR (https://github.com/github-api/github-api/pull/665) now shows a workaround that should work for all scenarios. The workaround will only execute when actually needed and will occur without caller of the github-api library knowing about it.

          I'll do the work to upgrade this dependency (already in progress) and this problem will go away. This is my top priority for the coming week.

          Liam Newman added a comment - We have determined that issue is being caused by a bug in the GitHub API. The problem is described in https://github.com/github-api/github-api/pull/669 . The linked PR ( https://github.com/github-api/github-api/pull/665 ) now shows a workaround that should work for all scenarios. The workaround will only execute when actually needed and will occur without caller of the github-api library knowing about it. I'll do the work to upgrade this dependency (already in progress) and this problem will go away. This is my top priority for the coming week.

          Jesse Glick added a comment -

          So has this been reported to GitHub?

          Jesse Glick added a comment - So has this been reported to GitHub?

          Liam Newman added a comment -

          jglickYes, I've reported it to github via a support ticket. I have not heard back from them beyond an automated response.

          Liam Newman added a comment - jglick Yes, I've reported it to github via a support ticket. I have not heard back from them beyond an automated response.

          Liam Newman added a comment -

          The fix for this issue has been merged and will be released in the next day or two in github-branch-source v2.6.0.

          If you want to try it out now:
          install github-api-plugin 1.6.0
          and then install the hpi from: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/github-branch-source/2.5.9-rc1028.3059575bf1cc/

          Liam Newman added a comment - The fix for this issue has been merged and will be released in the next day or two in github-branch-source v2.6.0. If you want to try it out now: install github-api-plugin 1.6.0 and then install the hpi from: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/github-branch-source/2.5.9-rc1028.3059575bf1cc/

          bitwiseman AFAICT with just github-api-plugin 1.106 the problem should go away? Of course it might do something less optimal

          Raihaan Shouhell added a comment - bitwiseman AFAICT with just github-api-plugin 1.106 the problem should go away? Of course it might do something less optimal

          Liam Newman added a comment -

          Github-branch-source-plugin v2.6.0

          Liam Newman added a comment - Github-branch-source-plugin v2.6.0

          Thanks for fixing this bitwiseman! It's been a long time frustration.

          Pascal Widdershoven added a comment - Thanks for fixing this bitwiseman ! It's been a long time frustration.

            bitwiseman Liam Newman
            soar Aleksey Smyrnov
            Votes:
            72 Vote for this issue
            Watchers:
            101 Start watching this issue

              Created:
              Updated:
              Resolved: