Jesse Glick added a comment - 2017-06-28 14:41 Best for stephenconnolly to evaluate where the problem lies.

Jesse Glick added a comment - 2017-06-28 20:17 Is this really a product bug? GitHub failed to service a request, so we failed to scan the organization. Not sure what else we could do. (Possibly branch-api could decide that if scanning a repo or org gets an error partway through, that it will nonetheless refresh any existing entries, or add any newly discovered entries, without orphaning existing entries that it did not encounter. But I doubt that would help much here.)

R. Tyler Croy added a comment - 2017-06-28 20:27 Well, it's consistently failing and the plugin(s?) doesn't appear to be retrying on timeouts, so that to me is likely a bug which should be corrected. Here's some updated logs from a more recent scan, so at the very least it's not consistently the same URL, in which case I would be opening a support request with GitHub. 00:40:14 GitHub API Usage: Current quota has 2105 remaining (4 over budget). Next quota of 5000 in 29 min. Sleeping for 28 sec. 00:40:43 GitHub API Usage: Current quota has 2105 remaining (26 under budget). Next quota of 5000 in 29 min 00:40:43 Connecting to https: //api.github.com using jenkinsadmin/****** (GitHub access token for jenkinsadmin) Looking up jenkinsci/plot-plugin ERROR: [Wed Jun 28 00:40:54 GMT 2017] Could not fetch sources from navigator org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator@5a856ef1 org.kohsuke.github.HttpException: Server returned HTTP response code: -1, message: ' null ' for URL: https: //api.github.com/repositories/1163695/collaborators?page=8 at org.kohsuke.github.Requester.parse(Requester.java:612) at org.kohsuke.github.Requester._to(Requester.java:262) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester.to(Requester.java:224) at org.kohsuke.github.GHRepository.getCollaboratorNames(GHRepository.java:478) at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.updateCollaboratorNames(GitHubSCMSource.java:482) at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.retrieve(GitHubSCMSource.java:461) at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:300) at jenkins.scm.api.SCMSource.fetch(SCMSource.java:254) at jenkins.branch.MultiBranchProjectFactory$BySCMSourceCriteria.recognizes(MultiBranchProjectFactory.java:263) at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.recognizes(OrganizationFolder.java:1274) at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.complete(OrganizationFolder.java:1289) at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.add(GitHubSCMNavigator.java:540) at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:362) at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:401) at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:266) at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:162) at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:852) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:405) Caused by: java.net.SocketTimeoutException: timeout at okio.Okio$3.newTimeoutException(Okio.java:207) at okio.AsyncTimeout.exit(AsyncTimeout.java:261) at okio.AsyncTimeout$2.read(AsyncTimeout.java:215) at okio.RealBufferedSource.indexOf(RealBufferedSource.java:306) at okio.RealBufferedSource.indexOf(RealBufferedSource.java:300) at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:196) at com.squareup.okhttp.internal.http.Http1xStream.readResponse(Http1xStream.java:186) at com.squareup.okhttp.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:127) at com.squareup.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:737) at com.squareup.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:609) at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:460) at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:399) at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getResponseCode(HttpURLConnectionImpl.java:527) at com.squareup.okhttp.internal.huc.DelegatingHttpsURLConnection.getResponseCode(DelegatingHttpsURLConnection.java:105) at com.squareup.okhttp.internal.huc.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:25) at org.kohsuke.github.Requester.parse(Requester.java:586) ... 25 more Caused by: java.net.SocketException: Socket closed at java.net.SocketInputStream.read(SocketInputStream.java:204) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at sun.security.ssl.InputRecord.read(InputRecord.java:503) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930) at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) at okio.Okio$2.read(Okio.java:139) at okio.AsyncTimeout$2.read(AsyncTimeout.java:211) ... 38 more [Wed Jun 28 00:40:54 GMT 2017] Finished organization scan. Scan took 39 min FATAL: Failed to recompute children of Plugins org.kohsuke.github.HttpException: Server returned HTTP response code: -1, message: ' null ' for URL: https: //api.github.com/repositories/1163695/collaborators?page=8 at org.kohsuke.github.Requester.parse(Requester.java:612) at org.kohsuke.github.Requester._to(Requester.java:262) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester.to(Requester.java:224) at org.kohsuke.github.GHRepository.getCollaboratorNames(GHRepository.java:478) at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.updateCollaboratorNames(GitHubSCMSource.java:482) at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.retrieve(GitHubSCMSource.java:461) at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:300) at jenkins.scm.api.SCMSource.fetch(SCMSource.java:254) at jenkins.branch.MultiBranchProjectFactory$BySCMSourceCriteria.recognizes(MultiBranchProjectFactory.java:263) at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.recognizes(OrganizationFolder.java:1274) at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.complete(OrganizationFolder.java:1289) at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.add(GitHubSCMNavigator.java:540) at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:362) at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:401) at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:266) at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:162) at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:852) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:405) Caused by: java.net.SocketTimeoutException: timeout at okio.Okio$3.newTimeoutException(Okio.java:207) at okio.AsyncTimeout.exit(AsyncTimeout.java:261) at okio.AsyncTimeout$2.read(AsyncTimeout.java:215) at okio.RealBufferedSource.indexOf(RealBufferedSource.java:306) at okio.RealBufferedSource.indexOf(RealBufferedSource.java:300) at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:196) at com.squareup.okhttp.internal.http.Http1xStream.readResponse(Http1xStream.java:186) at com.squareup.okhttp.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:127) at com.squareup.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:737) at com.squareup.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:609) at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:460) at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:399) at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getResponseCode(HttpURLConnectionImpl.java:527) at com.squareup.okhttp.internal.huc.DelegatingHttpsURLConnection.getResponseCode(DelegatingHttpsURLConnection.java:105) at com.squareup.okhttp.internal.huc.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:25) at org.kohsuke.github.Requester.parse(Requester.java:586) ... 25 more Caused by: java.net.SocketException: Socket closed at java.net.SocketInputStream.read(SocketInputStream.java:204) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at sun.security.ssl.InputRecord.read(InputRecord.java:503) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930) at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) at okio.Okio$2.read(Okio.java:139) at okio.AsyncTimeout$2.read(AsyncTimeout.java:211) ... 38 more Finished: FAILURE

Jesse Glick added a comment - 2017-06-28 21:05 Probably Requester.parse could be handling SocketTimeoutException . I can take a stab at it; would you install a snapshot?

R. Tyler Croy added a comment - 2017-06-28 21:13 ci.jenkins.io is a production system, so no. I would imagine it'd be relatively simple to reproduce by pointing a local Jenkins instance at GitHub with an org folder

Jesse Glick added a comment - 2017-06-28 21:18 Well if you indeed have a reproduction case, let us know, otherwise this will get released (or not) without testing.

Jesse Glick added a comment - 2017-06-29 19:16 I did manage to reproduce it! Will check effectiveness of fix.

Jesse Glick added a comment - 2017-06-29 19:45 So far it seems to be effective. Tricky to prove it because rate limits mean that the indexing is mostly just sleeping. But I have seen a couple of requests time out and then successfully retry. BTW from the stack trace the timeout happened in code while would be superseded by  JENKINS-36240 , once stephenconnolly finishes his refactoring and that is unblocked.

SCM/JIRA link daemon added a comment - 2017-06-29 20:29 Code changed in jenkins User: Jesse Glick Path: demo/Dockerfile demo/plugins.txt http://jenkins-ci.org/commit/github-branch-source-plugin/675bb4ae3628b9f5fa162fe6a4224e42ed822c76 Log: Demo updates, used to reproduce JENKINS-45142 .

SCM/JIRA link daemon added a comment - 2017-07-03 20:56 Code changed in jenkins User: Jesse Glick Path: pom.xml http://jenkins-ci.org/commit/github-api-plugin/efee616085d2ca85c6d5e24448b3985f64332950 Log: JENKINS-45142 Picking up 1.86-SNAPSHOT.

SCM/JIRA link daemon added a comment - 2017-07-03 20:56 Code changed in jenkins User: Jesse Glick Path: Jenkinsfile pom.xml http://jenkins-ci.org/commit/github-api-plugin/e22637123336e40286a44e1931330a5026a423aa Log: Merge pull request #14 from jglick/SocketTimeoutException- JENKINS-45142 JENKINS-45142 Picking up 1.86 Compare: https://github.com/jenkinsci/github-api-plugin/compare/9b682edccdf9...e22637123336

Spencer Malone added a comment - 2017-07-13 18:44 Could we make this default retry count be tweakable? Having it retry is very useful, but it'd be extra-nice if we could specify how many times to retry on a global level.

Jesse Glick added a comment - 2017-07-13 20:11 Yeah, if the patch seems to have been helpful ( rtyler ?) there could be a follow-up to make it configurable via system property perhaps.

Jesse Glick added a comment - 2017-07-20 17:35 Based on these errors  which seem to be after the patch  I think this is not fixed well enough. Perhaps this fix should be reverted and  github-branch-source should do some kind of exponential backoff in cases where it really needs a response? This particular trigger for the error becomes obsolete as of JENKINS-36240 (assuming the new trust strategy is selected) but there might be others.

Evan Borgstrom added a comment - 2017-08-16 18:04 - edited We recently upgraded: git 3.3.2 -> 3.5.0 github-branch-source 2.0.8 -> 2.2.3 git-client 2.4.6 -> 2.5.0 scm-api 2.1.1 -> 2.2.0 After doing so we now see this exception during pretty much every full scan of our GitHub org (~1000 repos): FATAL: Failed to recompute children of GitHub org.kohsuke.github.HttpException: Server returned HTTP response code: -1, message: 'null' for URL: https://api.github.com/repos/ORG/REPO at org.kohsuke.github.Requester.parse(Requester.java:622) at org.kohsuke.github.Requester.parse(Requester.java:620) at org.kohsuke.github.Requester.parse(Requester.java:620) at org.kohsuke.github.Requester.parse(Requester.java:584) at org.kohsuke.github.Requester._to(Requester.java:264) at org.kohsuke.github.Requester.to(Requester.java:226) at org.kohsuke.github.GitHub.getRepository(GitHub.java:420) at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.createProbe(GitHubSCMSource.java:1056) at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource$2.create(GitHubSCMSource.java:882) at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource$2.create(GitHubSCMSource.java:876) at jenkins.scm.api.trait.SCMSourceRequest.process(SCMSourceRequest.java:341) at jenkins.scm.api.trait.SCMSourceRequest.process(SCMSourceRequest.java:251) at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.retrieve(GitHubSCMSource.java:875) at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:355) at jenkins.scm.api.SCMSource.fetch(SCMSource.java:309) at jenkins.branch.MultiBranchProjectFactory$BySCMSourceCriteria.recognizes(MultiBranchProjectFactory.java:263) at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.recognizes(OrganizationFolder.java:1297) at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.complete(OrganizationFolder.java:1312) at jenkins.scm.api.trait.SCMNavigatorRequest.process(SCMNavigatorRequest.java:256) at jenkins.scm.api.trait.SCMNavigatorRequest.process(SCMNavigatorRequest.java:206) at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:961) at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:412) at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:276) at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:161) at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:863) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:405) Caused by: java.net.SocketTimeoutException: timeout at okio.Okio$3.newTimeoutException(Okio.java:207) at okio.AsyncTimeout.exit(AsyncTimeout.java:261) at okio.AsyncTimeout$2.read(AsyncTimeout.java:215) at okio.RealBufferedSource.indexOf(RealBufferedSource.java:306) at okio.RealBufferedSource.indexOf(RealBufferedSource.java:300) at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:196) at com.squareup.okhttp.internal.http.Http1xStream.readResponse(Http1xStream.java:186) at com.squareup.okhttp.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:127) at com.squareup.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:737) at com.squareup.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:609) at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:460) at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:399) at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getResponseCode(HttpURLConnectionImpl.java:527) at com.squareup.okhttp.internal.huc.DelegatingHttpsURLConnection.getResponseCode(DelegatingHttpsURLConnection.java:105) at com.squareup.okhttp.internal.huc.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:25) at org.kohsuke.github.Requester.parse(Requester.java:592) ... 24 more Caused by: java.net.SocketException: Socket closed at java.net.SocketInputStream.read(SocketInputStream.java:204) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at sun.security.ssl.InputRecord.read(InputRecord.java:503) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930) at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) at okio.Okio$2.read(Okio.java:139) at okio.AsyncTimeout$2.read(AsyncTimeout.java:211) ... 37 more Finished: FAILURE I've anonymized ORG & REPO here, but it can happen on any of our repos. We have had 1 successful scan last week, but this week we went to change some more options to leverage the new settings in the plugins and have been unable to get a full scan completed in the past 3 days because of this. FWIW, changing the pull request trust setting to "Contributors" or "Admin/Write" causes this to happen much more frequently (i.e. it will reliably throw this HTTP -1 code error in the first couple repos, where when we set it to Nobody we can typically get much further – I'm guessing because Nobody skips asking for contributors). EDIT: Actually, we are hitting the timeout retry: timed out accessing https://api.github.com/repos/ORG/REPO/contents/?ref=refs%2Fpull%2F1769%2Fhead; will try 1 more time(s) But I believe the problem is being exasperated because there is no sleep between tries. Could we simply add a delay of a second or two in the retry handler before parse is re-called?

Evan Borgstrom added a comment - 2017-08-18 00:10 Today has been REALLY bad for this. We're now getting failures responding to webhooks: [Thu Aug 17 15:56:33 PDT 2017] Push event to branch THE-BRANCH in repository ORG/REPO UPDATED event from 192.30.252.40 ⇒ https://JENKINSHOST/github-webhook/ with timestamp Thu Aug 17 15:56:27 PDT 2017 processed in 1.3 sec Checking branch THE-BRANCH Getting remote pull request #530... 1 branches were processed Checking pull-requests... ERROR: Server returned HTTP response code: -1, message: 'null' for URL: https://api.github.com/users/NerdWallet java.net.SocketException: Socket closed CC jglick , stephenconnolly – Can anything be done to help get this reprioritized to get looked at again? We can't do a full org scan and quite a few of our web hooks failed today.

Evan Borgstrom added a comment - 2017-08-19 14:42 Yesterday I spent some time trying to work around this issue in the github-api plugin as was able to finally complete a full org scan! FWIW, here is what I changed: https://github.com/kohsuke/github-api/compare/master...borgstrom:workaround-http--1 When I tried to add the retry to parse I could never actually get it to retry correctly. Moving the retry logic to the _to function is what worked. I'm sure this is not the correct way to fix this, but wanted to share if someone else is in my position and needs to get a full org scan done until a proper fix can be put in place.

Jesse Glick added a comment - 2017-08-21 19:49 AFAIK using Admin/Write as the trust check should greatly reduce the number of requests.

Jesse Glick added a comment - 2017-08-22 14:35 borgstrom if you have something that works for you, file it as a PR and it can be reviewed whenever someone maintaining this code has time. (Assign the issue to yourself, mark as In Review, create a link to the PR, make sure the PR links back to JIRA.) Regarding my previous comment: TrustContributors definitely makes a lot of API calls and should be discouraged—this was the original trust implementation prior to GitHub enhancements that allowed TrustPermission to be introduced, which does a more precise job of identifying people with write permission. My claim that TrustPermission should be cheaper is based on reading sources of  stephenconnolly ’s somewhat weird implementation, in LazyContributorNames .

Evan Borgstrom added a comment - 2017-08-23 15:13 jglick Sounds good. I will cleanup my branch an file it as a PR. WRT to TrustContributors I have it set to "Nobody" for forks, so I don't think that's the issue. When it was set to "Contributors" the issue here was definitely exasperated. We've noticed a bunch of GitHub "issues" over the past couple weeks (like very elevated web hook delivery times), so I assume we're just running into some instabilities in their system.

Evan Borgstrom added a comment - 2017-08-23 15:23 PR filed: https://github.com/kohsuke/github-api/pull/373 I do not have permissions to assign myself or edit labels.

Jesse Glick added a comment - 2017-08-23 15:25 Nobody or Everyone is definitely lower-traffic, Contributors is definitely higher-traffic; I think  that From users with Admin or Write permission (the most accurate mode) is relatively cheap.

Erik Dahlinghaus added a comment - 2017-09-20 20:34 Any updates on this issue? PR on github-api still has not been merged because Evan said it's still not working for him. This is causing issues at my organization.