-
Bug
-
Resolution: Unresolved
-
Critical
-
None
Noticed while opening the "plugin flood gates" on ci.jeknins.io (from here)
I would guesstimate that an underlying timeout isn't being handled properly, not sure why a higher level exception isn't being received though.
ERROR: [Mon Jun 26 20:42:01 GMT 2017] Could not fetch sources from navigator org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator@1c022c0c org.kohsuke.github.HttpException: Server returned HTTP response code: -1, message: 'null' for URL: https://api.github.com/repositories/1163431/collaborators?page=12 at org.kohsuke.github.Requester.parse(Requester.java:612) at org.kohsuke.github.Requester._to(Requester.java:262) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester.to(Requester.java:224) at org.kohsuke.github.GHRepository.getCollaboratorNames(GHRepository.java:478) at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.updateCollaboratorNames(GitHubSCMSource.java:482) at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.retrieve(GitHubSCMSource.java:461) at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:300) at jenkins.scm.api.SCMSource.fetch(SCMSource.java:254) at jenkins.branch.MultiBranchProjectFactory$BySCMSourceCriteria.recognizes(MultiBranchProjectFactory.java:263) at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.recognizes(OrganizationFolder.java:1274) at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.complete(OrganizationFolder.java:1289) at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.add(GitHubSCMNavigator.java:540) at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:362) at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:401) at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:266) at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:162) at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:852) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:405) Caused by: java.net.SocketTimeoutException: timeout at okio.Okio$3.newTimeoutException(Okio.java:207) at okio.AsyncTimeout.exit(AsyncTimeout.java:261) at okio.AsyncTimeout$2.read(AsyncTimeout.java:215) at okio.RealBufferedSource.indexOf(RealBufferedSource.java:306) at okio.RealBufferedSource.indexOf(RealBufferedSource.java:300) at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:196) at com.squareup.okhttp.internal.http.Http1xStream.readResponse(Http1xStream.java:186) at com.squareup.okhttp.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:127) at com.squareup.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:737) at com.squareup.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:609) at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:460) at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:399) at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getResponseCode(HttpURLConnectionImpl.java:527) at com.squareup.okhttp.internal.huc.DelegatingHttpsURLConnection.getResponseCode(DelegatingHttpsURLConnection.java:105) at com.squareup.okhttp.internal.huc.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:25) at org.kohsuke.github.Requester.parse(Requester.java:586) ... 29 more Caused by: java.net.SocketException: Socket closed at java.net.SocketInputStream.read(SocketInputStream.java:204) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at sun.security.ssl.InputRecord.read(InputRecord.java:503) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930) at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) at okio.Okio$2.read(Okio.java:139) at okio.AsyncTimeout$2.read(AsyncTimeout.java:211) ... 42 more [Mon Jun 26 20:42:01 GMT 2017] Finished organization scan. Scan took 1 min 57 sec FATAL: Failed to recompute children of Plugins org.kohsuke.github.HttpException: Server returned HTTP response code: -1, message: 'null' for URL: https://api.github.com/repositories/1163431/collaborators?page=12 at org.kohsuke.github.Requester.parse(Requester.java:612) at org.kohsuke.github.Requester._to(Requester.java:262) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester._to(Requester.java:270) at org.kohsuke.github.Requester.to(Requester.java:224) at org.kohsuke.github.GHRepository.getCollaboratorNames(GHRepository.java:478) at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.updateCollaboratorNames(GitHubSCMSource.java:482) at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.retrieve(GitHubSCMSource.java:461) at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:300) at jenkins.scm.api.SCMSource.fetch(SCMSource.java:254) at jenkins.branch.MultiBranchProjectFactory$BySCMSourceCriteria.recognizes(MultiBranchProjectFactory.java:263) at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.recognizes(OrganizationFolder.java:1274) at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.complete(OrganizationFolder.java:1289) at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.add(GitHubSCMNavigator.java:540) at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:362) at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:401) at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:266) at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:162) at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:852) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:405) Caused by: java.net.SocketTimeoutException: timeout at okio.Okio$3.newTimeoutException(Okio.java:207) at okio.AsyncTimeout.exit(AsyncTimeout.java:261) at okio.AsyncTimeout$2.read(AsyncTimeout.java:215) at okio.RealBufferedSource.indexOf(RealBufferedSource.java:306) at okio.RealBufferedSource.indexOf(RealBufferedSource.java:300) at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:196) at com.squareup.okhttp.internal.http.Http1xStream.readResponse(Http1xStream.java:186) at com.squareup.okhttp.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:127) at com.squareup.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:737) at com.squareup.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:609) at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:460) at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:399) at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getResponseCode(HttpURLConnectionImpl.java:527) at com.squareup.okhttp.internal.huc.DelegatingHttpsURLConnection.getResponseCode(DelegatingHttpsURLConnection.java:105) at com.squareup.okhttp.internal.huc.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:25) at org.kohsuke.github.Requester.parse(Requester.java:586) ... 29 more Caused by: java.net.SocketException: Socket closed at java.net.SocketInputStream.read(SocketInputStream.java:204) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at sun.security.ssl.InputRecord.read(InputRecord.java:503) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930) at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) at okio.Okio$2.read(Okio.java:139) at okio.AsyncTimeout$2.read(AsyncTimeout.java:211) ... 42 more Finished: FAILURE
- is duplicated by
-
-
- Resolved
-
-
-
- Resolved
-
- is related to
-
JENKINS-48406 Git webhooks should retry
-
- Open
-
- links to
(1 links to)
[JENKINS-45142] Appears a timeout isn't being handled properly: "Server returned HTTP response code: -1, message: 'null' for URL"
If a sleep is not something that would be a "good thing" universally, could we at least have a property to insert a sleep between retries?
Evan Borgstrom
added a comment - - edited If a sleep is not something that would be a "good thing" universally, could we at least have a property to insert a sleep between retries? Evan Borgstrom
added a comment - Today has been REALLY bad for this. We're now getting failures responding to webhooks:
[Thu Aug 17 15:56:33 PDT 2017] Push event to branch THE-BRANCH in repository ORG/REPO UPDATED event from 192.30.252.40 ⇒ https://JENKINSHOST/github-webhook/ with timestamp Thu Aug 17 15:56:27 PDT 2017 processed in 1.3 sec
Checking branch THE-BRANCH
Getting remote pull request #530...
1 branches were processed
Checking pull-requests...
ERROR: Server returned HTTP response code: -1, message: 'null' for URL: https://api.github.com/users/NerdWallet
java.net.SocketException: Socket closed
CC jglick, stephenconnolly – Can anything be done to help get this reprioritized to get looked at again? We can't do a full org scan and quite a few of our web hooks failed today.
Evan Borgstrom
added a comment - Today has been REALLY bad for this. We're now getting failures responding to webhooks:
[Thu Aug 17 15:56:33 PDT 2017] Push event to branch THE-BRANCH in repository ORG/REPO UPDATED event from 192.30.252.40 ⇒ https://JENKINSHOST/github-webhook/ with timestamp Thu Aug 17 15:56:27 PDT 2017 processed in 1.3 sec
Checking branch THE-BRANCH
Getting remote pull request #530...
1 branches were processed
Checking pull-requests...
ERROR: Server returned HTTP response code: -1, message: 'null' for URL: https://api.github.com/users/NerdWallet
java.net.SocketException: Socket closed
CC jglick , stephenconnolly – Can anything be done to help get this reprioritized to get looked at again? We can't do a full org scan and quite a few of our web hooks failed today. Evan Borgstrom
added a comment - Yesterday I spent some time trying to work around this issue in the github-api plugin as was able to finally complete a full org scan!
FWIW, here is what I changed: https://github.com/kohsuke/github-api/compare/master...borgstrom:workaround-http--1
When I tried to add the retry to parse I could never actually get it to retry correctly. Moving the retry logic to the _to function is what worked.
I'm sure this is not the correct way to fix this, but wanted to share if someone else is in my position and needs to get a full org scan done until a proper fix can be put in place.
Evan Borgstrom
added a comment - Yesterday I spent some time trying to work around this issue in the github-api plugin as was able to finally complete a full org scan!
FWIW, here is what I changed: https://github.com/kohsuke/github-api/compare/master...borgstrom:workaround-http--1
When I tried to add the retry to parse I could never actually get it to retry correctly. Moving the retry logic to the _to function is what worked.
I'm sure this is not the correct way to fix this, but wanted to share if someone else is in my position and needs to get a full org scan done until a proper fix can be put in place. AFAIK using Admin/Write as the trust check should greatly reduce the number of requests.
borgstrom if you have something that works for you, file it as a PR and it can be reviewed whenever someone maintaining this code has time. (Assign the issue to yourself, mark as In Review, create a link to the PR, make sure the PR links back to JIRA.)
Regarding my previous comment: TrustContributors definitely makes a lot of API calls and should be discouraged—this was the original trust implementation prior to GitHub enhancements that allowed TrustPermission to be introduced, which does a more precise job of identifying people with write permission. My claim that TrustPermission should be cheaper is based on reading sources of stephenconnolly’s somewhat weird implementation, in LazyContributorNames.
Evan Borgstrom
added a comment - jglick Sounds good. I will cleanup my branch an file it as a PR.
WRT to TrustContributors I have it set to "Nobody" for forks, so I don't think that's the issue. When it was set to "Contributors" the issue here was definitely exasperated. We've noticed a bunch of GitHub "issues" over the past couple weeks (like very elevated web hook delivery times), so I assume we're just running into some instabilities in their system.
Evan Borgstrom
added a comment - jglick Sounds good. I will cleanup my branch an file it as a PR.
WRT to TrustContributors I have it set to "Nobody" for forks, so I don't think that's the issue. When it was set to "Contributors" the issue here was definitely exasperated. We've noticed a bunch of GitHub "issues" over the past couple weeks (like very elevated web hook delivery times), so I assume we're just running into some instabilities in their system. Evan Borgstrom
added a comment - PR filed: https://github.com/kohsuke/github-api/pull/373
I do not have permissions to assign myself or edit labels.
Evan Borgstrom
added a comment - PR filed: https://github.com/kohsuke/github-api/pull/373
I do not have permissions to assign myself or edit labels. Nobody or Everyone is definitely lower-traffic, Contributors is definitely higher-traffic; I think that From users with Admin or Write permission (the most accurate mode) is relatively cheap.
Erik Dahlinghaus
added a comment - Any updates on this issue? PR on github-api still has not been merged because Evan said it's still not working for him. This is causing issues at my organization.
Erik Dahlinghaus
added a comment - Any updates on this issue? PR on github-api still has not been merged because Evan said it's still not working for him. This is causing issues at my organization. 
We recently upgraded:
After doing so we now see this exception during pretty much every full scan of our GitHub org (~1000 repos):
I've anonymized ORG & REPO here, but it can happen on any of our repos.
We have had 1 successful scan last week, but this week we went to change some more options to leverage the new settings in the plugins and have been unable to get a full scan completed in the past 3 days because of this.
FWIW, changing the pull request trust setting to "Contributors" or "Admin/Write" causes this to happen much more frequently (i.e. it will reliably throw this HTTP -1 code error in the first couple repos, where when we set it to Nobody we can typically get much further – I'm guessing because Nobody skips asking for contributors).
EDIT:
Actually, we are hitting the timeout retry:
But I believe the problem is being exasperated because there is no sleep between tries. Could we simply add a delay of a second or two in the retry handler before parse is re-called?