[JENKINS-41112] GitHub Branch Source should throttle calls to stay below rate limit

Type: Improvement
Resolution: Duplicate
Priority: Critical
Component/s: github-branch-source-plugin
Labels:
None
Environment:
2.0.0

Similar Issues:
Powered by SuggestiMate

Show

As a user, I would like to be able to use GitHub Branch Source plugin on big GitHub organizations.
Right now, though it's a lot better, we still hit the wall of the rate limit.

The workaround could be to specify a Repository name pattern but that wouldn't work for all our cases because we don't have naming rules that would enable that.

I think the plugin could for example:

check the number of repositories of the requested organization (and the given pattern)
do a rough calculation of the number of API calls
warn the user it's going to take (far) more time than pure CPU/network capacities should enable to be able to complete the operation.

Example:

say I need 10 API calls per repo and I find the user has 1000. This would be 10k calls, instead of the max of 5k. Without even counting the probable other calls coming from some other plugins and so on.

So, the plugin would regularly sleep from time to time to throttle and stay under the radar.

Here is an example of what the scanning provoked in term of API calls after I created a new Org Folder and clicked on Save:

$ while true
do curl -s -u USER:REDACTED_PAT https://api.github.com/rate_limit | jq -r .rate.remaining
sleep 10
done
4996
4996
4742
4518
4316
4072
3884
3707
3320
2800
2211
1895
1685
1466
1231
952
685
454
204
0
0

And sure, when 0 was reached, the plugin started failing as expected:

Started
[Mon Jan 16 20:50:34 UTC 2017] Starting organization scan...
[Mon Jan 16 20:50:34 UTC 2017] Updating actions...
Looking up details of ORG...
ERROR: [Mon Jan 16 20:50:34 UTC 2017] Could not refresh actions for navigator org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator@1164b181
org.jenkinsci.plugins.github_branch_source.RateLimitExceededException: GitHub API rate limit exceeded
	at org.jenkinsci.plugins.github_branch_source.Connector$1.onError(Connector.java:259)
	at org.kohsuke.github.Requester.handleApiError(Requester.java:649)
	at org.kohsuke.github.Requester._to(Requester.java:284)
	at org.kohsuke.github.Requester.to(Requester.java:225)
	at org.kohsuke.github.GitHub.checkApiUrlValidity(GitHub.java:669)
	at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.retrieveActions(GitHubSCMNavigator.java:476)
	at jenkins.scm.api.SCMNavigator.fetchActions(SCMNavigator.java:315)
	at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:371)
	at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:187)
	at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:139)
	at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:814)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:410)
[Mon Jan 16 20:50:34 UTC 2017] Consulting GitHub Organization
ERROR: [Mon Jan 16 20:50:34 UTC 2017] Could not fetch sources from navigator org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator@1164b181
org.jenkinsci.plugins.github_branch_source.RateLimitExceededException: GitHub API rate limit exceeded
	at org.jenkinsci.plugins.github_branch_source.Connector$1.onError(Connector.java:259)
	at org.kohsuke.github.Requester.handleApiError(Requester.java:649)
	at org.kohsuke.github.Requester._to(Requester.java:284)
	at org.kohsuke.github.Requester.to(Requester.java:225)
	at org.kohsuke.github.GitHub.checkApiUrlValidity(GitHub.java:669)
	at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:265)
	at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:420)
	at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:187)
	at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:139)
	at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:814)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:410)
[Mon Jan 16 20:50:34 UTC 2017] Finished organization scan. Scan took 0.24 sec
FATAL: Failed to recompute children of Plugins » Operations center all the things
org.jenkinsci.plugins.github_branch_source.RateLimitExceededException: GitHub API rate limit exceeded
	at org.jenkinsci.plugins.github_branch_source.Connector$1.onError(Connector.java:259)
	at org.kohsuke.github.Requester.handleApiError(Requester.java:649)
	at org.kohsuke.github.Requester._to(Requester.java:284)
	at org.kohsuke.github.Requester.to(Requester.java:225)
	at org.kohsuke.github.GitHub.checkApiUrlValidity(GitHub.java:669)
	at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:265)
	at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:420)
	at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:187)
	at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:139)
	at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:814)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:410)
Finished: FAILURE

relates to

JENKINS-36121 Github Branch Source plugin trips api rate limit

Closed

Baptiste Mathus created issue - 2017-01-16 21:01

Baptiste Mathus made changes - 2017-01-16 21:10

Environment

New: 2.0.0

Baptiste Mathus made changes - 2017-01-16 21:10

Description

Original: As a user, I would like to be able to use GitHub Branch Source plugin on big GitHub organizations.
Right now, though it's *a lot* better, we still hit the wall of the rate limit.

The workaround could be to specify a {{Repository name pattern}} but that wouldn't work for all our cases because we don't have naming rules that would enable that.

I think the plugin could for example:
* check the number of repositories of the requested organization (and the given pattern)
* do a rough calculation of the number of API calls
* warn the user it's going to take (far) more time than pure CPU/network capacities should enable to be able to complete the operation.

Example:

say I need 10 API calls per repo and I find the user has 1000. This would be 10k calls, instead of the max of 5k. Without even counting the probable other calls coming from some other plugins and so on.

So, the plugin would regularly _sleep_ from time to time to throttle and stay under the radar.

Here is an example of what the scanning provoked in term of API calls after I created a new Org Folder and clicked on Save:

{code:java}
$ while true
do curl -s -u USER:REDACTED_PAT https://api.github.com/rate_limit | jq -r .rate.remaining
sleep 10
done
4996
4996
4742
4518
4316
4072
3884
3707
3320
2800
2211
1895
1685
1466
1231
952
685
454
204
0
0
{code}

And sure, when 0 was reached, the plugin started failing as expected:

{noformat}
Started
[Mon Jan 16 20:50:34 UTC 2017] Starting organization scan...
[Mon Jan 16 20:50:34 UTC 2017] Updating actions...
Looking up details of ORG...
ERROR: [Mon Jan 16 20:50:34 UTC 2017] Could not refresh actions for navigator org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator@1164b181
org.jenkinsci.plugins.github_branch_source.RateLimitExceededException: GitHub API rate limit exceeded
at org.jenkinsci.plugins.github_branch_source.Connector$1.onError(Connector.java:259)
at org.kohsuke.github.Requester.handleApiError(Requester.java:649)
at org.kohsuke.github.Requester._to(Requester.java:284)
at org.kohsuke.github.Requester.to(Requester.java:225)
at org.kohsuke.github.GitHub.checkApiUrlValidity(GitHub.java:669)
at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.retrieveActions(GitHubSCMNavigator.java:476)
at jenkins.scm.api.SCMNavigator.fetchActions(SCMNavigator.java:315)
at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:371)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:187)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:139)
at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:814)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
[Mon Jan 16 20:50:34 UTC 2017] Consulting GitHub Organization
ERROR: [Mon Jan 16 20:50:34 UTC 2017] Could not fetch sources from navigator org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator@1164b181
org.jenkinsci.plugins.github_branch_source.RateLimitExceededException: GitHub API rate limit exceeded
at org.jenkinsci.plugins.github_branch_source.Connector$1.onError(Connector.java:259)
at org.kohsuke.github.Requester.handleApiError(Requester.java:649)
at org.kohsuke.github.Requester._to(Requester.java:284)
at org.kohsuke.github.Requester.to(Requester.java:225)
at org.kohsuke.github.GitHub.checkApiUrlValidity(GitHub.java:669)
at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:265)
at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:420)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:187)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:139)
at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:814)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
[Mon Jan 16 20:50:34 UTC 2017] Finished organization scan. Scan took 0.24 sec
FATAL: Failed to recompute children of Plugins » Operations center all the things
org.jenkinsci.plugins.github_branch_source.RateLimitExceededException: GitHub API rate limit exceeded
at org.jenkinsci.plugins.github_branch_source.Connector$1.onError(Connector.java:259)
at org.kohsuke.github.Requester.handleApiError(Requester.java:649)
at org.kohsuke.github.Requester._to(Requester.java:284)
at org.kohsuke.github.Requester.to(Requester.java:225)
at org.kohsuke.github.GitHub.checkApiUrlValidity(GitHub.java:669)
at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:265)
at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:420)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:187)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:139)
at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:814)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Finished: FAILURE
{noformat}

New: As a user, I would like to be able to use GitHub Branch Source plugin on big GitHub organizations.
Right now, though it's *a lot* better, we still hit the wall of the rate limit.

The workaround could be to specify a {{Repository name pattern}} but that wouldn't work for all our cases because we don't have naming rules that would enable that.

I think the plugin could for example:
* check the number of repositories of the requested organization (and the given pattern)
* do a rough calculation of the number of API calls
* warn the user it's going to take (far) more time than pure CPU/network capacities should enable to be able to complete the operation.

Example:

say I need 10 API calls per repo and I find the user has 1000. This would be 10k calls, instead of the max of 5k. Without even counting the probable other calls coming from some other plugins and so on.

So, the plugin would regularly _sleep_ from time to time to throttle and stay under the radar.

Here is an example of what the scanning provoked in term of API calls after I created a new Org Folder and clicked on Save:

{code:java}
$ while true
do curl -s -u USER:REDACTED_PAT https://api.github.com/rate_limit | jq -r .rate.remaining
sleep 10
done
4996
4996
4742
4518
4316
4072
3884
3707
3320
2800
2211
1895
1685
1466
1231
952
685
454
204
0
0
{code}

And sure, when 0 was reached, the plugin started failing as _expected_:

{noformat}
Started
[Mon Jan 16 20:50:34 UTC 2017] Starting organization scan...
[Mon Jan 16 20:50:34 UTC 2017] Updating actions...
Looking up details of ORG...
ERROR: [Mon Jan 16 20:50:34 UTC 2017] Could not refresh actions for navigator org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator@1164b181
org.jenkinsci.plugins.github_branch_source.RateLimitExceededException: GitHub API rate limit exceeded
at org.jenkinsci.plugins.github_branch_source.Connector$1.onError(Connector.java:259)
at org.kohsuke.github.Requester.handleApiError(Requester.java:649)
at org.kohsuke.github.Requester._to(Requester.java:284)
at org.kohsuke.github.Requester.to(Requester.java:225)
at org.kohsuke.github.GitHub.checkApiUrlValidity(GitHub.java:669)
at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.retrieveActions(GitHubSCMNavigator.java:476)
at jenkins.scm.api.SCMNavigator.fetchActions(SCMNavigator.java:315)
at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:371)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:187)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:139)
at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:814)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
[Mon Jan 16 20:50:34 UTC 2017] Consulting GitHub Organization
ERROR: [Mon Jan 16 20:50:34 UTC 2017] Could not fetch sources from navigator org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator@1164b181
org.jenkinsci.plugins.github_branch_source.RateLimitExceededException: GitHub API rate limit exceeded
at org.jenkinsci.plugins.github_branch_source.Connector$1.onError(Connector.java:259)
at org.kohsuke.github.Requester.handleApiError(Requester.java:649)
at org.kohsuke.github.Requester._to(Requester.java:284)
at org.kohsuke.github.Requester.to(Requester.java:225)
at org.kohsuke.github.GitHub.checkApiUrlValidity(GitHub.java:669)
at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:265)
at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:420)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:187)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:139)
at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:814)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
[Mon Jan 16 20:50:34 UTC 2017] Finished organization scan. Scan took 0.24 sec
FATAL: Failed to recompute children of Plugins » Operations center all the things
org.jenkinsci.plugins.github_branch_source.RateLimitExceededException: GitHub API rate limit exceeded
at org.jenkinsci.plugins.github_branch_source.Connector$1.onError(Connector.java:259)
at org.kohsuke.github.Requester.handleApiError(Requester.java:649)
at org.kohsuke.github.Requester._to(Requester.java:284)
at org.kohsuke.github.Requester.to(Requester.java:225)
at org.kohsuke.github.GitHub.checkApiUrlValidity(GitHub.java:669)
at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:265)
at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:420)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:187)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:139)
at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:814)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Finished: FAILURE
{noformat}

Stephen Connolly made changes - 2017-01-16 23:01

Link

New: This issue relates to ~~JENKINS-36121~~ [ ~~JENKINS-36121~~ ]

Stephen Connolly made changes - 2017-01-16 23:01

Resolution		New: Duplicate [ 3 ]
Status	Original: Open [ 1 ]	New: Closed [ 6 ]

Jenkins

Details

Description

Attachments

Issue Links

Activity

People

Dates