If you are using a valid API token, I am not sure what can be done about this other than complaining to GitHub that your rate limit is too low.

Could it make some sub 5000 number of requests, remember where it was, and pick it up again in the next hour? Doesn't the other github plugin do something like that?

I'm seeing the rate limit exhausted on a repo with just a hundred or so branches - this repo has a similar number of PRs, so the existing options about what gets built don't provide me with any help.

Is there any way I can control the frequency of branch indexing? It seems to be happening far, far too frequently for this repo (every few minutes by the looks?!)

Is it doing something silly like triggering a full branch indexing run when it gets a 'new branch created' webhook? Otherwise I can't see why it would need to reindex so often.

Seems to be exacerbated by lack of visibility into how often branch indexing is triggering (you can only see the last run, not the frequency or history of runs, so it's easy to miss the fact it's scanning every few minutes, and consuming vast amounts of API requests.)

Reading https://github.com/jenkinsci/github-branch-source-plugin/blob/master/src/main/java/org/jenkinsci/plugins/github_branch_source/PullRequestGHEventSubscriber.java#L100 - it certainly appears than any PR event will cause a full branch reindexing. Is that right?

Well, given that the downside of this is noted at JENKINS-34600, it's definitely not fair to complain to Github about that jglick. You're expected to authenticate the webhook and use the information it contains, not use it as a trigger for an unbounded number of API requests...

Had a quick go at a throwaway implementation of a fix. Mainly a hacky use of ParameterizedJobMixIn.scheduleBuild2 to schedule a new build for the branches specified in a push event.

However, I quickly ran into JENKINS-37920 - GBS's GitHubWebhookListenerImpl cannot receive the information it needs from github-plugin. (I notice github-plugin also correctly deals with the secret if one is configured, so if it weren't for the fact it's so hard to get the relevant information out, we could easily check for an authenticated webhook, parse the list of branches/PRs, and do the right thing)

In the meantime, I've had to completely disable webhooks.

If you simply created a few dozen branches in the multibranch-demo repo, you'd be easily able to reproduce this issue after a few pushes. After all, the set up instructions say to "Add a new webhook, ask to Send me everything" - which is a configuration in which branch indexing will trigger several times as you just click around GitHub e.g. adding labels to PRs.

My hacky fix branch is at https://github.com/vend/github-branch-source-plugin/pull/1 it'll only work for PR builds for now Comments welcome.

In this implementation, each PR event webhook triggers up to two builds (merged/unmerged), and takes about one Github API request to do so (I think, checking the collaborators for .isTrusted? Maybe 2-3). Pushes to a PR that is already open won't trigger a build, which is a shame (need to rewrite this logic into a pull event subscriber too). But editing the PR, or adding/removing a label should.