We are using Jenkins for large scale mutli-team projects, where CI jobs are done using Jenkins pipeline scripts stored in a multitude of Git repositories hosted on a Github Enterprise server. Right now our configuration has a single Github org containing all projects (many hundreds of projects), and we have a multi-branch pipeline job set up on Jenkins that indexes this org to pull in new repositories, branches and tags.

The mechanics of this configuration work alright, except for the fact that the org scanning logic regularly exhausts the number of available API requests to the Github server, for each scan of the org. For context, there are over 500 repositories in this org, and some of these repos have hundreds upon hundreds of branches, and even more tags ... all of which get indexed by the multi-branch pipeline job. Based on the logs, it appears as though a separate API request is made for each repository, each branch and each tag, which easily surpasses the available 5000 queries per hour that we are allotted. This has the adverse effect of slowing down the branch / repository indexing logic so much that we can barely get one, full reindex of the org per day.

We have tried increasing the cache size of the Git plugin, and enabling all the "throttling" options provided by the web UI, which helps improve stability but it does nothing for performance.

Is there any way you could rework the logic in this plugin, perhaps through the use of a GraphQL query to the Github API, so that you could minimize the number of API calls per-org (or even per-repo) to help alleviate these bottlenecks. The fact that it requires tens of thousands of API calls per-index seems excessive, even in the best of circumstances.

Conversely, if there are any hidden easter eggs or hacks we can leverage to help improve these behaviors, without having to make invasive changes to the plugin, please feel free to share.