-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Jenkins 2.462.3
Branch API Plugin 2.1200.v4b_a_3da_2eb_db_4
GitHub Branch Source Plugin 1807.v50351eb_7dd13
When Jenkins receives an incoming webhook that should trigger jobs in OrganizationFolders, it will iterate synchronously over every folder and visit its sources to check for triggerable jobs: https://github.com/jenkinsci/branch-api-plugin/blob/d9f35001c95c40294f9b3e9065f68cb9f2be0351/src/main/java/jenkins/branch/OrganizationFolder.java#L992 (see also line 1048).
When even one OrganizationFolder is misconfigured in a way that it targets e.g. a GitHub Organization, but has no credentials specified (see screenshot), it will use anonymous API calls (for GitHub the quota is 60 / hour). When this quota is reached, the corresponding plugin (in this case GitHub Branch Source Plugin) will pause the thread until quota is replenished: https://github.com/jenkinsci/github-branch-source-plugin/blob/088b5f22c7680d6e4d29ce39542d32f9c2b2998f/src/main/java/org/jenkinsci/plugins/github_branch_source/ApiRateLimitChecker.java#L266
Eventually, all available threads will get blocked by this sleep. When that happens, webhooks are no longer being processed for any folder, even the properly configured ones. This can be observed in the thread monitoring page of Jenkins as N number of threads in TIMED_WAIT state where N equals jenkins.scm.api.SCMEvent.EVENT_THREAD_POOL_SIZE.
In my organization's Jenkins, which has ~20 active OrganizationFolders and handles several webhooks per minute, one such folder disabled webhooks for all folders within an hour of rebooting the instance, hence I'm marking this as Major.
I believe the branch-api-plugin should handle the OrganizationFolder iteration asynchronously, or have some kind of a fail-safe (e.g. timeout) in case handling a Folder blocks the thread. This can happen with other SCM vendors (and corresponding plugins), hence I am opening this issue with this component.
Steps to reproduce
- Configure a GitHub Organization Folder with proper credentials, set up a webhook and verify that it works properly
- Add another GitHub Organization Folder for the same organization, but leave credentials empty
- Trigger webhook ~100 times within 1 hour
- 60 times to saturate anonymous GitHub API quota
- ~10 times to saturate thread pool of default size