Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-68321

SCMEvent threads waiting on rate limit when rate limit isn't close to being hit

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • github-branch-source-plugin
    • None
    • GitHub Branch Source Plugin version 2.11.4

      Our organization is experiencing large delays when jenkins processes webhooks (on the order of minutes up to hours). When looking at a threadDump, the following is observed in all SCMEvent threads:

      class 
org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber$SCMHeadEventImpl
 Wed Apr 20 14:18:16 EDT 2022 / SCMEvent [#4]"class org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber$SCMHeadEventImpl Wed Apr 20 14:18:16 EDT 2022 / SCMEvent [#4]" Id=2608 Group=main TIMED_WAITING
	at java.lang.Thread.sleep(Native Method)
	at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$LocalChecker.waitUntilRateLimit(ApiRateLimitChecker.java:325)
	at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$LocalChecker.checkRateLimit(ApiRateLimitChecker.java:261)
	at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$RateLimitCheckerAdapter.checkRateLimit(ApiRateLimitChecker.java:242)
	at org.kohsuke.github.GitHubRateLimitChecker.checkRateLimit(GitHubRateLimitChecker.java:128)
	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:383)
	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:355)
	at org.kohsuke.github.Requester.fetch(Requester.java:76)
	at org.kohsuke.github.GHRepository.read(GHRepository.java:132)
	at org.kohsuke.github.GHPerson.getRepository(GHPerson.java:146)
	at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSource(GitHubSCMNavigator.java:1389)
	at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:926)
	at jenkins.scm.api.SCMNavigator.visitSources(SCMNavigator.java:221)
	at jenkins.branch.OrganizationFolder$SCMEventListenerImpl.onSCMHeadEvent(OrganizationFolder.java:1049)
	at jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:246)
	at jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:229)
	at jenkins.scm.api.SCMEvent$Dispatcher.run(SCMEvent.java:505)
	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

	Number of locked synchronizers = 1
	- java.util.concurrent.ThreadPoolExecutor$Worker@594bc15a

      However, the rate limit for the service account has not come close to 0. The minimum observed is 3000 out of 5000 remaining. This is observed on dashboards as well as when testing the connection from the jenkins UI.

      We are using public GitHub. The rate limiting strategy is set to "Throttle at/near rate limit". It used to be set to "Normalize API requests", but this exacerbated the problem. 

      Notably, the following is seen in the github branch source logs:

      2022-04-20 20:09:02.440+0000 [id=247607]        INFO    o.j.p.g.ApiRateLimitChecker$RateLimitCheckerAdapter#checkRateLimit: LocalChecker for rate limit was not set for this thread. Configured using system settings.
2022-04-20 20:09:02.512+0000 [id=247609]        INFO    o.j.p.g.ApiRateLimitChecker$RateLimitCheckerAdapter#checkRateLimit: LocalChecker for rate limit was not set for this thread. Configured using system settings.
2022-04-20 20:09:02.512+0000 [id=247608]        INFO    o.j.p.g.ApiRateLimitChecker$RateLimitCheckerAdapter#checkRateLimit: LocalChecker for rate limit was not set for this thread. Configured using system settings.

      The following is also seen every few seconds in the github branch source logs:

      2022-04-20 20:09:21.187+0000 [id=247232]        FINE    jenkins.scm.api.SCMSource#defaultListener: Connecting to https://api.github.com using REDACTED

      Let me know if any other information would be helpful.

            Unassigned Unassigned
            gduffy Glenn Duffy
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: