One mis-configured job can be prevent all webhook triggered jobs from triggering

This issue is archived. You can view it, but you can't modify it. Learn more

XMLWordPrintable

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Minor
    • Component/s: github-branch-source-plugin
    • Environment:

      We're seeing huge delays in processing events from GitHub (at least 2 hours from PR open to build scheduled).

      PushGHEventSubscriber is where this begins.

      SCMEvent defines a thread pool with a max size of 10.

      I've attached a set of thread dumps, but you can see in this excerpt that all 10 threads are sleeping waiting for rate limiting

      This does not seem correct as we are running with GitHubApps and approximately 15000 api calls are allowed.

      I've checked a few of our GitHub apps and all of them have over 12,000 calls remaining.

      cc bitwiseman carroll dnusbaum

      I believe this behaviour only started after we upgraded to https://github.com/jenkinsci/github-branch-source-plugin/releases/tag/github-branch-source-2.10.1.

      I'm going to try either downgrade or revert the specific changes tomorrow to get us past this.

       class org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber$SCMHeadEventImpl Thu Apr 08 12:50:33 UTC 2021 / SCMEvent [#7]sleeping , holding [ 0x00000004b8a01310 ]
 class org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber$SCMHeadEventImpl Thu Apr 08 12:51:10 UTC 2021 / SCMEvent [#6]sleeping , holding [ 0x00000004aa801628 ]
 class org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber$SCMHeadEventImpl Thu Apr 08 12:51:33 UTC 2021 / SCMEvent [#1]sleeping , holding [ 0x0000000489521128 ]
 class org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber$SCMHeadEventImpl Thu Apr 08 12:51:48 UTC 2021 / SCMEvent [#9]sleeping , holding [ 0x00000004aa2001d0 ]
 class org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber$SCMHeadEventImpl Thu Apr 08 12:52:38 UTC 2021 / SCMEvent [#5]sleeping , holding [ 0x00000004a79cbb88 ]
 class org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber$SCMHeadEventImpl Thu Apr 08 12:50:49 UTC 2021 / SCMEvent [#3]sleeping , holding [ 0x0000000488800178 ]
 class org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber$SCMHeadEventImpl Thu Apr 08 12:51:21 UTC 2021 / SCMEvent [#2]sleeping , holding [ 0x0000000494c00220 ]
 class org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber$SCMHeadEventImpl Thu Apr 08 12:51:32 UTC 2021 / SCMEvent [#8]sleeping , holding [ 0x00000004b8a02290 ]
 class org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber$SCMHeadEventImpl Thu Apr 08 12:52:15 UTC 2021 / SCMEvent [#10]sleeping , holding [ 0x0000000488a02170 ]
 class org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber$SCMHeadEventImpl Thu Apr 08 12:52:37 UTC 2021 / SCMEvent [#4]sleeping , holding [ 0x00000004a7961600 ]
at java.lang.Thread.sleep(java.base@11.0.10/Native Method)
at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$LocalChecker.waitUntilRateLimit(ApiRateLimitChecker.java:323)
at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$LocalChecker.checkRateLimit(ApiRateLimitChecker.java:259)
at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$RateLimitCheckerAdapter.checkRateLimit(ApiRateLimitChecker.java:240)
at org.kohsuke.github.GitHubRateLimitChecker.checkRateLimit(GitHubRateLimitChecker.java:126)
at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:392)
at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:358)
at org.kohsuke.github.Requester.fetch(Requester.java:76)
at org.kohsuke.github.GHRepository.read(GHRepository.java:119)
at org.kohsuke.github.GHPerson.getRepository(GHPerson.java:156)
at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSource(GitHubSCMNavigator.java:1313)
at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:915)
at jenkins.scm.api.SCMNavigator.visitSources(SCMNavigator.java:221)
at jenkins.branch.OrganizationFolder$SCMEventListenerImpl.onSCMHeadEvent(OrganizationFolder.java:1165)
at jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:246)
at jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:229)
at jenkins.scm.api.SCMEvent$Dispatcher.run(SCMEvent.java:505)
at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.10/Executors.java:515)
at java.lang.Thread.run(java.base@11.0.10/Thread.java:834)

        1. performanceData.6.output.tar.gz
          4.40 MB
          Tim Jacomb

            Assignee:
            Unassigned
            Reporter:
            Tim Jacomb
            Archiver:
            Jenkins Service Account

              Created:
              Updated:
              Archived: