Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65354

One mis-configured job can be prevent all webhook triggered jobs from triggering

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      We're seeing huge delays in processing events from GitHub (at least 2 hours from PR open to build scheduled).

      PushGHEventSubscriber is where this begins.

      SCMEvent defines a thread pool with a max size of 10.

      I've attached a set of thread dumps, but you can see in this excerpt that all 10 threads are sleeping waiting for rate limiting

      This does not seem correct as we are running with GitHubApps and approximately 15000 api calls are allowed.

      I've checked a few of our GitHub apps and all of them have over 12,000 calls remaining.

      cc Liam Newman Carroll Chiou Devin Nusbaum

      I believe this behaviour only started after we upgraded to https://github.com/jenkinsci/github-branch-source-plugin/releases/tag/github-branch-source-2.10.1.

      I'm going to try either downgrade or revert the specific changes tomorrow to get us past this.

       class org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber$SCMHeadEventImpl Thu Apr 08 12:50:33 UTC 2021 / SCMEvent [#7]sleeping , holding [ 0x00000004b8a01310 ]
       class org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber$SCMHeadEventImpl Thu Apr 08 12:51:10 UTC 2021 / SCMEvent [#6]sleeping , holding [ 0x00000004aa801628 ]
       class org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber$SCMHeadEventImpl Thu Apr 08 12:51:33 UTC 2021 / SCMEvent [#1]sleeping , holding [ 0x0000000489521128 ]
       class org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber$SCMHeadEventImpl Thu Apr 08 12:51:48 UTC 2021 / SCMEvent [#9]sleeping , holding [ 0x00000004aa2001d0 ]
       class org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber$SCMHeadEventImpl Thu Apr 08 12:52:38 UTC 2021 / SCMEvent [#5]sleeping , holding [ 0x00000004a79cbb88 ]
       class org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber$SCMHeadEventImpl Thu Apr 08 12:50:49 UTC 2021 / SCMEvent [#3]sleeping , holding [ 0x0000000488800178 ]
       class org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber$SCMHeadEventImpl Thu Apr 08 12:51:21 UTC 2021 / SCMEvent [#2]sleeping , holding [ 0x0000000494c00220 ]
       class org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber$SCMHeadEventImpl Thu Apr 08 12:51:32 UTC 2021 / SCMEvent [#8]sleeping , holding [ 0x00000004b8a02290 ]
       class org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber$SCMHeadEventImpl Thu Apr 08 12:52:15 UTC 2021 / SCMEvent [#10]sleeping , holding [ 0x0000000488a02170 ]
       class org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber$SCMHeadEventImpl Thu Apr 08 12:52:37 UTC 2021 / SCMEvent [#4]sleeping , holding [ 0x00000004a7961600 ]
      at java.lang.Thread.sleep(java.base@11.0.10/Native Method)
      at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$LocalChecker.waitUntilRateLimit(ApiRateLimitChecker.java:323)
      at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$LocalChecker.checkRateLimit(ApiRateLimitChecker.java:259)
      at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$RateLimitCheckerAdapter.checkRateLimit(ApiRateLimitChecker.java:240)
      at org.kohsuke.github.GitHubRateLimitChecker.checkRateLimit(GitHubRateLimitChecker.java:126)
      at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:392)
      at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:358)
      at org.kohsuke.github.Requester.fetch(Requester.java:76)
      at org.kohsuke.github.GHRepository.read(GHRepository.java:119)
      at org.kohsuke.github.GHPerson.getRepository(GHPerson.java:156)
      at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSource(GitHubSCMNavigator.java:1313)
      at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:915)
      at jenkins.scm.api.SCMNavigator.visitSources(SCMNavigator.java:221)
      at jenkins.branch.OrganizationFolder$SCMEventListenerImpl.onSCMHeadEvent(OrganizationFolder.java:1165)
      at jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:246)
      at jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:229)
      at jenkins.scm.api.SCMEvent$Dispatcher.run(SCMEvent.java:505)
      at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
      at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.10/Executors.java:515)
      at java.lang.Thread.run(java.base@11.0.10/Thread.java:834)
      

        Attachments

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            timja Tim Jacomb
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated: