Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64911

Github Branch Source: rate limit sleep interrupted

    XMLWordPrintable

Details

    Description

      With the latest release of Github Branch Source Plugin, we encounter a rate limit of the authentification flow when token are renewed.

      We use the [Github App authentication|https://docs.cloudbees.com/docs/cloudbees-ci/latest/cloud-admin-guide/github-app-auth]

      The RateLimit settings is "Throttle at/near rate limit".

      The authentification process failed and leave the Job in a failed state, even if, according to our monitoring, it remains many quota available. (13k available requests at the time of exception).

      java.lang.InterruptedException: sleep interrupted
      	at java.lang.Thread.sleep(Native Method)
      	at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$LocalChecker.waitUntilRateLimit(ApiRateLimitChecker.java:283)
      	at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$LocalChecker.checkRateLimit(ApiRateLimitChecker.java:228)
      	at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$RateLimitCheckerAdapter.checkRateLimit(ApiRateLimitChecker.java:201)
      	at org.kohsuke.github.GitHubRateLimitChecker.checkRateLimit(GitHubRateLimitChecker.java:126)
      Caused: java.io.InterruptedIOException: sleep interrupted
      	at org.kohsuke.github.GitHubRateLimitChecker.checkRateLimit(GitHubRateLimitChecker.java:140)
      	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:392)
      Caused: org.kohsuke.github.HttpException: Server returned HTTP response code: -1, message: 'null' for URL: https://api.github.com/app
      	at org.kohsuke.github.GitHubClient.interpretApiError(GitHubClient.java:494)
      	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:414)
      	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:358)
      	at org.kohsuke.github.Requester.fetch(Requester.java:74)
      	at org.kohsuke.github.GitHub.getApp(GitHub.java:1094)
      	at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.generateAppInstallationToken(GitHubAppCredentials.java:169)
      Caused: java.lang.IllegalArgumentException: Couldn't authenticate with GitHub app ID XXXXX
      	at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.generateAppInstallationToken(GitHubAppCredentials.java:171)
      	at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.getPassword(GitHubAppCredentials.java:238)
      	at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.getPassword(GitHubAppCredentials.java:270)
      	at org.jenkinsci.plugins.github_branch_source.Connector.checkApiUrlValidity(Connector.java:323)
      	at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.checkApiUrlValidity(GitHubSCMSource.java:1541)
      	at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.retrieve(GitHubSCMSource.java:1596)
      	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:582)
      	at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:98)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:309)
      	at hudson.model.ResourceController.execute(ResourceController.java:97)
      	at hudson.model.Executor.run(Executor.java:429)
      Finished: FAILURE

       

      Attachments

        Issue Links

          Activity

            timja Tim Jacomb added a comment -

            We're hitting this too:

            09:16:09 09:16:09  Jenkins-Imposed API Limiter: Current quota for Github API usage has 0 remaining (15 over budget). Next quota of 60 in 3 min 26 sec. Sleeping until reset.
            09:16:09 09:16:09  Jenkins is attempting to evenly distribute GitHub API requests. To configure a different rate limiting strategy, such as having Jenkins restrict GitHub API requests only when near or above the GitHub rate limit, go to "GitHub API usage" under "Configure System" in the Jenkins settings.
            09:16:39  ERROR: Checkout failed
            09:16:39  java.lang.InterruptedException: sleep interrupted
            09:16:39  	at java.base/java.lang.Thread.sleep(Native Method)
            09:16:39  	at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$LocalChecker.waitUntilRateLimit(ApiRateLimitChecker.java:283)
            09:16:39  	at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$LocalChecker.checkRateLimit(ApiRateLimitChecker.java:228)
            09:16:39  	at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$RateLimitCheckerAdapter.checkRateLimit(ApiRateLimitChecker.java:201)
            09:16:39  	at org.kohsuke.github.GitHubRateLimitChecker.checkRateLimit(GitHubRateLimitChecker.java:126)
            09:16:39  Caused: java.io.InterruptedIOException: sleep interrupted
            09:16:39  	at org.kohsuke.github.GitHubRateLimitChecker.checkRateLimit(GitHubRateLimitChecker.java:140)
            09:16:39  	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:392)
            09:16:39  Caused: org.kohsuke.github.HttpException: Server returned HTTP response code: -1, message: 'null' for URL: https://api.github.com/app
            09:16:39  	at org.kohsuke.github.GitHubClient.interpretApiError(GitHubClient.java:494)
            09:16:39  	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:414)
            09:16:39  	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:358)
            09:16:39  	at org.kohsuke.github.Requester.fetch(Requester.java:74)
            09:16:39  	at org.kohsuke.github.GitHub.getApp(GitHub.java:1094)
            09:16:39  	at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.generateAppInstallationToken(GitHubAppCredentials.java:169)
            09:16:39  Caused: java.lang.IllegalArgumentException: Couldn't authenticate with GitHub app ID 52960
            09:16:39  	at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.generateAppInstallationToken(GitHubAppCredentials.java:171)
            09:16:39  	at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.getPassword(GitHubAppCredentials.java:238)
            09:16:39  	at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.getPassword(GitHubAppCredentials.java:270)
            09:16:39  	at org.jenkinsci.plugins.github_branch_source.Connector.checkApiUrlValidity(Connector.java:323)
            09:16:39  	at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.checkApiUrlValidity(GitHubSCMSource.java:1541)
            09:16:39  	at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.retrieve(GitHubSCMSource.java:1334)
            09:16:39  	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:636)
            09:16:39  	at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.lambda$retrieve$0(SCMSourceRetriever.java:92)
            09:16:39  	at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrySCMOperation(SCMSourceRetriever.java:107)
            09:16:39  	at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:92)
            09:16:39  	at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:157)
            09:16:39  	at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:138)
            09:16:39  	at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125)
            09:16:39  	at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065)
            09:16:39  	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
            09:16:39  	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
            09:16:39  	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
            09:16:39  	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
            09:16:39  	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
            09:16:39  	at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
            09:16:39  	at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
            09:16:39  	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142)
            09:16:39  	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
            09:16:39  	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:571)
            09:16:39  	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:523)
            09:16:39  	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:337)
            09:16:39  	at hudson.model.ResourceController.execute(ResourceController.java:97)
            09:16:39  	at hudson.model.Executor.run(Executor.java:429)
            09:16:39  ERROR: Maximum checkout retry attempts reached, aborting
            
            timja Tim Jacomb added a comment - We're hitting this too: 09:16:09 09:16:09 Jenkins-Imposed API Limiter: Current quota for Github API usage has 0 remaining (15 over budget). Next quota of 60 in 3 min 26 sec. Sleeping until reset. 09:16:09 09:16:09 Jenkins is attempting to evenly distribute GitHub API requests. To configure a different rate limiting strategy, such as having Jenkins restrict GitHub API requests only when near or above the GitHub rate limit, go to "GitHub API usage" under "Configure System " in the Jenkins settings. 09:16:39 ERROR: Checkout failed 09:16:39 java.lang.InterruptedException: sleep interrupted 09:16:39 at java.base/java.lang. Thread .sleep(Native Method) 09:16:39 at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$LocalChecker.waitUntilRateLimit(ApiRateLimitChecker.java:283) 09:16:39 at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$LocalChecker.checkRateLimit(ApiRateLimitChecker.java:228) 09:16:39 at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$RateLimitCheckerAdapter.checkRateLimit(ApiRateLimitChecker.java:201) 09:16:39 at org.kohsuke.github.GitHubRateLimitChecker.checkRateLimit(GitHubRateLimitChecker.java:126) 09:16:39 Caused: java.io.InterruptedIOException: sleep interrupted 09:16:39 at org.kohsuke.github.GitHubRateLimitChecker.checkRateLimit(GitHubRateLimitChecker.java:140) 09:16:39 at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:392) 09:16:39 Caused: org.kohsuke.github.HttpException: Server returned HTTP response code: -1, message: ' null ' for URL: https: //api.github.com/app 09:16:39 at org.kohsuke.github.GitHubClient.interpretApiError(GitHubClient.java:494) 09:16:39 at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:414) 09:16:39 at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:358) 09:16:39 at org.kohsuke.github.Requester.fetch(Requester.java:74) 09:16:39 at org.kohsuke.github.GitHub.getApp(GitHub.java:1094) 09:16:39 at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.generateAppInstallationToken(GitHubAppCredentials.java:169) 09:16:39 Caused: java.lang.IllegalArgumentException: Couldn't authenticate with GitHub app ID 52960 09:16:39 at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.generateAppInstallationToken(GitHubAppCredentials.java:171) 09:16:39 at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.getPassword(GitHubAppCredentials.java:238) 09:16:39 at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.getPassword(GitHubAppCredentials.java:270) 09:16:39 at org.jenkinsci.plugins.github_branch_source.Connector.checkApiUrlValidity(Connector.java:323) 09:16:39 at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.checkApiUrlValidity(GitHubSCMSource.java:1541) 09:16:39 at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.retrieve(GitHubSCMSource.java:1334) 09:16:39 at jenkins.scm.api.SCMSource.fetch(SCMSource.java:636) 09:16:39 at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.lambda$retrieve$0(SCMSourceRetriever.java:92) 09:16:39 at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrySCMOperation(SCMSourceRetriever.java:107) 09:16:39 at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:92) 09:16:39 at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:157) 09:16:39 at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:138) 09:16:39 at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125) 09:16:39 at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065) 09:16:39 at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603) 09:16:39 at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) 09:16:39 at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) 09:16:39 at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) 09:16:39 at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) 09:16:39 at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) 09:16:39 at groovy.lang.GroovyShell.parse(GroovyShell.java:700) 09:16:39 at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142) 09:16:39 at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127) 09:16:39 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:571) 09:16:39 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:523) 09:16:39 at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:337) 09:16:39 at hudson.model.ResourceController.execute(ResourceController.java:97) 09:16:39 at hudson.model.Executor.run(Executor.java:429) 09:16:39 ERROR: Maximum checkout retry attempts reached, aborting
            bitwiseman Liam Newman added a comment -

            I'll look at this tomorrow.

            bitwiseman Liam Newman added a comment - I'll look at this tomorrow.
            bitwiseman Liam Newman added a comment -

            I have a mitigation PR up.
            You can give it a try once it produces an incremental.

            NOTE: https://issues.jenkins.io/browse/JENKINS-62249 is the underlying source of this issue. If you want to collect logs as described there before trying this fix, that would be helpful.

            bitwiseman Liam Newman added a comment - I have a mitigation PR up. You can give it a try once it produces an incremental. NOTE: https://issues.jenkins.io/browse/JENKINS-62249 is the underlying source of this issue. If you want to collect logs as described there before trying this fix, that would be helpful.

            People

              bitwiseman Liam Newman
              vibioh Vincent
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: