Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64911

Github Branch Source: rate limit sleep interrupted

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      With the latest release of Github Branch Source Plugin, we encounter a rate limit of the authentification flow when token are renewed.

      We use the [Github App authentication|https://docs.cloudbees.com/docs/cloudbees-ci/latest/cloud-admin-guide/github-app-auth]

      The RateLimit settings is "Throttle at/near rate limit".

      The authentification process failed and leave the Job in a failed state, even if, according to our monitoring, it remains many quota available. (13k available requests at the time of exception).

      java.lang.InterruptedException: sleep interrupted
      	at java.lang.Thread.sleep(Native Method)
      	at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$LocalChecker.waitUntilRateLimit(ApiRateLimitChecker.java:283)
      	at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$LocalChecker.checkRateLimit(ApiRateLimitChecker.java:228)
      	at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$RateLimitCheckerAdapter.checkRateLimit(ApiRateLimitChecker.java:201)
      	at org.kohsuke.github.GitHubRateLimitChecker.checkRateLimit(GitHubRateLimitChecker.java:126)
      Caused: java.io.InterruptedIOException: sleep interrupted
      	at org.kohsuke.github.GitHubRateLimitChecker.checkRateLimit(GitHubRateLimitChecker.java:140)
      	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:392)
      Caused: org.kohsuke.github.HttpException: Server returned HTTP response code: -1, message: 'null' for URL: https://api.github.com/app
      	at org.kohsuke.github.GitHubClient.interpretApiError(GitHubClient.java:494)
      	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:414)
      	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:358)
      	at org.kohsuke.github.Requester.fetch(Requester.java:74)
      	at org.kohsuke.github.GitHub.getApp(GitHub.java:1094)
      	at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.generateAppInstallationToken(GitHubAppCredentials.java:169)
      Caused: java.lang.IllegalArgumentException: Couldn't authenticate with GitHub app ID XXXXX
      	at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.generateAppInstallationToken(GitHubAppCredentials.java:171)
      	at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.getPassword(GitHubAppCredentials.java:238)
      	at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.getPassword(GitHubAppCredentials.java:270)
      	at org.jenkinsci.plugins.github_branch_source.Connector.checkApiUrlValidity(Connector.java:323)
      	at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.checkApiUrlValidity(GitHubSCMSource.java:1541)
      	at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.retrieve(GitHubSCMSource.java:1596)
      	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:582)
      	at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:98)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:309)
      	at hudson.model.ResourceController.execute(ResourceController.java:97)
      	at hudson.model.Executor.run(Executor.java:429)
      Finished: FAILURE

       

        Attachments

          Issue Links

            Activity

            Hide
            timja Tim Jacomb added a comment -

            We're hitting this too:

            09:16:09 09:16:09  Jenkins-Imposed API Limiter: Current quota for Github API usage has 0 remaining (15 over budget). Next quota of 60 in 3 min 26 sec. Sleeping until reset.
            09:16:09 09:16:09  Jenkins is attempting to evenly distribute GitHub API requests. To configure a different rate limiting strategy, such as having Jenkins restrict GitHub API requests only when near or above the GitHub rate limit, go to "GitHub API usage" under "Configure System" in the Jenkins settings.
            09:16:39  ERROR: Checkout failed
            09:16:39  java.lang.InterruptedException: sleep interrupted
            09:16:39  	at java.base/java.lang.Thread.sleep(Native Method)
            09:16:39  	at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$LocalChecker.waitUntilRateLimit(ApiRateLimitChecker.java:283)
            09:16:39  	at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$LocalChecker.checkRateLimit(ApiRateLimitChecker.java:228)
            09:16:39  	at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$RateLimitCheckerAdapter.checkRateLimit(ApiRateLimitChecker.java:201)
            09:16:39  	at org.kohsuke.github.GitHubRateLimitChecker.checkRateLimit(GitHubRateLimitChecker.java:126)
            09:16:39  Caused: java.io.InterruptedIOException: sleep interrupted
            09:16:39  	at org.kohsuke.github.GitHubRateLimitChecker.checkRateLimit(GitHubRateLimitChecker.java:140)
            09:16:39  	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:392)
            09:16:39  Caused: org.kohsuke.github.HttpException: Server returned HTTP response code: -1, message: 'null' for URL: https://api.github.com/app
            09:16:39  	at org.kohsuke.github.GitHubClient.interpretApiError(GitHubClient.java:494)
            09:16:39  	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:414)
            09:16:39  	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:358)
            09:16:39  	at org.kohsuke.github.Requester.fetch(Requester.java:74)
            09:16:39  	at org.kohsuke.github.GitHub.getApp(GitHub.java:1094)
            09:16:39  	at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.generateAppInstallationToken(GitHubAppCredentials.java:169)
            09:16:39  Caused: java.lang.IllegalArgumentException: Couldn't authenticate with GitHub app ID 52960
            09:16:39  	at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.generateAppInstallationToken(GitHubAppCredentials.java:171)
            09:16:39  	at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.getPassword(GitHubAppCredentials.java:238)
            09:16:39  	at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.getPassword(GitHubAppCredentials.java:270)
            09:16:39  	at org.jenkinsci.plugins.github_branch_source.Connector.checkApiUrlValidity(Connector.java:323)
            09:16:39  	at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.checkApiUrlValidity(GitHubSCMSource.java:1541)
            09:16:39  	at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.retrieve(GitHubSCMSource.java:1334)
            09:16:39  	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:636)
            09:16:39  	at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.lambda$retrieve$0(SCMSourceRetriever.java:92)
            09:16:39  	at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrySCMOperation(SCMSourceRetriever.java:107)
            09:16:39  	at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:92)
            09:16:39  	at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:157)
            09:16:39  	at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:138)
            09:16:39  	at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125)
            09:16:39  	at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065)
            09:16:39  	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
            09:16:39  	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
            09:16:39  	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
            09:16:39  	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
            09:16:39  	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
            09:16:39  	at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
            09:16:39  	at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
            09:16:39  	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142)
            09:16:39  	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
            09:16:39  	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:571)
            09:16:39  	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:523)
            09:16:39  	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:337)
            09:16:39  	at hudson.model.ResourceController.execute(ResourceController.java:97)
            09:16:39  	at hudson.model.Executor.run(Executor.java:429)
            09:16:39  ERROR: Maximum checkout retry attempts reached, aborting
            
            Show
            timja Tim Jacomb added a comment - We're hitting this too: 09:16:09 09:16:09 Jenkins-Imposed API Limiter: Current quota for Github API usage has 0 remaining (15 over budget). Next quota of 60 in 3 min 26 sec. Sleeping until reset. 09:16:09 09:16:09 Jenkins is attempting to evenly distribute GitHub API requests. To configure a different rate limiting strategy, such as having Jenkins restrict GitHub API requests only when near or above the GitHub rate limit, go to "GitHub API usage" under "Configure System " in the Jenkins settings. 09:16:39 ERROR: Checkout failed 09:16:39 java.lang.InterruptedException: sleep interrupted 09:16:39 at java.base/java.lang. Thread .sleep(Native Method) 09:16:39 at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$LocalChecker.waitUntilRateLimit(ApiRateLimitChecker.java:283) 09:16:39 at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$LocalChecker.checkRateLimit(ApiRateLimitChecker.java:228) 09:16:39 at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$RateLimitCheckerAdapter.checkRateLimit(ApiRateLimitChecker.java:201) 09:16:39 at org.kohsuke.github.GitHubRateLimitChecker.checkRateLimit(GitHubRateLimitChecker.java:126) 09:16:39 Caused: java.io.InterruptedIOException: sleep interrupted 09:16:39 at org.kohsuke.github.GitHubRateLimitChecker.checkRateLimit(GitHubRateLimitChecker.java:140) 09:16:39 at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:392) 09:16:39 Caused: org.kohsuke.github.HttpException: Server returned HTTP response code: -1, message: ' null ' for URL: https: //api.github.com/app 09:16:39 at org.kohsuke.github.GitHubClient.interpretApiError(GitHubClient.java:494) 09:16:39 at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:414) 09:16:39 at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:358) 09:16:39 at org.kohsuke.github.Requester.fetch(Requester.java:74) 09:16:39 at org.kohsuke.github.GitHub.getApp(GitHub.java:1094) 09:16:39 at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.generateAppInstallationToken(GitHubAppCredentials.java:169) 09:16:39 Caused: java.lang.IllegalArgumentException: Couldn't authenticate with GitHub app ID 52960 09:16:39 at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.generateAppInstallationToken(GitHubAppCredentials.java:171) 09:16:39 at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.getPassword(GitHubAppCredentials.java:238) 09:16:39 at org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.getPassword(GitHubAppCredentials.java:270) 09:16:39 at org.jenkinsci.plugins.github_branch_source.Connector.checkApiUrlValidity(Connector.java:323) 09:16:39 at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.checkApiUrlValidity(GitHubSCMSource.java:1541) 09:16:39 at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.retrieve(GitHubSCMSource.java:1334) 09:16:39 at jenkins.scm.api.SCMSource.fetch(SCMSource.java:636) 09:16:39 at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.lambda$retrieve$0(SCMSourceRetriever.java:92) 09:16:39 at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrySCMOperation(SCMSourceRetriever.java:107) 09:16:39 at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:92) 09:16:39 at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:157) 09:16:39 at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:138) 09:16:39 at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125) 09:16:39 at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065) 09:16:39 at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603) 09:16:39 at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) 09:16:39 at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) 09:16:39 at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) 09:16:39 at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) 09:16:39 at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) 09:16:39 at groovy.lang.GroovyShell.parse(GroovyShell.java:700) 09:16:39 at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142) 09:16:39 at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127) 09:16:39 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:571) 09:16:39 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:523) 09:16:39 at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:337) 09:16:39 at hudson.model.ResourceController.execute(ResourceController.java:97) 09:16:39 at hudson.model.Executor.run(Executor.java:429) 09:16:39 ERROR: Maximum checkout retry attempts reached, aborting
            Hide
            bitwiseman Liam Newman added a comment -

            I'll look at this tomorrow.

            Show
            bitwiseman Liam Newman added a comment - I'll look at this tomorrow.
            Hide
            bitwiseman Liam Newman added a comment -

            I have a mitigation PR up.
            You can give it a try once it produces an incremental.

            NOTE: https://issues.jenkins.io/browse/JENKINS-62249 is the underlying source of this issue. If you want to collect logs as described there before trying this fix, that would be helpful.

            Show
            bitwiseman Liam Newman added a comment - I have a mitigation PR up. You can give it a try once it produces an incremental. NOTE: https://issues.jenkins.io/browse/JENKINS-62249 is the underlying source of this issue. If you want to collect logs as described there before trying this fix, that would be helpful.

              People

              Assignee:
              bitwiseman Liam Newman
              Reporter:
              vibioh Vincent
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: