Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47154

GitHub Rate Limits are compared using the Jenkins master time not the http response's Date

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      Using github-branch-source plugin 2.2.3. For full environment version information see the Environment meta detail at the top of this issue.

      Description of problem

      My multibranch pipeline job currently hangs for 10+ hours before it will apparently allow a build. The scan log output says it's due to me being over budget with my quota. I would like my job to run right away since there's available quota in the rate limit of my token.

      My actual available rate limit

      The limit reset is 17 minutes from now at the time of this writing.

      $ date -d @1506504539
      Wed Sep 27 02:28:59 PDT 2017
      
      $ curl -H "Authorization: token $GH_TOKEN" https://api.github.com/rate_limit
      {
        "resources": {
          "core": {
            "limit": 5000,
            "remaining": 4997,
            "reset": 1506504539
          },
          "search": {
            "limit": 30,
            "remaining": 30,
            "reset": 1506502499
          },
          "graphql": {
            "limit": 5000,
            "remaining": 5000,
            "reset": 1506506039
          }
        },
        "rate": {
          "limit": 5000,
          "remaining": 4997,
          "reset": 1506504539
        }
      }
      

      Scan log error

      My scan log from a multibranch pipeline job including the error:

      Started by user admin
      [Tue Sep 26 22:44:05 UTC 2017] Starting branch indexing...
      22:44:05 Connecting to https://api.github.com using samrocketman/****** (GitHub user and personal access token used by multibranch pipeline jobs for the GitHub API)
      22:44:06 GitHub API Usage: Current quota has 4997 remaining (35557 over budget). Next quota of 5000 in 10 hr. Sleeping for 9 hr 29 min.
      22:47:06 GitHub API Usage: Still sleeping, now only 9 hr 26 min remaining.
      22:50:06 GitHub API Usage: Still sleeping, now only 9 hr 23 min remaining.
      22:53:06 GitHub API Usage: Still sleeping, now only 9 hr 20 min remaining.
      22:56:06 GitHub API Usage: Still sleeping, now only 9 hr 17 min remaining.
      22:59:06 GitHub API Usage: Still sleeping, now only 9 hr 14 min remaining.
      23:02:06 GitHub API Usage: Still sleeping, now only 9 hr 11 min remaining.
      23:05:06 GitHub API Usage: Still sleeping, now only 9 hr 8 min remaining.
      23:08:06 GitHub API Usage: Still sleeping, now only 9 hr 5 min remaining.
      23:11:06 GitHub API Usage: Still sleeping, now only 9 hr 2 min remaining.
      

      See also

        Attachments

          Issue Links

            Activity

            Hide
            cpuid Nathan Sullivan added a comment -

            > because all your Jenkins instances are - per the GitHub ToS - supposed to be using the same API Key

            before reading this - I had always wondered if we could setup multiple GH "bot" accounts with keys to try parallelize things and work around this issue, but the ToS specifically says "You may not share API tokens to exceed GitHub's rate limitations.", so that idea is out  I understand why they specify that, and it's a valid clause.

            would it be possible to implement some form of log of the qty of each type of API call to GitHub (maybe by endpoint?) to see which areas of the usage are hurting the quotas the most? are there places where information can be cached to try cut down on API call volumes?

            or to rephrase - should <someone> be trying to make the plugin's API call usage more lean and efficient as a solution to the root cause?

            Show
            cpuid Nathan Sullivan added a comment - > because all your Jenkins instances are - per the GitHub ToS - supposed to be using the same API Key before reading this - I had always wondered if we could setup multiple GH "bot" accounts with keys to try parallelize things and work around this issue, but the ToS specifically says "You may not share API tokens to exceed GitHub's rate limitations.", so that idea is out  I understand why they specify that, and it's a valid clause. would it be possible to implement some form of log of the qty of each type of API call to GitHub (maybe by endpoint?) to see which areas of the usage are hurting the quotas the most? are there places where information can be cached to try cut down on API call volumes? or to rephrase - should <someone> be trying to make the plugin's API call usage more lean and efficient as a solution to the root cause?
            Hide
            ingwar Karol Lassak added a comment -

            I dont think its plugin issue..

            I found out BUG in GH api that sometimes shift reset time..
            I created even ticket for GH.. but so far they did nothing about it..

            Script that allow checking bug yourself (sometimes it needs few H to catch it for us)..

            def client = // here configure your client
            def oldStats = client.getRateLimit()
            while (true) {
                GHRateLimit stats = client.getRateLimit()
                if (oldStats.remaining < 4800 && stats.remaining < oldStats.remaining && stats.resetDate > oldStats.resetDate) {
                   log.error("Rate limits error: ${client.myself.login}\nold: ${oldStats}\nnew: ${stats}")
                }
                sleep 10000
            }
            

             

            And results are:

            [main] ERROR java.lang.Class - Rate limits error: xx
            old: GHRateLimit{remaining=4355, limit=5000, resetDate=Wed Sep 04 00:01:33 CEST 2019}
            new: GHRateLimit{remaining=4303, limit=5000, resetDate=Wed Sep 04 00:04:58 CEST 2019}
            [main] ERROR java.lang.Class - Rate limits error: xx
            old: GHRateLimit{remaining=1258, limit=5000, resetDate=Wed Sep 04 00:04:58 CEST 2019}
            new: GHRateLimit{remaining=1205, limit=5000, resetDate=Wed Sep 04 00:16:08 CEST 2019}
            [main] ERROR java.lang.Class - Rate limits error: xx
            old: GHRateLimit{remaining=3888, limit=5000, resetDate=Wed Sep 04 00:38:51 CEST 2019}
            new: GHRateLimit{remaining=3856, limit=5000, resetDate=Wed Sep 04 00:43:13 CEST 2019}

            As you see remaining limit was not reset but time was shifted.. and in that time our jenkins just stop waiting for quota..

             

             

            Show
            ingwar Karol Lassak added a comment - I dont think its plugin issue.. I found out BUG in GH api that sometimes shift reset time.. I created even ticket for GH.. but so far they did nothing about it.. Script that allow checking bug yourself (sometimes it needs few H to catch it for us).. def client = // here configure your client def oldStats = client.getRateLimit() while ( true ) { GHRateLimit stats = client.getRateLimit() if (oldStats.remaining < 4800 && stats.remaining < oldStats.remaining && stats.resetDate > oldStats.resetDate) { log.error( "Rate limits error: ${client.myself.login}\nold: ${oldStats}\nnew: ${stats}" ) } sleep 10000 }   And results are: [main] ERROR java.lang.Class - Rate limits error: xx old: GHRateLimit{remaining=4355, limit=5000, resetDate=Wed Sep 04 00:01:33 CEST 2019} new: GHRateLimit{remaining=4303, limit=5000, resetDate=Wed Sep 04 00:04:58 CEST 2019} [main] ERROR java.lang.Class - Rate limits error: xx old: GHRateLimit{remaining=1258, limit=5000, resetDate=Wed Sep 04 00:04:58 CEST 2019} new: GHRateLimit{remaining=1205, limit=5000, resetDate=Wed Sep 04 00:16:08 CEST 2019} [main] ERROR java.lang.Class - Rate limits error: xx old: GHRateLimit{remaining=3888, limit=5000, resetDate=Wed Sep 04 00:38:51 CEST 2019} new: GHRateLimit{remaining=3856, limit=5000, resetDate=Wed Sep 04 00:43:13 CEST 2019} As you see remaining limit was not reset but time was shifted.. and in that time our jenkins just stop waiting for quota..    
            Hide
            mshade Mike Shade added a comment - - edited

            I have yet to see an actual project where a larger burst results in better behaviour.

             

            Consider a project (or org) whose API key usage is shared across multiple tools (as the GH TOS require). What we're finding is that sometimes when another tool consumes a burst of requests, Jenkins is rate limiting itself to the overall detriment. We may never drop below, say, 3k available API requests, yet are unable to spend them on queued jobs in Jenkins. It's maddening to hit this and not have a TOS-compliant workaround available within the plugin.

            Any chance of reconsidering a simple toggle to disable the rate limit function? Forking and separately maintaining the plugin for this purpose seems excessive.

             

            Show
            mshade Mike Shade added a comment - - edited I have yet to see an actual project where a larger burst results in better behaviour.   Consider a project (or org) whose API key usage is shared across multiple tools (as the GH TOS require). What we're finding is that sometimes when another tool consumes a burst of requests, Jenkins is rate limiting itself to the overall detriment. We may never drop below, say, 3k available API requests, yet are unable to spend them on queued jobs in Jenkins. It's maddening to hit this and not have a TOS-compliant workaround available within the plugin. Any chance of reconsidering a simple toggle to disable the rate limit function? Forking and separately maintaining the plugin for this purpose seems excessive.  
            Hide
            bitwiseman Liam Newman added a comment - - edited

            As for disabling the rate limit function we've already got a PR in the works to provide a simpler implementation.  https://github.com/jenkinsci/github-branch-source-plugin/pull/242  But the last bit of testing seems to have stalled out.  Helping get this PR over the line would seem to be a much better use of your time than forking.

            However, not checking it is not an option.  If you exceed the rate limit, GitHub will make you feel their pain.

             

             

             

            Show
            bitwiseman Liam Newman added a comment - - edited As for disabling the rate limit function we've already got a PR in the works to provide a simpler implementation.  https://github.com/jenkinsci/github-branch-source-plugin/pull/242   But the last bit of testing seems to have stalled out.  Helping get this PR over the line would seem to be a much better use of your time than forking. However, not checking it is not an option.  If you exceed the rate limit, GitHub will make you feel their pain.      
            Hide
            bitwiseman Liam Newman added a comment -

            Mike Shade Stephen Connolly

            Here's a PR to address the time synchronization issue:

            https://github.com/github-api/github-api/pull/595 

            Show
            bitwiseman Liam Newman added a comment - Mike Shade Stephen Connolly Here's a PR to address the time synchronization issue: https://github.com/github-api/github-api/pull/595  

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              sag47 Sam Gleske
              Votes:
              3 Vote for this issue
              Watchers:
              13 Start watching this issue

                Dates

                Created:
                Updated: