-
Bug
-
Resolution: Duplicate
-
Major
-
None
-
Jenkins 2.19.1 (LTS)
GitHub API Plugin 1.7.7
GitHub plugin 1.22.1
After moving to Jenkins new LTS release, we found that we were constantly running into our rate limit with github. To start the investigation of this issue, I contacted github support, and requested any information they could provide about our use of the API over the last week.
This was the information github support sent back. (Org and projects redacted)
Hi Greg,
Here are some stats for your account, for the past week.
The top 10 API endpoints being hit:
/repositories/:repository_id/contents/?* 196,620 25.632%
/repositories/:repository_id 109,038 14.215%
/repositories/:repository_id/collaborators 103,631 13.51%
/user 79,102 10.312%
/ 78,743 10.265%
/repositories/:repository_id/branches 77,789 10.141%
/repositories/:repository_id/pulls 77,474 10.1%
/user/:user_id 14,014 1.827%
/repositories/:repository_id/git/refs/* 11,154 1.454%
/repositories/:repository_id/pulls/:id 10,582 1.38%
Top 10 specific resources:
/ 97,571 10.37%
/user 96,932 10.302%
/repos/{org}/{projectA}/contents/ 22,552 2.397%
/repos/{org}/{projectB}/contents/ 21,249 2.258%
/repositories/{repoA}/collaborators 20,004 2.126%
/repositories/{repoB}/collaborators 18,058 1.919%
/users/{org} 17,241 1.832%
/repositories/{repoC}/collaborators 16,538 1.758%
/repos/{org}/{projectC}/contents/ 10,657 1.133%
/repos/{org}/{projectD}/contents/ 9,751 1.036%
There are some very interesting things there, as I'm sure you notice. Over 20% of your total quota is wasted on fetching and re-fetching resources which either never change or change very rarely -- the root endpoint of the API / (which has only links to other resources) and the resource for the current user /user (which changes very rarely). And that /user endpoint is sometimes being called 100 times per minute. I can't imagine why you'd need to fetch those resources so often. Reducing or completely removing that would keep you well below the rate limit.
After getting this report back from GitHub support directly, I started to investigate, and found that it does not appear that the cache file is currently being used, or written to at all.
On my System Configuration page, I have configured GitHub to use the default 20mb cache, but my system has no normal cache directory. (I know it was there before I moved to LTS version, which was recent)
My creds are on the System Configuration page, and verified, and when I click test, I can see what my current rate is. But no cache is being used, and I imagine that is why over 20% of our api requests to github are to static, unchanged values.
This could be related to JENKINS-38935, which I also found while investigating this problem
- is related to
-
-
- Closed
-
[JENKINS-38937] GitHub API cache is not working
Greg Smith
added a comment - With 1.22.2 installed, the problem listed in JENKINS-38935 was resolved.
However, I still do not see any directory in my ${JENKINS_HOME}/org.jenkinsci.plugins.github.GitHubPlugin.cache
It does not appear that fixing the problem with loading the configuration load solved the problem with using / writing the cache.
Is there some other logging I could provide to determine why the cache file is not being created, or confirm if the cache is being used?
Greg Smith
added a comment - With 1.22.2 installed, the problem listed in JENKINS-38935 was resolved.
However, I still do not see any directory in my ${JENKINS_HOME}/org.jenkinsci.plugins.github.GitHubPlugin.cache
It does not appear that fixing the problem with loading the configuration load solved the problem with using / writing the cache.
Is there some other logging I could provide to determine why the cache file is not being created, or confirm if the cache is being used?
Neil Rhine
added a comment - Any updates here or suggestions to mitigate? Our organization is also constantly hitting the API limit on all of the indexing.
Neil Rhine
added a comment - Any updates here or suggestions to mitigate? Our organization is also constantly hitting the API limit on all of the indexing. You can check that cache is used by following steps:
1. Click on "Reregister hooks for all jobs"
2. Check that there is a folder in cache:
$ du -h /opt/jenkins/org.jenkinsci.plugins.github.GitHubPlugin.cache/ 796K /opt/jenkins/org.jenkinsci.plugins.github.GitHubPlugin.cache/76c844e5 800K /opt/jenkins/org.jenkinsci.plugins.github.GitHubPlugin.cache/
3. Set cache size to 0, then save global config
4. Now cache should be cleared:
$ du -h /opt/jenkins/org.jenkinsci.plugins.github.GitHubPlugin.cache/ 4.0K /opt/jenkins/org.jenkinsci.plugins.github.GitHubPlugin.cache/
5. Set cache to any positive num back and resave global config. Then check steps 1-2.
Then please write results. Also please provide full list of plugins (maybe some of them eat your limits without using cache from GitHub plugin's client?)
The cache is being written to after all, just in a very minor capacity, before it's cleared it's only 16kb and since the rate limit is being hit I expected to see more usage. I think it is more related to JENKINS-36121 and less this bug after all, but this is my list of plugins:
Pipeline: API (workflow-api): 2.4 Gradle Plugin (gradle): 1.25 Job Configuration History Plugin (jobConfigHistory): 2.15 Git client plugin (git-client): 2.1.0 JavaScript GUI Lib: ACE Editor bundle plugin (ace-editor): 1.1 Jackson 2 API Plugin (jackson2-api): 2.7.3 LDAP Plugin (ldap): 1.12 Branch API Plugin (branch-api): 1.11.1 Pipeline: SCM Step (workflow-scm-step): 2.2 OWASP Markup Formatter Plugin (antisamy-markup-formatter): 1.5 MapDB API Plugin (mapdb-api): 1.0.9.0 GitHub Pull Request Builder (ghprb): 1.33.1 Structs Plugin (structs): 1.5 Node and Label parameter plugin (nodelabelparameter): 1.7.2 Checkstyle Plug-in (checkstyle): 3.46 Ant Plugin (ant): 1.4 Icon Shim Plugin (icon-shim): 2.0.3 Xvnc plugin (xvnc): 1.24 Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.4 Job DSL (job-dsl): 1.50 Git plugin (git): 3.0.0 Groovy Postbuild (groovy-postbuild): 2.3.1 Throttle Concurrent Builds Plug-in (throttle-concurrents): 1.9.0 Credentials Binding Plugin (credentials-binding): 1.9 Audit Trail (audit-trail): 2.2 xUnit plugin (xunit): 1.102 Pipeline: Step API (workflow-step-api): 2.4 GitHub Organization Folder Plugin (github-organization-folder): 1.5 JavaScript GUI Lib: Handlebars bundle plugin (handlebars): 1.1.1 Post-Build Script Plug-in (postbuildscript): 0.17 Email Extension Plugin (email-ext): 2.47 Display URL API (display-url-api): 0.5 Jira Issue Updater (jenkins-jira-issue-updater): 1.18 CloudBees Amazon Web Services Credentials Plugin (aws-credentials): 1.16 Pipeline: Basic Steps (workflow-basic-steps): 2.1 build timeout plugin (build-timeout): 1.17.1 TAP Plugin (tap): 2.0.1 Pipeline: Supporting APIs (workflow-support): 2.8 GitHub plugin (github): 1.22.4 ShiningPanda Plugin (shiningpanda): 0.23 JIRA plugin (jira): 2.2.1 Xvfb plugin (xvfb): 1.1.3 Pipeline: Build Step (pipeline-build-step): 2.2 Artifactory Plugin (artifactory): 2.6.0 Email Extension Template Plugin (emailext-template): 1.0 Pipeline: Stage Step (pipeline-stage-step): 2.2 GIT server Plugin (git-server): 1.7 Environment Injector Plugin (envinject): 1.92.1 OWASP Dependency-Check Plugin (dependency-check-jenkins-plugin): 1.4.2 Pipeline: Job (workflow-job): 2.6 Token Macro Plugin (token-macro): 1.12.1 Matrix Authorization Strategy Plugin (matrix-auth): 1.4 Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 1.1 Amazon Web Services SDK (aws-java-sdk): 1.11.37 GitHub Integration Plugin (github-pullrequest): 0.1.0-rc12 JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached): 1.2.1 Timestamper (timestamper): 1.8.5 Folders Plugin (cloudbees-folder): 5.12 Dashboard View (dashboard-view): 2.9.10 External Monitor Job Type Plugin (external-monitor-job): 1.6 EZ Templates (ez-templates): 1.2.1 SSH Slaves plugin (ssh-slaves): 1.11 Copy Artifact Plugin (copyartifact): 1.38.1 Pipeline (workflow-aggregator): 2.3 JiraTestResultReporter plugin (JiraTestResultReporter): 2.0.3 HTML Publisher plugin (htmlpublisher): 1.11 Pipeline: Multibranch (workflow-multibranch): 2.9 Port Allocator Plug-in (port-allocator): 1.9-SNAPSHOT (private-61b7bdb7-neil.rhine) Script Security Plugin (script-security): 1.23 PMD Plug-in (pmd): 3.45 JUnit Plugin (junit): 1.18 Pipeline: REST API Plugin (pipeline-rest-api): 2.0 SSH Credentials Plugin (ssh-credentials): 1.12 Javadoc Plugin (javadoc): 1.4 Workspace Cleanup Plugin (ws-cleanup): 0.30 Hudson Post build task (postbuild-task): 1.8 SSH Agent Plugin (ssh-agent): 1.13 Amazon EC2 plugin (ec2): 1.36 GitHub API Plugin (github-api): 1.79 SCM API Plugin (scm-api): 1.3 Static Analysis Utilities (analysis-core): 1.79 Multijob plugin (jenkins-multijob-plugin): 1.22 SAML Plugin (saml): 0.6 GitHub Authentication plugin (github-oauth): 0.24 Plain Credentials Plugin (plain-credentials): 1.2 Conditional BuildStep (conditional-buildstep): 1.3.5 Groovy (groovy): 1.29 Matrix Project Plugin (matrix-project): 1.7.1 Credentials Plugin (credentials): 2.1.4 JaCoCo plugin (jacoco): 2.0.1 user build vars plugin (build-user-vars-plugin): 1.5 Notification plugin (notification): 1.10 JavaScript GUI Lib: Moment.js bundle plugin (momentjs): 1.1.1 Run Condition Plugin (run-condition): 1.0 Mailer Plugin (mailer): 1.17 Parameterized Trigger plugin (parameterized-trigger): 2.32 GitHub Branch Source Plugin (github-branch-source): 1.10 Maven Integration plugin (maven-plugin): 2.13 Node Iterator API Plugin (node-iterator-api): 1.5.0 Pipeline: Groovy (workflow-cps): 2.19 Windows Slaves Plugin (windows-slaves): 1.2 Pipeline: Stage View Plugin (pipeline-stage-view): 2.0 PAM Authentication plugin (pam-auth): 1.3 Ivy Plugin (ivy): 1.26 Subversion Plug-in (subversion): 2.6 Pipeline: Nodes and Processes (workflow-durable-task-step): 2.4 jQuery plugin (jquery): 1.11.2-0 Durable Task Plugin (durable-task): 1.12 Config File Provider Plugin (config-file-provider): 2.11 FindBugs Plug-in (findbugs): 4.65 Pipeline: Input Step (pipeline-input-step): 2.1 Fail The Build Plugin (fail-the-build-plugin): 1.0 bouncycastle API Plugin (bouncycastle-api): 1.648.3 Static Analysis Collector Plug-in (analysis-collector): 1.48
With a jenkins version of 2.7.3
Neil Rhine
added a comment - - edited The cache is being written to after all, just in a very minor capacity, before it's cleared it's only 16kb and since the rate limit is being hit I expected to see more usage. I think it is more related to JENKINS-36121 and less this bug after all, but this is my list of plugins:
Pipeline: API (workflow-api): 2.4
Gradle Plugin (gradle): 1.25
Job Configuration History Plugin (jobConfigHistory): 2.15
Git client plugin (git-client): 2.1.0
JavaScript GUI Lib: ACE Editor bundle plugin (ace-editor): 1.1
Jackson 2 API Plugin (jackson2-api): 2.7.3
LDAP Plugin (ldap): 1.12
Branch API Plugin (branch-api): 1.11.1
Pipeline: SCM Step (workflow-scm-step): 2.2
OWASP Markup Formatter Plugin (antisamy-markup-formatter): 1.5
MapDB API Plugin (mapdb-api): 1.0.9.0
GitHub Pull Request Builder (ghprb): 1.33.1
Structs Plugin (structs): 1.5
Node and Label parameter plugin (nodelabelparameter): 1.7.2
Checkstyle Plug-in (checkstyle): 3.46
Ant Plugin (ant): 1.4
Icon Shim Plugin (icon-shim): 2.0.3
Xvnc plugin (xvnc): 1.24
Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.4
Job DSL (job-dsl): 1.50
Git plugin (git): 3.0.0
Groovy Postbuild (groovy-postbuild): 2.3.1
Throttle Concurrent Builds Plug-in (throttle-concurrents): 1.9.0
Credentials Binding Plugin (credentials-binding): 1.9
Audit Trail (audit-trail): 2.2
xUnit plugin (xunit): 1.102
Pipeline: Step API (workflow-step-api): 2.4
GitHub Organization Folder Plugin (github-organization-folder): 1.5
JavaScript GUI Lib: Handlebars bundle plugin (handlebars): 1.1.1
Post-Build Script Plug-in (postbuildscript): 0.17
Email Extension Plugin (email-ext): 2.47
Display URL API (display-url-api): 0.5
Jira Issue Updater (jenkins-jira-issue-updater): 1.18
CloudBees Amazon Web Services Credentials Plugin (aws-credentials): 1.16
Pipeline: Basic Steps (workflow-basic-steps): 2.1
build timeout plugin (build-timeout): 1.17.1
TAP Plugin (tap): 2.0.1
Pipeline: Supporting APIs (workflow-support): 2.8
GitHub plugin (github): 1.22.4
ShiningPanda Plugin (shiningpanda): 0.23
JIRA plugin (jira): 2.2.1
Xvfb plugin (xvfb): 1.1.3
Pipeline: Build Step (pipeline-build-step): 2.2
Artifactory Plugin (artifactory): 2.6.0
Email Extension Template Plugin (emailext-template): 1.0
Pipeline: Stage Step (pipeline-stage-step): 2.2
GIT server Plugin (git-server): 1.7
Environment Injector Plugin (envinject): 1.92.1
OWASP Dependency-Check Plugin (dependency-check-jenkins-plugin): 1.4.2
Pipeline: Job (workflow-job): 2.6
Token Macro Plugin (token-macro): 1.12.1
Matrix Authorization Strategy Plugin (matrix-auth): 1.4
Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 1.1
Amazon Web Services SDK (aws-java-sdk): 1.11.37
GitHub Integration Plugin (github-pullrequest): 0.1.0-rc12
JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached): 1.2.1
Timestamper (timestamper): 1.8.5
Folders Plugin (cloudbees-folder): 5.12
Dashboard View (dashboard-view): 2.9.10
External Monitor Job Type Plugin (external-monitor-job): 1.6
EZ Templates (ez-templates): 1.2.1
SSH Slaves plugin (ssh-slaves): 1.11
Copy Artifact Plugin (copyartifact): 1.38.1
Pipeline (workflow-aggregator): 2.3
JiraTestResultReporter plugin (JiraTestResultReporter): 2.0.3
HTML Publisher plugin (htmlpublisher): 1.11
Pipeline: Multibranch (workflow-multibranch): 2.9
Port Allocator Plug-in (port-allocator): 1.9-SNAPSHOT ( private -61b7bdb7-neil.rhine)
Script Security Plugin (script-security): 1.23
PMD Plug-in (pmd): 3.45
JUnit Plugin (junit): 1.18
Pipeline: REST API Plugin (pipeline- rest -api): 2.0
SSH Credentials Plugin (ssh-credentials): 1.12
Javadoc Plugin (javadoc): 1.4
Workspace Cleanup Plugin (ws-cleanup): 0.30
Hudson Post build task (postbuild-task): 1.8
SSH Agent Plugin (ssh-agent): 1.13
Amazon EC2 plugin (ec2): 1.36
GitHub API Plugin (github-api): 1.79
SCM API Plugin (scm-api): 1.3
Static Analysis Utilities (analysis-core): 1.79
Multijob plugin (jenkins-multijob-plugin): 1.22
SAML Plugin (saml): 0.6
GitHub Authentication plugin (github-oauth): 0.24
Plain Credentials Plugin (plain-credentials): 1.2
Conditional BuildStep (conditional-buildstep): 1.3.5
Groovy (groovy): 1.29
Matrix Project Plugin (matrix-project): 1.7.1
Credentials Plugin (credentials): 2.1.4
JaCoCo plugin (jacoco): 2.0.1
user build vars plugin (build-user-vars-plugin): 1.5
Notification plugin (notification): 1.10
JavaScript GUI Lib: Moment.js bundle plugin (momentjs): 1.1.1
Run Condition Plugin (run-condition): 1.0
Mailer Plugin (mailer): 1.17
Parameterized Trigger plugin (parameterized-trigger): 2.32
GitHub Branch Source Plugin (github-branch-source): 1.10
Maven Integration plugin (maven-plugin): 2.13
Node Iterator API Plugin (node-iterator-api): 1.5.0
Pipeline: Groovy (workflow-cps): 2.19
Windows Slaves Plugin (windows-slaves): 1.2
Pipeline: Stage View Plugin (pipeline-stage-view): 2.0
PAM Authentication plugin (pam-auth): 1.3
Ivy Plugin (ivy): 1.26
Subversion Plug-in (subversion): 2.6
Pipeline: Nodes and Processes (workflow-durable-task-step): 2.4
jQuery plugin (jquery): 1.11.2-0
Durable Task Plugin (durable-task): 1.12
Config File Provider Plugin (config-file-provider): 2.11
FindBugs Plug-in (findbugs): 4.65
Pipeline: Input Step (pipeline-input-step): 2.1
Fail The Build Plugin (fail-the-build-plugin): 1.0
bouncycastle API Plugin (bouncycastle-api): 1.648.3
Static Analysis Collector Plug-in (analysis-collector): 1.48
With a jenkins version of 2.7.3 Yep, gh-branch-source creates and uses own gh client, so i can't help in gh-plugin with issue 
Please verify if this issue is an issue with GitHub Branch Source 2.0.0-beta-1 (available from the experimental update center now or 2.0.0 (available in early January 2017)
Closing as a duplicate of JENKINS-36121. The cache directory stuff caused endless issues as I understand it, so it was dropped (that made the rate limit stuff worse). We may look into a different caching approach that can be unified in the branch-api or scm-api plugin so that all SCM systems can benefit (also the cache needs are different as we know hashes are immutable, so if we have checked hash cafebabedeadbeef for a /Jenkinsfile that answer will never change and can probably be stored more efficiently than a Http request cache
I did find the name of the directory that should be created: org.jenkinsci.plugins.github.GitHubPlugin.cache
Though, it appears that this bug should not be filed under github api plugin, that it should instead be moved to the github plugin – as the cache functions are not managed by the API directly, but the github plugin that wrappers the api plugin.
Adding that to the components list - please remove github api plugin if that is indeed the case.