Search

Our team is using Github APP which has a rate limit of 15000. Our github API rate limit was set at throttle (later at Normalize API rate limit). 

When it was at throttle, we have noticed that it is counting the API limit budget wrong and erroneously tagging that jobs are over the limit. 

For example:

+LCAAAAAAAAP9b85aBtbiIwSa/KF0vKzUvOzOvODlTryCnNB3I0kvPLMkoTYpPKkrMS86IL84vLUpO1XPPLPEoTXLOzyvOz0n1yy9JZYAARiYGRi8GzpLM3NTiksTcgooiBqmM0pTi/Dy9ZIhiPayaGCoKgHRT7d27GwCBFv4JjgAAAA==^[[0mJenkins-Imposed API Limiter: Current quota for Github API usage has 4118 remaining (2 over budget). Next quota of 15000 in 17 min. Sleeping for 25 sec.

• Since we are running pipeline jobs, our jobs started hogging controllers, causing a queue in the jobs waiting.
• After wrongly calculated budget refreshed, the jobs ran and finished, however the controller at this point starved for executors or threads (we are not using controller to run jobs in the controller), did not release those jobs – causing us to restart application. (Which means someone had to watch the controller all the time)

Workaround we are using -

• We are setting the rate limit to "never check" which is not recommended.

What is expected for the fix

• Make sure that calculations of rate limit is correctly done
• Make sure that controllers either fail the job when rate limit is hit or put allow us to put a timer to fail.
• Allow us to graciously kill/clear the controller executor without restarting application.