Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64418

Add exponential backoff to BitBucket rate limit retry loop

    • 933.3.0

      The current implementation of retry for Bitbucket rate limits sleeps 5 seconds and tries again.

      We have Bitbucket folders that have hundreds of repos some of which have hundreds of branches. This can sometimes take hours to do a folder scan. During that time the Bitbucket rate limit is constantly at its maximum

      Adding exponential backoff (with jitter for randomness) will allow other API requests (such as to download the Jenkinsfile when a multibranch pipeline is building) to go through and will alleviate the constant pressure on the rate limit.

          [JENKINS-64418] Add exponential backoff to BitBucket rate limit retry loop

          b_dean Provided a PR (https://github.com/jenkinsci/bitbucket-branch-source-plugin/pull/414) for exponential back-off for the Bitbucket API Client.

          As stated, by me and bitwiseman in the PR, it would be a much more stable and reliable solution to make use of the rate limit headers of the Bitbucket API for throttling the client request rate to the API.

          bitwiseman provided a great link to the Github Branch Source plugin, showing how the rate limit throttling was solved there: https://github.com/jenkinsci/github-branch-source-plugin/blob/master/src/main/java/org/jenkinsci/plugins/github_branch_source/ApiRateLimitChecker.java

          I will look into creating a PR to control request throttling in a similar manner to the Github Branch Source plugin.

           

          Morten Hekkvang added a comment - b_dean  Provided a PR ( https://github.com/jenkinsci/bitbucket-branch-source-plugin/pull/414)  for exponential back-off for the Bitbucket API Client. As stated, by me and bitwiseman  in the PR, it would be a much more stable and reliable solution to make use of the rate limit headers of the Bitbucket API for throttling the client request rate to the API. bitwiseman  provided a great link to the Github Branch Source plugin, showing how the rate limit throttling was solved there: https://github.com/jenkinsci/github-branch-source-plugin/blob/master/src/main/java/org/jenkinsci/plugins/github_branch_source/ApiRateLimitChecker.java I will look into creating a PR to control request throttling in a similar manner to the Github Branch Source plugin.  

          Scott Marlow added a comment -

          I heard feedback on https://github.com/jenkinsci/openstack-cloud-plugin/pull/365 that adding exponential backoff support to another part of Jenkins was not correct and learned that a deeper approach would be better.  I thought I would echo that here in case it helps those still working on this issue. 

          Also, https://www.jenkins.io/projects/gsoc/2023/project-ideas/agent_reconnections_exponential_backoff/ may also be related to this issue as well in that it outlines other areas that could also support exponential backoff in a way that minimizes spikes in the Jenkins host machine CPU usage by randomizing the time to sleep between retries (e.g. with jitter project as mentioned in https://www.baeldung.com/resilience4j-backoff-jitter).

          Scott Marlow added a comment - I heard feedback on https://github.com/jenkinsci/openstack-cloud-plugin/pull/365 that adding exponential backoff support to another part of Jenkins was not correct and learned that a deeper approach would be better.  I thought I would echo that here in case it helps those still working on this issue.  Also, https://www.jenkins.io/projects/gsoc/2023/project-ideas/agent_reconnections_exponential_backoff/ may also be related to this issue as well in that it outlines other areas that could also support exponential backoff in a way that minimizes spikes in the Jenkins host machine CPU usage by randomizing the time to sleep between retries (e.g. with jitter project as mentioned in https://www.baeldung.com/resilience4j-backoff-jitter ).

            nfalco Nikolas Falco
            b_dean Ben Dean
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: