-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major
-
Component/s: bitbucket-branch-source-plugin
-
Environment:Jenkins 2.568.3-lts-jdk21
cloudbees-bitbucket-branch-source 937.3.10
Self-hosted Bitbucket Server/Data Center, one credential per endpoint
Organization folders, default concurrent indexing limit (5)
Sustained HTTP 429 blocks indexing threads for ~85 minutes per call; unreleased
master adds an unbounded retry recursion. Affects any installation using a single
credential, on both Bitbucket Cloud and Server/Data Center.
--------------------------------------------------------------------------
Part 1 - released behaviour (937.3.10 and 937.3.11)
--------------------------------------------------------------------------
AbstractBitbucketApi#createClient installs the retry strategy unconditionally:
Â
  .setRetryStrategy(new ExponentialBackoffRetryStrategy(
      2, TimeUnit.SECONDS.toMillis(5), TimeUnit.HOURS.toMillis(1)))
Â
ExponentialBackoffRetryStrategy#retryRequest retries on 429 and 503 while the
computed interval is below maxExpiryInMillis, and never consults maxRetries:
Â
  return getRetryInterval(executionCount) < maxExpiryInMillis
      && (statusCode == HttpStatus.SC_TOO_MANY_REQUESTS
      || statusCode == HttpStatus.SC_SERVICE_UNAVAILABLE);
Â
Â
With (backOffRate=2, initialExpiry=5s, maxExpiry=1h) the delays are
5, 10, 20, 40, 80, 160, 320, 640, 1280 and 2560 seconds. The eleventh interval
would reach the 1 hour cap, so retrying stops there. That is 10 retries and
about 5115 seconds - roughly 85 minutes - during which the calling thread is
blocked.
For folder indexing this means one API call can hold an indexing slot for ~85
minutes. With the default limit of 5 concurrent computations, five such calls
stall indexing entirely for that period.
Observed in production. Our Bitbucket Server began returning 429
under load from Jenkins organization folder indexing. The cause of that load was
on our side and is already fixed. What follows is independent of how the volume
arose: once 429 started, the installation did not recover on its own:
 - Scan slots were never released; the queue showed
  "At maximum indexing capacity" against the default limit of 5.
 - Per-project last-scan timestamps stopped advancing: scans started but did
  not finish.
 - The indexing queue did not drain over many hours. Cancelling queue items
  did not help, as the upstream trigger kept firing.
 - Bitbucket CPU stayed elevated and other, non-Jenkins clients of the same
  instance were rate-limited as collateral damage.
We froze folder scans via a QueueDecisionHandler to let Bitbucket recover. But
the retry behaviour means any installation that reaches 429 for any reason will
behave the same way: it responds to "too many requests" by holding threads for
85 minutes and then trying again.
One further note on the released code: Retry-After is honoured for the length of
the delay in getRetryInterval(HttpResponse, int, HttpContext), but the decision
to keep retrying comes solely from the exponential value. A server asking for a
short pause therefore still receives all 10 retries.
--------------------------------------------------------------------------
Part 2 - regression on master, not yet released
--------------------------------------------------------------------------
Current master (ae3f03428c76c0a9fbfaf4f867be0ee153d4a21b, 2026-11-03) adds a
429 branch to AbstractBitbucketApi#executeMethod that is not present in any
released version:
Â
  if (statusCode == HttpStatus.SC_TOO_MANY_REQUESTS) {     if (authenticator != null) {       authenticator.markLimitReached();       response.close();       // retry       return executeMethod(request);     } else {       String content = getResponseContent(response);       throw buildResponseException(response, content);     }   }
Â
Â
This retries with no delay, no attempt counter and no upper bound.
BitbucketAuthenticator#markLimitReached() is a no-op by default; only
BitbucketMultiCredentialsAuthenticator overrides it to rotate credentials:
Â
So for any single-credential configuration nothing changes between attempts and
the recursion cannot terminate. Because it is real recursion rather than a loop,
sustained 429 also risks StackOverflowError.
It compounds with Part 1. On master the retry strategy is applied only when the
authenticator does not support rate limiting:
Â
  if (authenticator != null) {     if (!authenticator.supportsRateLimit()) {       httpClientBuilder.setRetryStrategy(new ExponentialBackoffRetryStrategy(           2, TimeUnit.SECONDS.toMillis(5), TimeUnit.HOURS.toMillis(1)));     }     ...   }
Â
Â
supportsRateLimit() is false by default:
Â
A single-credential installation therefore gets both: the ~85 minute ladder,
followed by an unbounded recursion that starts the whole ladder again.
Note also that the strategy is now installed inside the
"if (authenticator != null)" block, so anonymous clients - which received the
backoff unconditionally in 937.3.x - get no retry strategy at all on master.
This looks unintended.
Finally, BitbucketMultiCredentialsAuthenticator#markLimitReached hard-codes
Bitbucket Cloud's one-hour rolling window (DateUtils.addHours(entry.getValue(), 1)),
citing the Cloud rate-limit documentation. Bitbucket Server/Data Center uses a
token bucket with different refill behaviour, so a credential banned for a fixed
hour is both too long and not aligned with when quota actually returns there.
Deriving the ban duration from Retry-After when the server supplies it would
cover both platforms.
--------------------------------------------------------------------------
Relation to existing tickets
--------------------------------------------------------------------------
JENKINS-76471 and PRs #1232 and #1248 address Bitbucket Cloud rate limits with
pools of fallback credentials, using X-RateLimit-NearLimit and the Cloud rolling
window. Those mechanisms do not exist on Bitbucket Server/Data Center, and a
credential pool does not address either problem above: it only provides something
to rotate to. An installation with a single credential, on either platform, still
blocks for 85 minutes per call and - on master - still recurses without bound.
--------------------------------------------------------------------------
Suggested fix
--------------------------------------------------------------------------
 - Gate the executeMethod 429 branch on authenticator.supportsRateLimit(), the
  same condition already used to select the retry strategy in createClient.
  Only BitbucketMultiCredentialsAuthenticator overrides markLimitReached() to
  rotate; for every other authenticator it is a no-op, so the retry changes
  nothing and the recursion cannot terminate. Authenticators without a
  credential pool should fall through to the retry strategy and then surface
  the 429.
 - Treat 429 as an instruction rather than a transient error. When the server
  asks for a pause, honour it and then stop: retry at most once or twice,
  and when Retry-After is present do not retry beyond it. Today Retry-After
  only shortens the wait; the decision to keep retrying still comes solely
  from the exponential value, so a server asking for a short pause still
  receives all ten attempts.
 - Distinguish 429 from 503. The response-based retryRequest treats both
  identically, but 503 means the server is temporarily unavailable, where
  retrying is appropriate, while 429 means the client is sending too much,
  where retrying adds to the problem. A 429 is an instruction to stop, not a
  transient failure.
 - Default to a short retry budget, and make it configurable. The current
  values (backOffRate=2, initialExpiry=5s, maxExpiry=1h) are hard-coded, and
  because retryRequest stops only when the interval reaches maxExpiry rather
  than on an attempt count, maxRetries has no effect. An administrator facing
  sustained 429 has no setting to adjust. A short default matters more than
  the setting, since most installations will never change it.
 - For folder indexing specifically, fail fast and release the indexing slot
  instead of blocking it. A scan runs again on the next periodic trigger or
  webhook, so waiting 85 minutes inside one of five shared slots retries work
  that would have been repeated anyway.
