Pull request jobs using large repositories significantly slow to start

XMLWordPrintable

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Critical
    • Component/s: atlassian-bitbucket-server-integration-plugin
    • None
    • Environment:

      We have a large repository in Bitbucket server,
       * 10GB in size (as per Bitbucket UI)
       * 2500 branches approximately
       * 150 open Pull Requests

      When this repository is configured for a multi branch pipeline, we've noticed that the refspec that is set by the BitbucketSCMSource to be the following.

       +refs/heads/*:refs/remotes/origin/*

      We first noticed at the checkout stage that we run after a pod is allocated, which was cloning the entire repository than the branch it requires, taking a long time (5 minutes+). We then custom set the refspec to the checkout to be a specific branch or the PR, the performance changed back to what we used to have with cloudbees-bitbucket-branch-source.

      However, for pull requests, it seems like even before the job reaches the steps in the Jenkinsfile it is doing a full repository clone, which takes again around 5 mins, before we see "[Pipeline] Start of Pipeline" line on the logs.

      To make sure this was the problem, we got the code for the plugin and made the following change to the BitbucketSCMSource.java, and the performance is back to what normal, which is few seconds to start the job.

      diff --git a/src/main/java/com/atlassian/bitbucket/jenkins/internal/scm/BitbucketSCMSource.java b/src/main/java/com/atlassian/bitbucket/jenkins/internal/scm/BitbucketSCMSource.java
index 06cf811..7d62922 100644
--- a/src/main/java/com/atlassian/bitbucket/jenkins/internal/scm/BitbucketSCMSource.java
+++ b/src/main/java/com/atlassian/bitbucket/jenkins/internal/scm/BitbucketSCMSource.java
@@ -70,7 +70,6 @@ import static org.apache.commons.lang3.StringUtils.isNotBlank;
 public class BitbucketSCMSource extends SCMSource {
 
     private static final Logger LOGGER = Logger.getLogger(BitbucketSCMSource.class.getName());
-    private static final String REFSPEC_DEFAULT = "+refs/heads/*:refs/remotes/@\{remote}/*";
     @UpgradeHandled(handledBy = "Uses the same remote variable as REFSPEC_DEFAULT", removeAnnotationInVersion = "4.1")
     private static final String REFSPEC_TAGS = "+refs/tags/*:refs/remotes/@\{remote}/*";
 
@@ -161,7 +160,11 @@ public class BitbucketSCMSource extends SCMSource {
         if (head.getClass().equals(BitbucketTagSCMHead.class)) {
             builder.withRefSpec(REFSPEC_TAGS);
         } else {
-            builder.withRefSpec(REFSPEC_DEFAULT);
+            SCMHead refSpecHead = head;
+            if (head instanceof BitbucketPullRequestSCMHead) {
+                refSpecHead = ((BitbucketPullRequestSCMHead) head).getTarget();
+            }
+            builder.withRefSpec("+refs/heads/" + refSpecHead.getName() + ":refs/remotes/@\{remote}/" + refSpecHead.getName());
         }
 
         builder.withTraits(traits);

      We've tested this with branches and PRs.

      • On branch builds, the build starts almost instantly, and the checkouts running later are much faster as well.
      • On PRs, the start time is has halved (better, but not best), as it still wait for minutes doing 
        git fetch --no-tags --force --progress – https://.../scm/.../....git +refs/heads/:refs/remotes/origin/ # timeout=10

        at the beginning of the job before it reaches steps in the Jenkinsfile. Later checkout are faster.

      As we run these builds in agents in Kubernetes, its kind of a waste doing a clone in the server side like this and merging the change, which will then again be done on an agent at a later point. Would be even better if we could skip this step, or use the Bitbucket REST API to check the mergeability of the PR instead.

      Think the above mentioned change is still worth going in as its a step improvement. Happy to raise a PR.

      Will be investigating further to see how we could make this even better. Will report back if there's any further updates.

      Update 03/Nov/2025 14:17:

      • For the time being we've switched back to  cloudbees-bitbucket-branch-source plugin for our large repository, as the cost of the fetch of few mins adds up to the total time PRs have to queue up to be merged to mainline. I've raised a separate feature request for this JENKINS-76259
      • It would be great to get the change proposed in this task to fix the refspecs as it will improve performance of all builds in general.

            Assignee:
            Unassigned
            Reporter:
            Venushka Perera
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: