Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61493

Lightweight checkout fails in Bitbucket Server 7.0; refs/pull-requests/*/merge does not exist

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Minor Minor
    • None
    • Jenkins 2.204.5
      Bitbucket Branch Source Plugin 2.7.0
      Bitbucket Server 7.0.1

       

      NOTE

      The linked comment below indicates that in BitBucket 7 Atlassian has removed the ability to do lightweight checkout while using the "Merging" strategy. This is "intentional" and "by-design" behavior implemented by Atlassian. It is beyond the control of Jenkins project to fix this at this time.

      If your build fails while attempting to checkout, be sure you've enabled "Call changes API" for your Bitbucket 7 endpoints.  

      As a workaround to heavyweight checkout, you can try changing to use the "Current Pull Request revision" strategy instead.

      Keep in mind that with "Current Pull Request revision" PR builds it is possible for a PR build to succeed and result in an unstable main branch after changes are merged. If you are an Atlassian customer and do not like this behavior, tell Atlassian this feature is important to you or perhaps someone can make a plugin that can be added to BitBucket server that reintroduces this behavior.

      Suggestions have been made for potential changes on the Jenkins side. These have sounded somewhat complicated or brittle, but PRs are welcome.

      https://jira.atlassian.com/browse/BSERV-12284?focusedCommentId=2389584&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-2389584

       

       

      After upgrading to Bitbucket Server 7.0.1, we get this kind of error for every pull request in multibranch projects in which Jenkins has been configured to build the result of merging the pull request:

       ERROR: Could not do lightweight checkout, falling back to heavyweight
       java.io.FileNotFoundException: URL: /rest/api/1.0/projects/REDACTED/repos/redacted/browse/Jenkinsfile?at=pull-requests%2F771%2Fmerge&start=0&limit=500
       	at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getRequest(BitbucketServerAPIClient.java:831)
       	at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getFileContent(BitbucketServerAPIClient.java:1123)
       	at com.cloudbees.jenkins.plugins.bitbucket.filesystem.BitbucketSCMFile.content(BitbucketSCMFile.java:98)
       	at jenkins.scm.api.SCMFile.contentAsString(SCMFile.java:335)
       	at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:107)
       	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:303)
       	at hudson.model.ResourceController.execute(ResourceController.java:97)
       	at hudson.model.Executor.run(Executor.java:427)
      

      git ls-remote origin shows that Bitbucket Server now publishes "refs/pull-requests/771/from" but not "refs/pull-requests/771/merge", even though the pull request has no merge conflicts (the target branch even has no new commits). This might be related to "the switch from a 3-way diff to a 2-way diff in pull requests" in Bitbucket Server 7.0.0.

      Jenkins then does the heavyweight checkout and successfully merges the pull request, so the project ultimately builds OK, but the heavyweight checkout consumes a lot of time and disk space.

      There might not be any simple way to improve this in the bitbucket-branch-source plugin. If Bitbucket Server has deliberately ceased publishing "refs/pull-requests/*/merge", perhaps the plugin could do a three-way merge on Jenkinsfile only, rather than on the entire tree.

          [JENKINS-61493] Lightweight checkout fails in Bitbucket Server 7.0; refs/pull-requests/*/merge does not exist

          Liam Newman added a comment - - edited

          kon
          huber

          The linked comment below indicates that in BitBucket 7 Atlassian has removed the ability to do lightweight checkout while using the "Merging" strategy. This is intentional, by-design behavior implemented by Atlassian. It is beyond the control of Jenkins project to fix this.

          As a workaround, please try changing to use the "Current Pull Request revision" strategy instead and see if that works. 

          refs/pull-requests/*/merge has been fully removed in 7.0 and will never be created/updated by the system again. For pull requests that were opened prior to upgrading to 7.x, which have a merge ref, that ref will be removed the first time the pull request is updated (or when it's merged or declined, as ever). With the switch from 3-way diffs (which were built around displaying the merge's diff) to 2-way diffs (which use the common ancestor), the system no longer retains the merges it creates to check mergeability; it runs git merge to see what happens, and discards everything that results. This prevents the accumulation of "unreachable" pull request merge objects in repositories, which can (greatly) slow down clone/fetch/pull/push performance as they build up. Any tool that wishes to build a pull request's merge will need to create that merge on its own, going forward.

          Prior to 7.0, because pull request diffs were built around the merge, checking mergeability (which happens automatically after page load when a pull request is created via the UI) would create the public refs/pull-requests entries for that pull request. In 7.x, since the diff is now using the common ancestor and no longer needs the merge, checking mergeability doesn't create any refs. (This is shown in the log snippets above.) Instead, the refs are created when a pull request's diff is viewed. Plugins which are installed in Bitbucket Server can simulate this by either using the PullRequestService to stream the pull request's changes, or (more efficiently), by using ScmService.getPullRequestCommandFactory(pullRequest).effectiveDiff() (which PullRequestService uses under the hood). Remote callers can use rest/api/1.0/projects/KEY/repos/slug/pull-requests/N/changes?limit=1 to do the same. Any of those will ensure refs/pull-requests/*/from is created (or updated) to match the pull request's current state.

          I'd wager many webhook apps (or other apps that notify external systems, however they do so) have logic in them for using PullRequestService.canMerge, MergeRequestCheckService.check or ScmPullRequestCommandFactory.tryMerge to try and force pull request refs to be created/updated. (The Post Webhooks for Bitbucket app did, back when it was open source; the source has since been made private, or at least moved off Github, so I'm not sure these days.) Those approaches will not work in 7.x. Instead, such apps need to be updated to use PullRequestService.streamChanges or ScmPullRequestCommandFactory.effectiveDiff to force the from ref to be created/updated. As noted earlier, there is no way in 7.x to get a merge ref.

          I'll reiterate here that Bitbucket Server's pull request refs are not API, and have never been considered API, so the way they work is dangerous to rely on. I'll also readily acknowledge that that's hardly a satisfactory answer, to receive or to give, and that it would be nice if it changed. I am not a PM for Bitbucket Server, so I can't say when/if such improvements will hit the roadmap. Perhaps I'll see if I can use some upcoming 20% time to make some improvements. I'm sorry that, at least for the moment, I can't offer better.

          Best regards,

          Bryan Turner

          Atlassian Bitbucket

          https://jira.atlassian.com/browse/BSERV-12284?focusedCommentId=2389584&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-2389584

           

          If you do not like this behavior, tell Atlassian this feature is important to you. 

          Liam Newman added a comment - - edited kon huber The linked comment below indicates that in BitBucket 7 Atlassian has removed the ability to do lightweight checkout while using the "Merging" strategy. This is intentional, by-design behavior implemented by Atlassian. It is beyond the control of Jenkins project to fix this. As a workaround, please try changing to use the "Current Pull Request revision" strategy instead and see if that works.  refs/pull-requests/*/merge has been fully removed in 7.0 and will never be created/updated by the system again. For pull requests that were opened prior to upgrading to 7.x, which have a merge ref, that ref will be removed the first time the pull request is updated (or when it's merged or declined, as ever). With the switch from 3-way diffs (which were built around displaying the merge's diff) to 2-way diffs (which use the common ancestor), the system no longer retains the merges it creates to check mergeability; it runs git merge to see what happens, and discards everything that results. This prevents the accumulation of "unreachable" pull request merge objects in repositories, which can (greatly) slow down clone/fetch/pull/push performance as they build up. Any tool that wishes to build a pull request's merge will need to create that merge on its own, going forward. Prior to 7.0, because pull request diffs were built around the merge, checking mergeability (which happens automatically after page load when a pull request is created via the UI) would create the public refs/pull-requests entries for that pull request. In 7.x, since the diff is now using the common ancestor and no longer needs the merge, checking mergeability doesn't create any refs. (This is shown in the log snippets above.) Instead, the refs are created when a pull request's diff is viewed. Plugins which are installed in Bitbucket Server can simulate this by either using the PullRequestService to stream the pull request's changes, or (more efficiently), by using ScmService.getPullRequestCommandFactory(pullRequest).effectiveDiff() (which PullRequestService uses under the hood). Remote callers can use rest/api/1.0/projects/KEY/repos/slug/pull-requests/N/changes?limit=1 to do the same. Any of those will ensure refs/pull-requests/*/from is created (or updated) to match the pull request's current state. I'd wager many webhook apps (or other apps that notify external systems, however they do so) have logic in them for using PullRequestService.canMerge, MergeRequestCheckService.check or ScmPullRequestCommandFactory.tryMerge to try and force pull request refs to be created/updated. (The Post Webhooks for Bitbucket app did, back when it was open source; the source has since been made private, or at least moved off Github, so I'm not sure these days.) Those approaches will not work in 7.x. Instead, such apps need to be updated to use PullRequestService.streamChanges or ScmPullRequestCommandFactory.effectiveDiff to force the from ref to be created/updated. As noted earlier, there is no way in 7.x to get a merge ref. I'll reiterate here that Bitbucket Server's pull request refs are not API, and have never been considered API, so the way they work is dangerous to rely on. I'll also readily acknowledge that that's hardly a satisfactory answer, to receive or to give, and that it would be nice if it changed. I am not a PM for Bitbucket Server, so I can't say when/if such improvements will hit the roadmap. Perhaps I'll see if I can use some upcoming 20% time to make some improvements. I'm sorry that, at least for the moment, I can't offer better. Best regards, Bryan Turner Atlassian Bitbucket https://jira.atlassian.com/browse/BSERV-12284?focusedCommentId=2389584&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-2389584   If you do not like this behavior, tell Atlassian this feature is important to you. 

          "Current Pull Request revision" is not useful for me because:

          • I have Jenkins already building the branches that are used as sources of the pull requests.
          • I often have pull requests merging the same source branch to multiple target branches.
          • I want Jenkins to detect problems that only exist in the merged version, rather than in the source or target branches. E.g. if the source branch added calls of a method that was meanwhile deleted from the target branch.
          • It generates non-descriptive job titles like "PR-1234". (JENKINS-55348)

          It seems to me that the trivial case where the file in question is identical in the source and target branches could be implemented in bitbucket-branch-source-plugin without changing Bitbucket Server, as follows:

          1. The Jenkins master wants to read Jenkinsfile in a multibranch project configured with "Merging the pull request with the current target branch revision".
          2. It uses lightweight checkout to read Jenkinsfile from the source branch.
          3. It also uses lightweight checkout to read Jenkinsfile from the target branch.
          4. If Jenkinsfile is bit-for-bit identical in both branches, then the Jenkins master uses that.
          5. Otherwise, the Jenkins master uses heavyweight checkout and merge. To minimize disk usage and network I/O on the Jenkins master, this could perhaps be a sparse checkout with Git LFS disabled.

          Kalle Niemitalo added a comment - "Current Pull Request revision" is not useful for me because: I have Jenkins already building the branches that are used as sources of the pull requests. I often have pull requests merging the same source branch to multiple target branches. I want Jenkins to detect problems that only exist in the merged version, rather than in the source or target branches. E.g. if the source branch added calls of a method that was meanwhile deleted from the target branch. It generates non-descriptive job titles like "PR-1234". ( JENKINS-55348 ) It seems to me that the trivial case where the file in question is identical in the source and target branches could be implemented in bitbucket-branch-source-plugin without changing Bitbucket Server, as follows: The Jenkins master wants to read Jenkinsfile in a multibranch project configured with "Merging the pull request with the current target branch revision". It uses lightweight checkout to read Jenkinsfile from the source branch. It also uses lightweight checkout to read Jenkinsfile from the target branch. If Jenkinsfile is bit-for-bit identical in both branches, then the Jenkins master uses that. Otherwise, the Jenkins master uses heavyweight checkout and merge. To minimize disk usage and network I/O on the Jenkins master, this could perhaps be a sparse checkout with Git LFS disabled.

          Liam Newman added a comment -

          kon

          I've been informed of a possible fix.  Please try enabling "Call changes API" for your Bitbucket 7 endpoints.  See if that fixes the issue for you.  

          This wasn't turned on by default because it makes Jenkins make additional API calls, which some users did not want.  I think it makes sense to force this to "enabled" when using "Merging the pull request with the current target branch revision" on BB 7.x, but at very least there should be a helpful messing pointing to how to get lightweight checkout. 

          Liam Newman added a comment - kon I've been informed of a possible fix.  Please try enabling "Call changes API" for your Bitbucket 7 endpoints.  See if that fixes the issue for you.   This wasn't turned on by default because it makes Jenkins make additional API calls, which some users did not want.  I think it makes sense to force this to "enabled" when using "Merging the pull request with the current target branch revision" on BB 7.x, but at very least there should be a helpful messing pointing to how to get lightweight checkout. 

          Kalle Niemitalo added a comment - - edited

          bitwiseman, the "Call Changes api" setting on the "Configure System" page is intended to make Bitbucket Server update its "refs/pull-requests/*/from" refs that would be used by the "Current Pull Request revision" strategy. I have tested that it does not help with the "refs/pull-requests/*/merge" refs that the "Merging the pull request with the current target branch revision" strategy tries to use for lightweight checkout. I would like to keep this issue focused on the latter.

          This test was with Jenkins 2.235.5, Bitbucket Branch Source Plugin 2.9.2, and Bitbucket Server 7.1.1.

          Kalle Niemitalo added a comment - - edited bitwiseman , the "Call Changes api" setting on the "Configure System" page is intended to make Bitbucket Server update its "refs/pull-requests/*/from" refs that would be used by the "Current Pull Request revision" strategy. I have tested that it does not help with the "refs/pull-requests/*/merge" refs that the "Merging the pull request with the current target branch revision" strategy tries to use for lightweight checkout. I would like to keep this issue focused on the latter. This test was with Jenkins 2.235.5, Bitbucket Branch Source Plugin 2.9.2, and Bitbucket Server 7.1.1.

          Zhenlei Huang added a comment -

          Hi kon,

          The "Call Changes api" enforce Bitbucket Server update its "refs/pull-requests/*/from", regardless the strategy.

          The lightweight checkout was intended to work with hosting vendor that provide merged refs, such as GitHub, Bitbucket 6.x. This feature offloads the merging process to remote hosting other than Jenkins itself.

          Zhenlei Huang added a comment - Hi kon , The "Call Changes api" enforce Bitbucket Server update its "refs/pull-requests/*/from", regardless the strategy. The lightweight checkout was intended to work with hosting vendor that provide merged refs, such as GitHub, Bitbucket 6.x. This feature offloads the merging process to remote hosting other than Jenkins itself.

          Zhenlei Huang added a comment -

          Hi, bitwiseman

          I'd propose reimplement the "lightweight checkout" feature to also work with vendors that does not provide merged refs. A straight simple way is we fetch the target ref and source ref, then merge locally in Jenkins temp workspace, producing the merged version.

          As for single file "lightweight checkout", we can create a temp local git repository, fetch the file from source ref url and target ref url, and then merge them in the created local git repository.

          Compared to the legacy "lightweight checkout", this approach does have cons:

          1. It will need one more http request / git pull request
          2. Need some resources on Jenkins instance to do the merge.

          How do you think this approach?

          Zhenlei Huang added a comment - Hi, bitwiseman I'd propose reimplement the "lightweight checkout" feature to also work with vendors that does not provide merged refs. A straight simple way is we fetch the target ref and source ref, then merge locally in Jenkins temp workspace, producing the merged version. As for single file "lightweight checkout", we can create a temp local git repository, fetch the file from source ref url and target ref url, and then merge them in the created local git repository. Compared to the legacy "lightweight checkout", this approach does have cons: It will need one more http request / git pull request Need some resources on Jenkins instance to do the merge. How do you think this approach?

          Kalle Niemitalo added a comment - - edited

          gmshake, I don't think you can merge just two versions of a file such as Jenkinsfile; you need a base version as well. Otherwise, you wouldn't know whether differences mean that lines were added or that they were removed. As I mentioned in the description of this issue, the Bitbucket Server REST API does not seem to provide a way to get the commit ID or file content from the merge base of a pull request. This is why I proposed only implementing the trivial case where the two versions of the file are identical (although the PR has modified other files) and there is no need for an actual merge.

          Kalle Niemitalo added a comment - - edited gmshake , I don't think you can merge just two versions of a file such as Jenkinsfile; you need a base version as well. Otherwise, you wouldn't know whether differences mean that lines were added or that they were removed. As I mentioned in the description of this issue, the Bitbucket Server REST API does not seem to provide a way to get the commit ID or file content from the merge base of a pull request. This is why I proposed only implementing the trivial case where the two versions of the file are identical (although the PR has modified other files) and there is no need for an actual merge.

          Liam Newman added a comment -

          kon
          Thank you for testing that. I can confirm you are correct. Without the "Call Changes API" neither of the merge strategies work at all for new PRs - the `/from` ref is missing and the run fails immediately.

          gmshake and kon

          A straight simple way is we fetch the target ref and source ref, then merge locally in Jenkins temp workspace, producing the merged version.

          This is why I proposed only implementing the trivial case where the two versions of the file are identical (although the PR has modified other files) and there is no need for an actual merge.

          The solutions you both are advocating are both possible, but I would not advise them. The number of edge cases (and not-so-edge complexities) in this area is high. Any solution that doesn't use an scm-provider-side solution, is likely be a bug farm.
          gmshake - Kalle pointed to some limitation to what you proposed. Also, what about forks? The current implementation expects to find all files in the target repo, switching to a fork for some would be problematic.
          kon Your proposal is simpler but it would only work if the pipeline does not call any of the steps that do additional reads (readTrusted, etc), and there is no way to know if those steps are present until during the run.

          Overall, what you are suggesting is doing a lot of work to mitigate something that is almost trivial to do on on the SCM server. You would be better served by creating a plugin/app for Bitbucket that implements the merge ref creation.

          If you would like to implement and provide indefinite support for this case, I'm open to taking PRs that add functionality as long is it off by default.

          Liam Newman added a comment - kon Thank you for testing that. I can confirm you are correct. Without the "Call Changes API" neither of the merge strategies work at all for new PRs - the `/from` ref is missing and the run fails immediately. gmshake and kon A straight simple way is we fetch the target ref and source ref, then merge locally in Jenkins temp workspace, producing the merged version. This is why I proposed only implementing the trivial case where the two versions of the file are identical (although the PR has modified other files) and there is no need for an actual merge. The solutions you both are advocating are both possible, but I would not advise them. The number of edge cases (and not-so-edge complexities) in this area is high. Any solution that doesn't use an scm-provider-side solution, is likely be a bug farm. gmshake - Kalle pointed to some limitation to what you proposed. Also, what about forks? The current implementation expects to find all files in the target repo, switching to a fork for some would be problematic. kon Your proposal is simpler but it would only work if the pipeline does not call any of the steps that do additional reads (readTrusted, etc), and there is no way to know if those steps are present until during the run. Overall, what you are suggesting is doing a lot of work to mitigate something that is almost trivial to do on on the SCM server. You would be better served by creating a plugin/app for Bitbucket that implements the merge ref creation. If you would like to implement and provide indefinite support for this case, I'm open to taking PRs that add functionality as long is it off by default.

          bitwiseman, I guess I should then file a separate request to allow disabling Git LFS in the "Merging the pull request with the current target branch revision" heavyweight checkout at the Jenkins master. That should at least be simple to implement and would reduce the disk space and network I/O costs in my scenario.

          Kalle Niemitalo added a comment - bitwiseman , I guess I should then file a separate request to allow disabling Git LFS in the "Merging the pull request with the current target branch revision" heavyweight checkout at the Jenkins master. That should at least be simple to implement and would reduce the disk space and network I/O costs in my scenario.

          Zhenlei Huang added a comment -

          There's another edge case not covered by my previous proposal. Say we have a file named 'Jenkinsfile.old' in base branch, and it was renamed to 'Jenkinsfile' in PR's head branch, it looks difficult to get the original file name unless SCM server provide the information in the PR. 

          bitwiseman,
          I agree with you that it is trivial to do on the SCM server. It's frustrating to hack on Jenkins' side

          Zhenlei Huang added a comment - There's another edge case not covered by my previous proposal. Say we have a file named 'Jenkinsfile.old' in base branch, and it was renamed to 'Jenkinsfile' in PR's head branch, it looks difficult to get the original file name unless SCM server provide the information in the PR.  bitwiseman , I agree with you that it is trivial to do on the SCM server. It's frustrating to hack on Jenkins' side

            Unassigned Unassigned
            kon Kalle Niemitalo
            Votes:
            15 Vote for this issue
            Watchers:
            19 Start watching this issue

              Created:
              Updated: