Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61493

Lightweight checkout fails in Bitbucket Server 7.0; refs/pull-requests/*/merge does not exist

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Labels:
      None
    • Environment:
      Jenkins 2.204.5
      Bitbucket Branch Source Plugin 2.7.0
      Bitbucket Server 7.0.1
    • Similar Issues:

      Description

       

      NOTE

      The linked comment below indicates that in BitBucket 7 Atlassian has removed the ability to do lightweight checkout while using the "Merging" strategy. This is "intentional" and "by-design" behavior implemented by Atlassian. It is beyond the control of Jenkins project to fix this at this time.

      If your build fails while attempting to checkout, be sure you've enabled "Call changes API" for your Bitbucket 7 endpoints.  

      As a workaround to heavyweight checkout, you can try changing to use the "Current Pull Request revision" strategy instead.

      Keep in mind that with "Current Pull Request revision" PR builds it is possible for a PR build to succeed and result in an unstable main branch after changes are merged. If you are an Atlassian customer and do not like this behavior, tell Atlassian this feature is important to you or perhaps someone can make a plugin that can be added to BitBucket server that reintroduces this behavior.

      Suggestions have been made for potential changes on the Jenkins side. These have sounded somewhat complicated or brittle, but PRs are welcome.

      https://jira.atlassian.com/browse/BSERV-12284?focusedCommentId=2389584&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-2389584

       

       

      After upgrading to Bitbucket Server 7.0.1, we get this kind of error for every pull request in multibranch projects in which Jenkins has been configured to build the result of merging the pull request:

       ERROR: Could not do lightweight checkout, falling back to heavyweight
       java.io.FileNotFoundException: URL: /rest/api/1.0/projects/REDACTED/repos/redacted/browse/Jenkinsfile?at=pull-requests%2F771%2Fmerge&start=0&limit=500
       	at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getRequest(BitbucketServerAPIClient.java:831)
       	at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getFileContent(BitbucketServerAPIClient.java:1123)
       	at com.cloudbees.jenkins.plugins.bitbucket.filesystem.BitbucketSCMFile.content(BitbucketSCMFile.java:98)
       	at jenkins.scm.api.SCMFile.contentAsString(SCMFile.java:335)
       	at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:107)
       	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:303)
       	at hudson.model.ResourceController.execute(ResourceController.java:97)
       	at hudson.model.Executor.run(Executor.java:427)
      

      git ls-remote origin shows that Bitbucket Server now publishes "refs/pull-requests/771/from" but not "refs/pull-requests/771/merge", even though the pull request has no merge conflicts (the target branch even has no new commits). This might be related to "the switch from a 3-way diff to a 2-way diff in pull requests" in Bitbucket Server 7.0.0.

      Jenkins then does the heavyweight checkout and successfully merges the pull request, so the project ultimately builds OK, but the heavyweight checkout consumes a lot of time and disk space.

      There might not be any simple way to improve this in the bitbucket-branch-source plugin. If Bitbucket Server has deliberately ceased publishing "refs/pull-requests/*/merge", perhaps the plugin could do a three-way merge on Jenkinsfile only, rather than on the entire tree.

        Attachments

          Activity

          Hide
          kon Kalle Niemitalo added a comment -

          Mentioned in BSERV-11477 (comment).

          Show
          kon Kalle Niemitalo added a comment - Mentioned in BSERV-11477 (comment) .
          Hide
          alexey_pelykh Alexey Pelykh added a comment -

          Seems that on 7.1.0 "/from" are not revealed via ls-remove as well

          Show
          alexey_pelykh Alexey Pelykh added a comment - Seems that on 7.1.0 "/from" are not revealed via ls-remove as well
          Hide
          alexey_pelykh Alexey Pelykh added a comment -

          It seems that unless non-author user actually tries to view the PR, the "/from" ref is not created. So when I just use PRs on my own - "/from" is not created whatsoever, that's seems to be a severe case

          Show
          alexey_pelykh Alexey Pelykh added a comment - It seems that unless non-author user actually tries to view the PR, the "/from" ref is not created. So when I just use PRs on my own - "/from" is not created whatsoever, that's seems to be a severe case
          Hide
          alexey_pelykh Alexey Pelykh added a comment -
          Show
          alexey_pelykh Alexey Pelykh added a comment - Seems  https://jira.atlassian.com/browse/BSERV-12198?focusedCommentId=2374169  is relatede as well
          Hide
          alexey_pelykh Alexey Pelykh added a comment -

          Update: I can trigger appearance of the "/from" ref only by viewing the diff. Super-bad behaviour

          Show
          alexey_pelykh Alexey Pelykh added a comment - Update: I can trigger appearance of the "/from" ref only by viewing the diff. Super-bad behaviour
          Hide
          kon Kalle Niemitalo added a comment -

          The missing "/from" was filed two years ago as JENKINS-45997, and fixed in https://github.com/jenkinsci/bitbucket-branch-source-plugin/pull/116 by making the plugin poke the /rest/api/1.0/projects/{owner}/repos/{repo}/pull-requests/{id}/merge API, but that apparently stopped working in Bitbucket Server 7.

          Show
          kon Kalle Niemitalo added a comment - The missing "/from" was filed two years ago as JENKINS-45997 , and fixed in https://github.com/jenkinsci/bitbucket-branch-source-plugin/pull/116 by making the plugin poke the /rest/api/1.0/projects/{owner}/repos/{repo}/pull-requests/{id}/merge API, but that apparently stopped working in Bitbucket Server 7.
          Hide
          alexey_pelykh Alexey Pelykh added a comment -

          Kalle Niemitalo seems so, huge blocker  will take a look on various APIs maybe it's possible to trigger that somehow otherwise

          Show
          alexey_pelykh Alexey Pelykh added a comment - Kalle Niemitalo seems so, huge blocker  will take a look on various APIs maybe it's possible to trigger that somehow otherwise
          Hide
          kon Kalle Niemitalo added a comment - - edited

          I can think of two approaches that do not rely on Bitbucket Server updating the "refs/pull-requests/{id}/from" ref:

          Show
          kon Kalle Niemitalo added a comment - - edited I can think of two approaches that do not rely on Bitbucket Server updating the "refs/pull-requests/{id}/from" ref: Take the commit ID from the webhook event payload . In BSERV-8268 , Atlassian enabled uploadpack.allowReachableSHA1InWant , so this should work even if that commit is no longer the tip of any branch, as long as it is still reachable. However, if a user of Jenkins manually triggers a build, then I guess the webhook payload will not be available. Query /rest/api/1.0/projects/{projectKey}/repos/{repositorySlug}/pull-requests/{pullRequestId} and get the name of the source branch from within the "fromRef" property.
          Hide
          kon Kalle Niemitalo added a comment -

          The newer Bitbucket Server Integration plugin does not even support pull requests yet, according to JENKINS-60342.

          Show
          kon Kalle Niemitalo added a comment - The newer Bitbucket Server Integration plugin does not even support pull requests yet, according to JENKINS-60342 .
          Hide
          alexey_pelykh Alexey Pelykh added a comment -

          Second approach sounds really robust, yet I have no real idea what implications are there.

          Show
          alexey_pelykh Alexey Pelykh added a comment - Second approach sounds really robust, yet I have no real idea what implications are there.
          Hide
          kon Kalle Niemitalo added a comment -

          It looks like that is what the Bitbucket Branch Source plugin does anyway, for Bitbucket Cloud. So, fixing the "refs/pull-requests/{id}/from" problem for pull requests within the same repository in Bitbucket Server could be just a matter of removing a few instanceof BitbucketCloudApiClient checks in BitbucketGitSCMBuilder.java and BitbucketSCMFileSystem.java. Fixing it for pull requests between forks would need more changes.

          Show
          kon Kalle Niemitalo added a comment - It looks like that is what the Bitbucket Branch Source plugin does anyway, for Bitbucket Cloud. So, fixing the "refs/pull-requests/{id}/from" problem for pull requests within the same repository in Bitbucket Server could be just a matter of removing a few instanceof BitbucketCloudApiClient checks in BitbucketGitSCMBuilder.java and BitbucketSCMFileSystem.java . Fixing it for pull requests between forks would need more changes.
          Hide
          alexey_pelykh Alexey Pelykh added a comment -

          Call to `/rest/api/1.0/projects/{PROJECT}/repos/{REPO}/pull-requests/19/diff` causes `/from` to appear

          Show
          alexey_pelykh Alexey Pelykh added a comment - Call to `/rest/api/1.0/projects/{PROJECT}/repos/{REPO}/pull-requests/19/diff` causes `/from` to appear
          Hide
          kon Kalle Niemitalo added a comment -

          Does Bitbucket Server ensure that the "refs/pull-requests/{id}/from" ref appears (or is updated) immediately after the diff request, or can there be a delay sometimes?

          Show
          kon Kalle Niemitalo added a comment - Does Bitbucket Server ensure that the "refs/pull-requests/{id}/from" ref appears (or is updated) immediately after the diff request, or can there be a delay sometimes?
          Hide
          alexey_pelykh Alexey Pelykh added a comment -

          I'm checking manually, so no way to track the time between that. I've created a pull request, did ls-remove, made API call using Postman and re-did ls-remote to verify it appeared

          Show
          alexey_pelykh Alexey Pelykh added a comment - I'm checking manually, so no way to track the time between that. I've created a pull request, did ls-remove, made API call using Postman and re-did ls-remote to verify it appeared
          Hide
          alexey_pelykh Alexey Pelykh added a comment -

          I want to try HEAD instead of GET request next time

          Show
          alexey_pelykh Alexey Pelykh added a comment - I want to try HEAD instead of GET request next time
          Hide
          kon Kalle Niemitalo added a comment -

          I would recommend doing the "fromRef" thing instead of testing which REST requests update "refs/pull-requests/*/from" as an undocumented side effect.

          Show
          kon Kalle Niemitalo added a comment - I would recommend doing the "fromRef" thing instead of testing which REST requests update "refs/pull-requests/*/from" as an undocumented side effect.
          Hide
          alexey_pelykh Alexey Pelykh added a comment -

          That sounds much better, yet I'm not sure I'm qualified enough to make such PR

          Show
          alexey_pelykh Alexey Pelykh added a comment - That sounds much better, yet I'm not sure I'm qualified enough to make such PR
          Hide
          rirl_dev Rick Landon added a comment -

          While this is being sorted out, does anyone have a suggested workaround?

          Show
          rirl_dev Rick Landon added a comment - While this is being sorted out, does anyone have a suggested workaround?
          Hide
          huber Kevin Huber added a comment -

          You can use the hpi published by this PR: https://github.com/jenkinsci/bitbucket-branch-source-plugin/pull/294
          HPI: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/cloudbees-bitbucket-branch-source/2.7.1-rc629.f242e60a14b4/

          It falls back to heavyweight checkout, but at least PRs can be build with Bitbucket >= 7.x

          Show
          huber Kevin Huber added a comment - You can use the hpi published by this PR: https://github.com/jenkinsci/bitbucket-branch-source-plugin/pull/294 HPI: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/cloudbees-bitbucket-branch-source/2.7.1-rc629.f242e60a14b4/ It falls back to heavyweight checkout, but at least PRs can be build with Bitbucket >= 7.x
          Hide
          kon Kalle Niemitalo added a comment - - edited

          BSERV-12284 "Creating pull request does not reflect the stash-refs/pull-requests/<#> immediately" was moved to SHORT TERM BACKLOG on 2020-04-07. I wonder what Atlassian intends to do.

          It was then moved down to GATHERING INTEREST on 2020-04-14, and the priority was removed.

          Show
          kon Kalle Niemitalo added a comment - - edited BSERV-12284 "Creating pull request does not reflect the stash-refs/pull-requests/<#> immediately" was moved to SHORT TERM BACKLOG on 2020-04-07. I wonder what Atlassian intends to do. It was then moved down to GATHERING INTEREST on 2020-04-14, and the priority was removed.
          Hide
          kon Kalle Niemitalo added a comment -

          Bitbucket Branch Source 2.9.0 now includes Pull Request #294, which updates the documentation to say that lightweight checkout won't work with Bitbucket Server 7 when using the "Merging the pull request with the current target branch revision" strategy. (The pull request also includes other changes.)

          I still hope lightweight checkout can be made to work somehow. If it only worked in the trivial case where the file in question is identical in the source and target branches, that would cover most of my use.

          Show
          kon Kalle Niemitalo added a comment - Bitbucket Branch Source 2.9.0 now includes Pull Request #294 , which updates the documentation to say that lightweight checkout won't work with Bitbucket Server 7 when using the "Merging the pull request with the current target branch revision" strategy. (The pull request also includes other changes.) I still hope lightweight checkout can be made to work somehow. If it only worked in the trivial case where the file in question is identical in the source and target branches, that would cover most of my use.
          Hide
          bitwiseman Liam Newman added a comment - - edited

          Kalle Niemitalo
          Kevin Huber

          The linked comment below indicates that in BitBucket 7 Atlassian has removed the ability to do lightweight checkout while using the "Merging" strategy. This is intentional, by-design behavior implemented by Atlassian. It is beyond the control of Jenkins project to fix this.

          As a workaround, please try changing to use the "Current Pull Request revision" strategy instead and see if that works. 

          refs/pull-requests/*/merge has been fully removed in 7.0 and will never be created/updated by the system again. For pull requests that were opened prior to upgrading to 7.x, which have a merge ref, that ref will be removed the first time the pull request is updated (or when it's merged or declined, as ever). With the switch from 3-way diffs (which were built around displaying the merge's diff) to 2-way diffs (which use the common ancestor), the system no longer retains the merges it creates to check mergeability; it runs git merge to see what happens, and discards everything that results. This prevents the accumulation of "unreachable" pull request merge objects in repositories, which can (greatly) slow down clone/fetch/pull/push performance as they build up. Any tool that wishes to build a pull request's merge will need to create that merge on its own, going forward.

          Prior to 7.0, because pull request diffs were built around the merge, checking mergeability (which happens automatically after page load when a pull request is created via the UI) would create the public refs/pull-requests entries for that pull request. In 7.x, since the diff is now using the common ancestor and no longer needs the merge, checking mergeability doesn't create any refs. (This is shown in the log snippets above.) Instead, the refs are created when a pull request's diff is viewed. Plugins which are installed in Bitbucket Server can simulate this by either using the PullRequestService to stream the pull request's changes, or (more efficiently), by using ScmService.getPullRequestCommandFactory(pullRequest).effectiveDiff() (which PullRequestService uses under the hood). Remote callers can use rest/api/1.0/projects/KEY/repos/slug/pull-requests/N/changes?limit=1 to do the same. Any of those will ensure refs/pull-requests/*/from is created (or updated) to match the pull request's current state.

          I'd wager many webhook apps (or other apps that notify external systems, however they do so) have logic in them for using PullRequestService.canMerge, MergeRequestCheckService.check or ScmPullRequestCommandFactory.tryMerge to try and force pull request refs to be created/updated. (The Post Webhooks for Bitbucket app did, back when it was open source; the source has since been made private, or at least moved off Github, so I'm not sure these days.) Those approaches will not work in 7.x. Instead, such apps need to be updated to use PullRequestService.streamChanges or ScmPullRequestCommandFactory.effectiveDiff to force the from ref to be created/updated. As noted earlier, there is no way in 7.x to get a merge ref.

          I'll reiterate here that Bitbucket Server's pull request refs are not API, and have never been considered API, so the way they work is dangerous to rely on. I'll also readily acknowledge that that's hardly a satisfactory answer, to receive or to give, and that it would be nice if it changed. I am not a PM for Bitbucket Server, so I can't say when/if such improvements will hit the roadmap. Perhaps I'll see if I can use some upcoming 20% time to make some improvements. I'm sorry that, at least for the moment, I can't offer better.

          Best regards,

          Bryan Turner

          Atlassian Bitbucket

          https://jira.atlassian.com/browse/BSERV-12284?focusedCommentId=2389584&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-2389584

           

          If you do not like this behavior, tell Atlassian this feature is important to you. 

          Show
          bitwiseman Liam Newman added a comment - - edited Kalle Niemitalo Kevin Huber The linked comment below indicates that in BitBucket 7 Atlassian has removed the ability to do lightweight checkout while using the "Merging" strategy. This is intentional, by-design behavior implemented by Atlassian. It is beyond the control of Jenkins project to fix this. As a workaround, please try changing to use the "Current Pull Request revision" strategy instead and see if that works.  refs/pull-requests/*/merge has been fully removed in 7.0 and will never be created/updated by the system again. For pull requests that were opened prior to upgrading to 7.x, which have a merge ref, that ref will be removed the first time the pull request is updated (or when it's merged or declined, as ever). With the switch from 3-way diffs (which were built around displaying the merge's diff) to 2-way diffs (which use the common ancestor), the system no longer retains the merges it creates to check mergeability; it runs git merge to see what happens, and discards everything that results. This prevents the accumulation of "unreachable" pull request merge objects in repositories, which can (greatly) slow down clone/fetch/pull/push performance as they build up. Any tool that wishes to build a pull request's merge will need to create that merge on its own, going forward. Prior to 7.0, because pull request diffs were built around the merge, checking mergeability (which happens automatically after page load when a pull request is created via the UI) would create the public refs/pull-requests entries for that pull request. In 7.x, since the diff is now using the common ancestor and no longer needs the merge, checking mergeability doesn't create any refs. (This is shown in the log snippets above.) Instead, the refs are created when a pull request's diff is viewed. Plugins which are installed in Bitbucket Server can simulate this by either using the PullRequestService to stream the pull request's changes, or (more efficiently), by using ScmService.getPullRequestCommandFactory(pullRequest).effectiveDiff() (which PullRequestService uses under the hood). Remote callers can use rest/api/1.0/projects/KEY/repos/slug/pull-requests/N/changes?limit=1 to do the same. Any of those will ensure refs/pull-requests/*/from is created (or updated) to match the pull request's current state. I'd wager many webhook apps (or other apps that notify external systems, however they do so) have logic in them for using PullRequestService.canMerge, MergeRequestCheckService.check or ScmPullRequestCommandFactory.tryMerge to try and force pull request refs to be created/updated. (The Post Webhooks for Bitbucket app did, back when it was open source; the source has since been made private, or at least moved off Github, so I'm not sure these days.) Those approaches will not work in 7.x. Instead, such apps need to be updated to use PullRequestService.streamChanges or ScmPullRequestCommandFactory.effectiveDiff to force the from ref to be created/updated. As noted earlier, there is no way in 7.x to get a merge ref. I'll reiterate here that Bitbucket Server's pull request refs are not API, and have never been considered API, so the way they work is dangerous to rely on. I'll also readily acknowledge that that's hardly a satisfactory answer, to receive or to give, and that it would be nice if it changed. I am not a PM for Bitbucket Server, so I can't say when/if such improvements will hit the roadmap. Perhaps I'll see if I can use some upcoming 20% time to make some improvements. I'm sorry that, at least for the moment, I can't offer better. Best regards, Bryan Turner Atlassian Bitbucket https://jira.atlassian.com/browse/BSERV-12284?focusedCommentId=2389584&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-2389584   If you do not like this behavior, tell Atlassian this feature is important to you. 
          Hide
          kon Kalle Niemitalo added a comment -

          "Current Pull Request revision" is not useful for me because:

          • I have Jenkins already building the branches that are used as sources of the pull requests.
          • I often have pull requests merging the same source branch to multiple target branches.
          • I want Jenkins to detect problems that only exist in the merged version, rather than in the source or target branches. E.g. if the source branch added calls of a method that was meanwhile deleted from the target branch.
          • It generates non-descriptive job titles like "PR-1234". (JENKINS-55348)

          It seems to me that the trivial case where the file in question is identical in the source and target branches could be implemented in bitbucket-branch-source-plugin without changing Bitbucket Server, as follows:

          1. The Jenkins master wants to read Jenkinsfile in a multibranch project configured with "Merging the pull request with the current target branch revision".
          2. It uses lightweight checkout to read Jenkinsfile from the source branch.
          3. It also uses lightweight checkout to read Jenkinsfile from the target branch.
          4. If Jenkinsfile is bit-for-bit identical in both branches, then the Jenkins master uses that.
          5. Otherwise, the Jenkins master uses heavyweight checkout and merge. To minimize disk usage and network I/O on the Jenkins master, this could perhaps be a sparse checkout with Git LFS disabled.
          Show
          kon Kalle Niemitalo added a comment - "Current Pull Request revision" is not useful for me because: I have Jenkins already building the branches that are used as sources of the pull requests. I often have pull requests merging the same source branch to multiple target branches. I want Jenkins to detect problems that only exist in the merged version, rather than in the source or target branches. E.g. if the source branch added calls of a method that was meanwhile deleted from the target branch. It generates non-descriptive job titles like "PR-1234". ( JENKINS-55348 ) It seems to me that the trivial case where the file in question is identical in the source and target branches could be implemented in bitbucket-branch-source-plugin without changing Bitbucket Server, as follows: The Jenkins master wants to read Jenkinsfile in a multibranch project configured with "Merging the pull request with the current target branch revision". It uses lightweight checkout to read Jenkinsfile from the source branch. It also uses lightweight checkout to read Jenkinsfile from the target branch. If Jenkinsfile is bit-for-bit identical in both branches, then the Jenkins master uses that. Otherwise, the Jenkins master uses heavyweight checkout and merge. To minimize disk usage and network I/O on the Jenkins master, this could perhaps be a sparse checkout with Git LFS disabled.
          Hide
          bitwiseman Liam Newman added a comment -

          Kalle Niemitalo

          I've been informed of a possible fix.  Please try enabling "Call changes API" for your Bitbucket 7 endpoints.  See if that fixes the issue for you.  

          This wasn't turned on by default because it makes Jenkins make additional API calls, which some users did not want.  I think it makes sense to force this to "enabled" when using "Merging the pull request with the current target branch revision" on BB 7.x, but at very least there should be a helpful messing pointing to how to get lightweight checkout. 

          Show
          bitwiseman Liam Newman added a comment - Kalle Niemitalo I've been informed of a possible fix.  Please try enabling "Call changes API" for your Bitbucket 7 endpoints.  See if that fixes the issue for you.   This wasn't turned on by default because it makes Jenkins make additional API calls, which some users did not want.  I think it makes sense to force this to "enabled" when using "Merging the pull request with the current target branch revision" on BB 7.x, but at very least there should be a helpful messing pointing to how to get lightweight checkout. 
          Hide
          kon Kalle Niemitalo added a comment - - edited

          Liam Newman, the "Call Changes api" setting on the "Configure System" page is intended to make Bitbucket Server update its "refs/pull-requests/*/from" refs that would be used by the "Current Pull Request revision" strategy. I have tested that it does not help with the "refs/pull-requests/*/merge" refs that the "Merging the pull request with the current target branch revision" strategy tries to use for lightweight checkout. I would like to keep this issue focused on the latter.

          This test was with Jenkins 2.235.5, Bitbucket Branch Source Plugin 2.9.2, and Bitbucket Server 7.1.1.

          Show
          kon Kalle Niemitalo added a comment - - edited Liam Newman , the "Call Changes api" setting on the "Configure System" page is intended to make Bitbucket Server update its "refs/pull-requests/*/from" refs that would be used by the "Current Pull Request revision" strategy. I have tested that it does not help with the "refs/pull-requests/*/merge" refs that the "Merging the pull request with the current target branch revision" strategy tries to use for lightweight checkout. I would like to keep this issue focused on the latter. This test was with Jenkins 2.235.5, Bitbucket Branch Source Plugin 2.9.2, and Bitbucket Server 7.1.1.
          Hide
          gmshake Zhenlei Huang added a comment -

          Hi Kalle Niemitalo,

          The "Call Changes api" enforce Bitbucket Server update its "refs/pull-requests/*/from", regardless the strategy.

          The lightweight checkout was intended to work with hosting vendor that provide merged refs, such as GitHub, Bitbucket 6.x. This feature offloads the merging process to remote hosting other than Jenkins itself.

          Show
          gmshake Zhenlei Huang added a comment - Hi Kalle Niemitalo , The "Call Changes api" enforce Bitbucket Server update its "refs/pull-requests/*/from", regardless the strategy. The lightweight checkout was intended to work with hosting vendor that provide merged refs, such as GitHub, Bitbucket 6.x. This feature offloads the merging process to remote hosting other than Jenkins itself.
          Hide
          gmshake Zhenlei Huang added a comment -

          Hi, Liam Newman

          I'd propose reimplement the "lightweight checkout" feature to also work with vendors that does not provide merged refs. A straight simple way is we fetch the target ref and source ref, then merge locally in Jenkins temp workspace, producing the merged version.

          As for single file "lightweight checkout", we can create a temp local git repository, fetch the file from source ref url and target ref url, and then merge them in the created local git repository.

          Compared to the legacy "lightweight checkout", this approach does have cons:

          1. It will need one more http request / git pull request
          2. Need some resources on Jenkins instance to do the merge.

          How do you think this approach?

          Show
          gmshake Zhenlei Huang added a comment - Hi, Liam Newman I'd propose reimplement the "lightweight checkout" feature to also work with vendors that does not provide merged refs. A straight simple way is we fetch the target ref and source ref, then merge locally in Jenkins temp workspace, producing the merged version. As for single file "lightweight checkout", we can create a temp local git repository, fetch the file from source ref url and target ref url, and then merge them in the created local git repository. Compared to the legacy "lightweight checkout", this approach does have cons: It will need one more http request / git pull request Need some resources on Jenkins instance to do the merge. How do you think this approach?
          Hide
          kon Kalle Niemitalo added a comment - - edited

          Zhenlei Huang, I don't think you can merge just two versions of a file such as Jenkinsfile; you need a base version as well. Otherwise, you wouldn't know whether differences mean that lines were added or that they were removed. As I mentioned in the description of this issue, the Bitbucket Server REST API does not seem to provide a way to get the commit ID or file content from the merge base of a pull request. This is why I proposed only implementing the trivial case where the two versions of the file are identical (although the PR has modified other files) and there is no need for an actual merge.

          Show
          kon Kalle Niemitalo added a comment - - edited Zhenlei Huang , I don't think you can merge just two versions of a file such as Jenkinsfile; you need a base version as well. Otherwise, you wouldn't know whether differences mean that lines were added or that they were removed. As I mentioned in the description of this issue, the Bitbucket Server REST API does not seem to provide a way to get the commit ID or file content from the merge base of a pull request. This is why I proposed only implementing the trivial case where the two versions of the file are identical (although the PR has modified other files) and there is no need for an actual merge.
          Hide
          bitwiseman Liam Newman added a comment -

          Kalle Niemitalo
          Thank you for testing that. I can confirm you are correct. Without the "Call Changes API" neither of the merge strategies work at all for new PRs - the `/from` ref is missing and the run fails immediately.

          Zhenlei Huang and Kalle Niemitalo

          A straight simple way is we fetch the target ref and source ref, then merge locally in Jenkins temp workspace, producing the merged version.

          This is why I proposed only implementing the trivial case where the two versions of the file are identical (although the PR has modified other files) and there is no need for an actual merge.

          The solutions you both are advocating are both possible, but I would not advise them. The number of edge cases (and not-so-edge complexities) in this area is high. Any solution that doesn't use an scm-provider-side solution, is likely be a bug farm.
          Zhenlei Huang - Kalle pointed to some limitation to what you proposed. Also, what about forks? The current implementation expects to find all files in the target repo, switching to a fork for some would be problematic.
          Kalle Niemitalo Your proposal is simpler but it would only work if the pipeline does not call any of the steps that do additional reads (readTrusted, etc), and there is no way to know if those steps are present until during the run.

          Overall, what you are suggesting is doing a lot of work to mitigate something that is almost trivial to do on on the SCM server. You would be better served by creating a plugin/app for Bitbucket that implements the merge ref creation.

          If you would like to implement and provide indefinite support for this case, I'm open to taking PRs that add functionality as long is it off by default.

          Show
          bitwiseman Liam Newman added a comment - Kalle Niemitalo Thank you for testing that. I can confirm you are correct. Without the "Call Changes API" neither of the merge strategies work at all for new PRs - the `/from` ref is missing and the run fails immediately. Zhenlei Huang and Kalle Niemitalo A straight simple way is we fetch the target ref and source ref, then merge locally in Jenkins temp workspace, producing the merged version. This is why I proposed only implementing the trivial case where the two versions of the file are identical (although the PR has modified other files) and there is no need for an actual merge. The solutions you both are advocating are both possible, but I would not advise them. The number of edge cases (and not-so-edge complexities) in this area is high. Any solution that doesn't use an scm-provider-side solution, is likely be a bug farm. Zhenlei Huang - Kalle pointed to some limitation to what you proposed. Also, what about forks? The current implementation expects to find all files in the target repo, switching to a fork for some would be problematic. Kalle Niemitalo Your proposal is simpler but it would only work if the pipeline does not call any of the steps that do additional reads (readTrusted, etc), and there is no way to know if those steps are present until during the run. Overall, what you are suggesting is doing a lot of work to mitigate something that is almost trivial to do on on the SCM server. You would be better served by creating a plugin/app for Bitbucket that implements the merge ref creation. If you would like to implement and provide indefinite support for this case, I'm open to taking PRs that add functionality as long is it off by default.
          Hide
          kon Kalle Niemitalo added a comment -

          Liam Newman, I guess I should then file a separate request to allow disabling Git LFS in the "Merging the pull request with the current target branch revision" heavyweight checkout at the Jenkins master. That should at least be simple to implement and would reduce the disk space and network I/O costs in my scenario.

          Show
          kon Kalle Niemitalo added a comment - Liam Newman , I guess I should then file a separate request to allow disabling Git LFS in the "Merging the pull request with the current target branch revision" heavyweight checkout at the Jenkins master. That should at least be simple to implement and would reduce the disk space and network I/O costs in my scenario.
          Hide
          gmshake Zhenlei Huang added a comment -

          There's another edge case not covered by my previous proposal. Say we have a file named 'Jenkinsfile.old' in base branch, and it was renamed to 'Jenkinsfile' in PR's head branch, it looks difficult to get the original file name unless SCM server provide the information in the PR. 

          Liam Newman,
          I agree with you that it is trivial to do on the SCM server. It's frustrating to hack on Jenkins' side

          Show
          gmshake Zhenlei Huang added a comment - There's another edge case not covered by my previous proposal. Say we have a file named 'Jenkinsfile.old' in base branch, and it was renamed to 'Jenkinsfile' in PR's head branch, it looks difficult to get the original file name unless SCM server provide the information in the PR.  Liam Newman , I agree with you that it is trivial to do on the SCM server. It's frustrating to hack on Jenkins' side

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            kon Kalle Niemitalo
            Votes:
            13 Vote for this issue
            Watchers:
            17 Start watching this issue

              Dates

              Created:
              Updated: