Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63033

Implement Lightweight Checkout for Multibranch Pipeline Jobs

    • 3.0.0

      I tried to migrate a multibranch project from Bitbucket Branch Source Plugin to Bitbucket Server Integration. Before the migration, the Jenkins master read just the Jenkinsfile of each branch from Bitbucket Server over HTTP, and the Jenkins agent then cloned the repository and submodules and pulled Git LFS files. After the migration, the Jenkins master cloned all branches of the repository and pulled Git LFS files. We cannot afford to let the Jenkins master do that, because of time and disk space. I had to revert the migration for this project.

      Please change the Bitbucket Server Integration plugin to support lightweight checkout for Jenkinsfile. Perhaps that is part of "SCM API 2.0" JENKINS-62002; I don't know.

          [JENKINS-63033] Implement Lightweight Checkout for Multibranch Pipeline Jobs

          Now that JENKINS-63069 has been fixed, this JENKINS-63033 is the only known issue blocking our migration from Bitbucket Branch Source Plugin.

          Kalle Niemitalo added a comment - Now that JENKINS-63069 has been fixed, this JENKINS-63033 is the only known issue blocking our migration from Bitbucket Branch Source Plugin.

          Lightweight checkout in Bitbucket Branch Source Plugin was requested in JENKINS-42518 and implemented in PR#78. The diffstat was:

          58      0       src/main/java/com/cloudbees/jenkins/plugins/bitbucket/Utils.java
          23      1       src/main/java/com/cloudbees/jenkins/plugins/bitbucket/api/BitbucketApi.java
          57      5       src/main/java/com/cloudbees/jenkins/plugins/bitbucket/client/BitbucketCloudApiClient.java
          82      0       src/main/java/com/cloudbees/jenkins/plugins/bitbucket/client/repository/BitbucketRepositorySource.java
          104     0       src/main/java/com/cloudbees/jenkins/plugins/bitbucket/filesystem/BitbucketSCMFile.java
          147     0       src/main/java/com/cloudbees/jenkins/plugins/bitbucket/filesystem/BitbucketSCMFileSystem.java
          81      10      src/main/java/com/cloudbees/jenkins/plugins/bitbucket/server/client/BitbucketServerAPIClient.java
          4       0       src/main/java/com/cloudbees/jenkins/plugins/bitbucket/server/client/BitbucketServerApiFactory.java
          

          The main thing seems to be class BitbucketSCMFileSystem (source, Javadoc), which extends SCMFileSystem (source, Javadoc). In contrast, the source code of Bitbucket Server Integration never mentions SCMFileSystem.

          From SCM API Implementation Guide, I get the impression that SCMProbe (source, Javadoc) is an alternative to SCMFileSystem. Because the source code of Bitbucket Branch Source Plugin does not mention SCMProbe, I think Bitbucket Server Integration need not implement that.

          The BitbucketServerAPIClient.getFileContent method in Bitbucket Branch Source Plugin uses the "/rest/api/1.0/projects/{owner}/repos/{repo}/browse{/path*}{?at}" (REST API doc) URI template and apparently supports UTF-8 text files only. I wonder why that was chosen instead of "/rest/api/1.0/projects/{owner}/repos/{repo}/raw{/path*}{?at}" (REST API doc), which would respond with an octet stream.

          Kalle Niemitalo added a comment - Lightweight checkout in Bitbucket Branch Source Plugin was requested in JENKINS-42518 and implemented in PR#78 . The diffstat was: 58 0 src/main/java/com/cloudbees/jenkins/plugins/bitbucket/Utils.java 23 1 src/main/java/com/cloudbees/jenkins/plugins/bitbucket/api/BitbucketApi.java 57 5 src/main/java/com/cloudbees/jenkins/plugins/bitbucket/client/BitbucketCloudApiClient.java 82 0 src/main/java/com/cloudbees/jenkins/plugins/bitbucket/client/repository/BitbucketRepositorySource.java 104 0 src/main/java/com/cloudbees/jenkins/plugins/bitbucket/filesystem/BitbucketSCMFile.java 147 0 src/main/java/com/cloudbees/jenkins/plugins/bitbucket/filesystem/BitbucketSCMFileSystem.java 81 10 src/main/java/com/cloudbees/jenkins/plugins/bitbucket/server/client/BitbucketServerAPIClient.java 4 0 src/main/java/com/cloudbees/jenkins/plugins/bitbucket/server/client/BitbucketServerApiFactory.java The main thing seems to be class BitbucketSCMFileSystem ( source , Javadoc ), which extends SCMFileSystem ( source , Javadoc ). In contrast, the source code of Bitbucket Server Integration never mentions SCMFileSystem . From SCM API Implementation Guide , I get the impression that SCMProbe ( source , Javadoc ) is an alternative to SCMFileSystem . Because the source code of Bitbucket Branch Source Plugin does not mention SCMProbe , I think Bitbucket Server Integration need not implement that. The BitbucketServerAPIClient.getFileContent method in Bitbucket Branch Source Plugin uses the "/rest/api/1.0/projects/{owner}/repos/{repo}/browse{/path*}{?at}" ( REST API doc ) URI template and apparently supports UTF-8 text files only. I wonder why that was chosen instead of "/rest/api/1.0/projects/{owner}/repos/{repo}/raw{/path*}{?at}" ( REST API doc ), which would respond with an octet stream.

          Joerg Schwaerzler added a comment - - edited

          Will this ticket cover lightweight checkout for pull requests, too? See also: JENKINS-60342

          Joerg Schwaerzler added a comment - - edited Will this ticket cover lightweight checkout for pull requests, too? See also: JENKINS-60342

          This would just bring the same capability that the Bitbucket Branch Source plugin already has. Lightweight checkout for the result of merging a pull request no longer works in the Bitbucket Branch Source plugin used with Bitbucket Server (JENKINS-61493), and the Bitbucket Server Integration plugin would then have the same restriction.

          Kalle Niemitalo added a comment - This would just bring the same capability that the Bitbucket Branch Source plugin already has. Lightweight checkout for the result of merging a pull request no longer works in the Bitbucket Branch Source plugin used with Bitbucket Server ( JENKINS-61493 ), and the Bitbucket Server Integration plugin would then have the same restriction.

          Martin Henschke added a comment - - edited

          I am spiking this ticket at the moment. For anyone watching/waiting on this ticket, are you interested in this feature for pipeline jobs (as opposed to Multibranch Pipeline jobs)? I have been spiking the solution and am not confident a functionally complete solution is possible given how pipeline jobs are implemented, so we're considering just releasing support for Multibranch for now. If anyone is interested in this for pipeline specifically, please file a separate ticket.
          EDIT: After a bit more investigation I've confirmed it's impossible for every case, but we can do pipeline so long as the branch selector points to a single branch (no wildcards/selectors)- it appears this is what the GitSCM implementation does so we'll follow suit.

          Martin Henschke added a comment - - edited I am spiking this ticket at the moment. For anyone watching/waiting on this ticket, are you interested in this feature for pipeline jobs (as opposed to Multibranch Pipeline jobs)? I have been spiking the solution and am not confident a functionally complete solution is possible given how pipeline jobs are implemented, so we're considering just releasing support for Multibranch for now. If anyone is interested in this for pipeline specifically, please file a separate ticket. EDIT: After a bit more investigation I've confirmed it's impossible for every case, but we can do pipeline so long as the branch selector points to a single branch (no wildcards/selectors)- it appears this is what the GitSCM implementation does so we'll follow suit.

          Kalle Niemitalo added a comment - - edited

          Kalle Niemitalo added a comment - - edited mhenschke_atlassian , what do you mean with "spiking"? I imagine it is not "5. (figuratively, journalism) To discard; to decide not to publish or make public."

          kon quite the opposite! I meant spike in the sense of minimal prototyping.

          Martin Henschke added a comment - kon quite the opposite! I meant spike in the sense of minimal prototyping .

          kgiloo added a comment -

          what is the aftermath of the issue? Hard to see it clearly here.

           

          kgiloo added a comment - what is the aftermath of the issue? Hard to see it clearly here.  

          Vic added a comment -

          Does this issue have been resolved now?

          Because we still meet this issue in Bitbucket 7.21.x

          Vic added a comment - Does this issue have been resolved now? Because we still meet this issue in Bitbucket 7.21.x

            mhenschke_atlassian Martin Henschke
            kon Kalle Niemitalo
            Votes:
            7 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated:
              Resolved: