Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63033

Implement Lightweight Checkout for Multibranch Pipeline Jobs

    XMLWordPrintable

Details

    • 3.0.0

    Description

      I tried to migrate a multibranch project from Bitbucket Branch Source Plugin to Bitbucket Server Integration. Before the migration, the Jenkins master read just the Jenkinsfile of each branch from Bitbucket Server over HTTP, and the Jenkins agent then cloned the repository and submodules and pulled Git LFS files. After the migration, the Jenkins master cloned all branches of the repository and pulled Git LFS files. We cannot afford to let the Jenkins master do that, because of time and disk space. I had to revert the migration for this project.

      Please change the Bitbucket Server Integration plugin to support lightweight checkout for Jenkinsfile. Perhaps that is part of "SCM API 2.0" JENKINS-62002; I don't know.

      Attachments

        Issue Links

          Activity

            Now that JENKINS-63069 has been fixed, this JENKINS-63033 is the only known issue blocking our migration from Bitbucket Branch Source Plugin.

            kon Kalle Niemitalo added a comment - Now that JENKINS-63069 has been fixed, this JENKINS-63033 is the only known issue blocking our migration from Bitbucket Branch Source Plugin.

            Lightweight checkout in Bitbucket Branch Source Plugin was requested in JENKINS-42518 and implemented in PR#78. The diffstat was:

            58      0       src/main/java/com/cloudbees/jenkins/plugins/bitbucket/Utils.java
            23      1       src/main/java/com/cloudbees/jenkins/plugins/bitbucket/api/BitbucketApi.java
            57      5       src/main/java/com/cloudbees/jenkins/plugins/bitbucket/client/BitbucketCloudApiClient.java
            82      0       src/main/java/com/cloudbees/jenkins/plugins/bitbucket/client/repository/BitbucketRepositorySource.java
            104     0       src/main/java/com/cloudbees/jenkins/plugins/bitbucket/filesystem/BitbucketSCMFile.java
            147     0       src/main/java/com/cloudbees/jenkins/plugins/bitbucket/filesystem/BitbucketSCMFileSystem.java
            81      10      src/main/java/com/cloudbees/jenkins/plugins/bitbucket/server/client/BitbucketServerAPIClient.java
            4       0       src/main/java/com/cloudbees/jenkins/plugins/bitbucket/server/client/BitbucketServerApiFactory.java
            

            The main thing seems to be class BitbucketSCMFileSystem (source, Javadoc), which extends SCMFileSystem (source, Javadoc). In contrast, the source code of Bitbucket Server Integration never mentions SCMFileSystem.

            From SCM API Implementation Guide, I get the impression that SCMProbe (source, Javadoc) is an alternative to SCMFileSystem. Because the source code of Bitbucket Branch Source Plugin does not mention SCMProbe, I think Bitbucket Server Integration need not implement that.

            The BitbucketServerAPIClient.getFileContent method in Bitbucket Branch Source Plugin uses the "/rest/api/1.0/projects/{owner}/repos/{repo}/browse{/path*}{?at}" (REST API doc) URI template and apparently supports UTF-8 text files only. I wonder why that was chosen instead of "/rest/api/1.0/projects/{owner}/repos/{repo}/raw{/path*}{?at}" (REST API doc), which would respond with an octet stream.

            kon Kalle Niemitalo added a comment - Lightweight checkout in Bitbucket Branch Source Plugin was requested in JENKINS-42518 and implemented in PR#78 . The diffstat was: 58 0 src/main/java/com/cloudbees/jenkins/plugins/bitbucket/Utils.java 23 1 src/main/java/com/cloudbees/jenkins/plugins/bitbucket/api/BitbucketApi.java 57 5 src/main/java/com/cloudbees/jenkins/plugins/bitbucket/client/BitbucketCloudApiClient.java 82 0 src/main/java/com/cloudbees/jenkins/plugins/bitbucket/client/repository/BitbucketRepositorySource.java 104 0 src/main/java/com/cloudbees/jenkins/plugins/bitbucket/filesystem/BitbucketSCMFile.java 147 0 src/main/java/com/cloudbees/jenkins/plugins/bitbucket/filesystem/BitbucketSCMFileSystem.java 81 10 src/main/java/com/cloudbees/jenkins/plugins/bitbucket/server/client/BitbucketServerAPIClient.java 4 0 src/main/java/com/cloudbees/jenkins/plugins/bitbucket/server/client/BitbucketServerApiFactory.java The main thing seems to be class BitbucketSCMFileSystem ( source , Javadoc ), which extends SCMFileSystem ( source , Javadoc ). In contrast, the source code of Bitbucket Server Integration never mentions SCMFileSystem . From SCM API Implementation Guide , I get the impression that SCMProbe ( source , Javadoc ) is an alternative to SCMFileSystem . Because the source code of Bitbucket Branch Source Plugin does not mention SCMProbe , I think Bitbucket Server Integration need not implement that. The BitbucketServerAPIClient.getFileContent method in Bitbucket Branch Source Plugin uses the "/rest/api/1.0/projects/{owner}/repos/{repo}/browse{/path*}{?at}" ( REST API doc ) URI template and apparently supports UTF-8 text files only. I wonder why that was chosen instead of "/rest/api/1.0/projects/{owner}/repos/{repo}/raw{/path*}{?at}" ( REST API doc ), which would respond with an octet stream.
            macdrega Joerg Schwaerzler added a comment - - edited

            Will this ticket cover lightweight checkout for pull requests, too? See also: JENKINS-60342

            macdrega Joerg Schwaerzler added a comment - - edited Will this ticket cover lightweight checkout for pull requests, too? See also: JENKINS-60342

            This would just bring the same capability that the Bitbucket Branch Source plugin already has. Lightweight checkout for the result of merging a pull request no longer works in the Bitbucket Branch Source plugin used with Bitbucket Server (JENKINS-61493), and the Bitbucket Server Integration plugin would then have the same restriction.

            kon Kalle Niemitalo added a comment - This would just bring the same capability that the Bitbucket Branch Source plugin already has. Lightweight checkout for the result of merging a pull request no longer works in the Bitbucket Branch Source plugin used with Bitbucket Server ( JENKINS-61493 ), and the Bitbucket Server Integration plugin would then have the same restriction.
            mhenschke_atlassian Martin Henschke added a comment - - edited

            I am spiking this ticket at the moment. For anyone watching/waiting on this ticket, are you interested in this feature for pipeline jobs (as opposed to Multibranch Pipeline jobs)? I have been spiking the solution and am not confident a functionally complete solution is possible given how pipeline jobs are implemented, so we're considering just releasing support for Multibranch for now. If anyone is interested in this for pipeline specifically, please file a separate ticket.
            EDIT: After a bit more investigation I've confirmed it's impossible for every case, but we can do pipeline so long as the branch selector points to a single branch (no wildcards/selectors)- it appears this is what the GitSCM implementation does so we'll follow suit.

            mhenschke_atlassian Martin Henschke added a comment - - edited I am spiking this ticket at the moment. For anyone watching/waiting on this ticket, are you interested in this feature for pipeline jobs (as opposed to Multibranch Pipeline jobs)? I have been spiking the solution and am not confident a functionally complete solution is possible given how pipeline jobs are implemented, so we're considering just releasing support for Multibranch for now. If anyone is interested in this for pipeline specifically, please file a separate ticket. EDIT: After a bit more investigation I've confirmed it's impossible for every case, but we can do pipeline so long as the branch selector points to a single branch (no wildcards/selectors)- it appears this is what the GitSCM implementation does so we'll follow suit.
            kon Kalle Niemitalo added a comment - - edited mhenschke_atlassian , what do you mean with "spiking"? I imagine it is not "5. (figuratively, journalism) To discard; to decide not to publish or make public."

            kon quite the opposite! I meant spike in the sense of minimal prototyping.

            mhenschke_atlassian Martin Henschke added a comment - kon quite the opposite! I meant spike in the sense of minimal prototyping .
            kgiloo kgiloo added a comment -

            what is the aftermath of the issue? Hard to see it clearly here.

             

            kgiloo kgiloo added a comment - what is the aftermath of the issue? Hard to see it clearly here.  

            People

              mhenschke_atlassian Martin Henschke
              kon Kalle Niemitalo
              Votes:
              7 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: