Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42518

Provide lightweight checkout capability for bitbucket to avoid repository clone for multi-branch pipeline jobs

    • Blue Ocean 1.4 - beta 2

      JENKINS-33273 introduced a lightweight checkout capability to allow SCM implementors to avoid having checkout the scm on the master to read the contents of the Jenkinsfile from the SCM.  The comments in that bug mention that git doesn't offer a way to do so but I would think this is possible to do using Bitbucket APIs to read a particular file's contents on a branch.  

      In our use case, our repository is multi-gigabyte and we have many developers working within it creating many branches so cloning the repo on master for every branch is very costly in terms of time and storage.

          [JENKINS-42518] Provide lightweight checkout capability for bitbucket to avoid repository clone for multi-branch pipeline jobs

          Peter Hayes created issue -

          Patrick Wolf added a comment -

          This is already included in workflow-cps-plugin (core Pipeline) version 2.29 released on March 3.

          https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Groovy+Plugin

          Patrick Wolf added a comment - This is already included in workflow-cps-plugin (core Pipeline) version 2.29 released on March 3. https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Groovy+Plugin

          Peter Hayes added a comment -

          I just downloaded the very latest Jenkins, updated all plugins and created a Multibranch Pipeline job configuring Bitbucket as the SCM supplying my Bitbucket Server Web credentials and I still get an "@script" clone for each branch identified as well as a clone for the pipeline checkout step.  

          Peter Hayes added a comment - I just downloaded the very latest Jenkins, updated all plugins and created a Multibranch Pipeline job configuring Bitbucket as the SCM supplying my Bitbucket Server Web credentials and I still get an "@script" clone for each branch identified as well as a clone for the pipeline checkout step.  
          Patrick Wolf made changes -
          Assignee Original: Antonio Muñiz [ amuniz ] New: CloudBees Inc. [ cloudbees ]
          Patrick Wolf made changes -
          Link New: This issue is related to JENKINS-33272 [ JENKINS-33272 ]

          Albert V added a comment -

          I've got an updated Jenkins and plugins too and I've experimenting the same issue using a Multibranch job with Bitbucket as a source.

          I have a repo of 2,6GB with 2 branches and in the Jenkins Master workspace directory I found:
          2,6G part_of_the_branch_name_1-UBEA5XUZNXNXC3VOM6COTGUSIEU26KDGQPKDMGV36VVYRJ6MBLRQ@script
          2,6G part_of_the_branch_name_1-BMHC5R4EOUCRCXQQ77PBNZ2YTY6J5NEQLJ5DXELS3CWYZLG3BJQA@script

          I'm assuming that the famous JENKINS-33273 issue is not applied in the Bitbucket sources case.
          Would if be possible to make a lightweight "checkout" of the Jenkinsfile with something like https://bitbucketServer.url/projects/project-name/repos/repo-name/browse/Jenkinsfile?raw ???

          Nowadays, as it is working right now is making my CI/CD environment un-maintainable.

          Albert V added a comment - I've got an updated Jenkins and plugins too and I've experimenting the same issue using a Multibranch job with Bitbucket as a source. I have a repo of 2,6GB with 2 branches and in the Jenkins Master workspace directory I found: 2,6G part_of_the_branch_name_1-UBEA5XUZNXNXC3VOM6COTGUSIEU26KDGQPKDMGV36VVYRJ6MBLRQ@script 2,6G part_of_the_branch_name_1-BMHC5R4EOUCRCXQQ77PBNZ2YTY6J5NEQLJ5DXELS3CWYZLG3BJQA@script I'm assuming that the famous JENKINS-33273 issue is not applied in the Bitbucket sources case. Would if be possible to make a lightweight "checkout" of the Jenkinsfile with something like https://bitbucketServer.url/projects/project-name/repos/repo-name/browse/Jenkinsfile?raw ??? Nowadays, as it is working right now is making my CI/CD environment un-maintainable.

          Andrew Reslan added a comment -

          I have a similar situation, my Bitbucket Git Repo is 3GB, it takes at least 1hr to clone with jenkins and Bitbucket on the same LAN.

          The bitbucket-branch-source-plugin has resolved the initial scan for the branches with the Jenkinsfile's,

          I am now looking at how to minimize the time required to checkout the individual Jenkinsfile's, using the BitBucket REST API to checkout the raw contents of the Jenkinsfile looks like a potential option?

          https://api.bitbucket.org/1.0/repositories/\{accountname}/{repo_slug}/raw/{revision}/{path}

          Andrew Reslan added a comment - I have a similar situation, my Bitbucket Git Repo is 3GB, it takes at least 1hr to clone with jenkins and Bitbucket on the same LAN. The bitbucket-branch-source-plugin has resolved the initial scan for the branches with the Jenkinsfile's, I am now looking at how to minimize the time required to checkout the individual Jenkinsfile's, using the BitBucket REST API to checkout the raw contents of the Jenkinsfile looks like a potential option? https://api.bitbucket.org/1.0/repositories/\ {accountname}/{repo_slug}/raw/{revision}/{path}

          Utopic Men added a comment -

          With an updated Jenkins instance, I have a weir behavior.

          I created 2 projects : 

          • a Bitbucket Team/Project and
          • a Multibranch Pipeline.

          The first one still create this "script" folder (my repo is 4 Go) so I experience the same issue described above.

          But the Multibranch pipeline works perfectly without creating the script folder.

          So I don't think it's Github or Bitbucket related...

          Utopic Men added a comment - With an updated Jenkins instance, I have a weir behavior. I created 2 projects :  a Bitbucket Team/Project and a Multibranch Pipeline. The first one still create this "script" folder (my repo is 4 Go) so I experience the same issue described above. But the Multibranch pipeline works perfectly without creating the script folder. So I don't think it's Github or Bitbucket related...

          Utopic Men added a comment -

          I still want to use this Bitbucket plugin because of the PR integration (build status pushed to Bibucket).

          Utopic Men added a comment - I still want to use this Bitbucket plugin because of the PR integration (build status pushed to Bibucket).

          Just adding my observations to this:

          On the master node, for every branch of every repo, I also get different behavior when using GitSCMSource vs BitbucketSCMSource plugins.

          • GitSCMSource: No "@script" folder, Jenkinsfile fetched directly from Git without requiring a checkout, but shared libraries put in the "@libs" folder when using:
            <sources class="jenkins.branch.MultiBranchProject$BranchSourceList">
               <data>
                  <jenkins.branch.BranchSource>
                     <source class="jenkins.plugins.git.GitSCMSource">
                        ...
          • ..causing a minimal footprint

           

          • BitbucketSCMSource: Git clone with full history (sometimes years of history) and checkout (minus LFS objects, also no submodules) in workspace "@script" folder when using:
            <sources class="jenkins.branch.MultiBranchProject$BranchSourceList">
               <data>
                  <jenkins.branch.BranchSource plugin="branch-api">
                     <source plugin="cloudbees-bitbucket-branch-source" class="com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource">
                        ...
          • ..causing a huge footprint, hundreds of megabytes and for some projects several GBs for every branch in every repo

           

          Anders Holmblad added a comment - Just adding my observations to this: On the master node, for every branch of every repo, I also get different behavior when using GitSCMSource vs BitbucketSCMSource plugins. GitSCMSource: No "@script" folder, Jenkinsfile fetched directly from Git without requiring a checkout, but shared libraries put in the "@libs" folder when using: <sources class="jenkins.branch.MultiBranchProject$BranchSourceList">     <data>       <jenkins.branch.BranchSource>           <source class="jenkins.plugins.git.GitSCMSource">             ... ..causing a minimal footprint   BitbucketSCMSource: Git clone with full history (sometimes years of history) and checkout (minus LFS objects, also no submodules) in workspace "@script" folder when using: <sources class="jenkins.branch.MultiBranchProject$BranchSourceList">     <data>       <jenkins.branch.BranchSource plugin="branch-api">           <source plugin="cloudbees-bitbucket-branch-source" class="com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource">             ... ..causing a huge footprint, hundreds of megabytes and for some projects several GBs for every branch in every repo  

            vivek Vivek Pandey
            petehayes Peter Hayes
            Votes:
            18 Vote for this issue
            Watchers:
            33 Start watching this issue

              Created:
              Updated:
              Resolved: