• Icon: New Feature New Feature
    • Resolution: Unresolved
    • Icon: Minor Minor
    • git-plugin
    • None

      Git supports filter commands since some time, so i'm now interested in blobless filter.

      With that filter the clone and fetch commands copy the full tree, but only required blobs.
      This has two improvements:

      • less transfer and repo size
      • still full git history for tools which need it

          [JENKINS-70094] Support blobless clone

          Gene added a comment -

          Yeah, it would be nice to have built-in support for this feature. For now, I'm doing it by manually calling `git` commands in a `sh` step, the following example partially clones the repo, checks out SOME_BRANCH, merges master into it, and pushes:
           

          mkdir -p repo
          cd repo
          git init -b master
          git remote add origin ${REMOTE_URL}
          # Use git's [https://git-scm.com/docs/partial-clone](Partial Clone) feature to save a lot of time
          git fetch --filter=tree:0 origin master ${SOME_BRANCH}
          git checkout --track origin/${SOME_BRANCH}
          git merge origin/master
          git push
          # ... do some more stuff with the branch

          Gene added a comment - Yeah, it would be nice to have built-in support for this feature. For now, I'm doing it by manually calling `git` commands in a `sh` step, the following example partially clones the repo, checks out SOME_BRANCH, merges master into it, and pushes:   mkdir -p repo cd repo git init -b master git remote add origin ${REMOTE_URL} # Use git's [https: //git-scm.com/docs/partial-clone](Partial Clone) feature to save a lot of time git fetch --filter=tree:0 origin master ${SOME_BRANCH} git checkout --track origin/${SOME_BRANCH} git merge origin/master git push # ... do some more stuff with the branch

          Joshua added a comment -

          Please consider adding support for treeless clones as well. GitHub has a great post about this at:

          https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/

          Joshua added a comment - Please consider adding support for treeless clones as well. GitHub has a great post about this at: https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/

          markewaite Would you be open to make the filter parameter configurable via an option?

          Our pipelines would really benefit from blobless cloning.

          Michael Kriese added a comment - markewaite Would you be open to make the filter parameter configurable via an option? Our pipelines would really benefit from blobless cloning.

          Gene added a comment - - edited

          If you look around, there are higher priority issues that stick around for years without a resolution. My advice: either learn Jenkins development and submit a PR implementing what you want, or find alternative options.

          In my case, I made a Jenkins Shared Library with the stuff I commonly use. One of them is a script that includes git cloning, with various options that I needed. I uploaded it here: https://gist.github.com/gene-pavlovsky/dcb93598f46ae0b512dff7f5c0088f32

          Example usage:            

          gitScript.checkoutRepo(remoteUrl: gitUrl, targetDir: repoDir, branches: [ 'master', branch ], cloneMode: 'treeless')

          Gene added a comment - - edited If you look around, there are higher priority issues that stick around for years without a resolution. My advice: either learn Jenkins development and submit a PR implementing what you want, or find alternative options. In my case, I made a Jenkins Shared Library with the stuff I commonly use. One of them is a script that includes git cloning, with various options that I needed. I uploaded it here: https://gist.github.com/gene-pavlovsky/dcb93598f46ae0b512dff7f5c0088f32 Example usage:             gitScript.checkoutRepo(remoteUrl: gitUrl, targetDir: repoDir, branches: [ 'master', branch ], cloneMode: 'treeless')

          Support for treeless checkout would help us as well with our very large repositories. In our case is clone not relevant because the jobs are running in pre-built docker images containing a clone of the repository. Our problem is that checkout is not using the credentials (https://github.com/jenkinsci/git-client-plugin/blob/3052db7fd4ce0cfb513ad486745779ffa5d39185/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L2751).

          Of course, we can perform git operations with own scripts what means reproducing the checkout logic of checkout scm (for instance for PRs). A native support would be far cleaner.

          markewaite is your 2021 reject of treeless still valid? (https://issues.jenkins.io/browse/JENKINS-64844?focusedId=404578&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-404578)
          We would (try to) contribute support for it, if such a functionality would be welcome for the plugin.

           

          Marc Guillemot added a comment - Support for treeless checkout would help us as well with our very large repositories. In our case is clone not relevant because the jobs are running in pre-built docker images containing a clone of the repository. Our problem is that checkout is not using the credentials ( https://github.com/jenkinsci/git-client-plugin/blob/3052db7fd4ce0cfb513ad486745779ffa5d39185/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L2751). Of course, we can perform git operations with own scripts what means reproducing the checkout logic of checkout scm (for instance for PRs). A native support would be far cleaner. markewaite is your 2021 reject of treeless still valid? ( https://issues.jenkins.io/browse/JENKINS-64844?focusedId=404578&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-404578) We would (try to) contribute support for it, if such a functionality would be welcome for the plugin.  

          Mark Waite added a comment - - edited

          My earlier objection to the --filter argument continues because (revised quote based on current status)

          The --filter argument seems to only have been added to git 2.27 and later. The git plugin needs to support command line git versions on supported Linux operating systems. Command line git 2.27 is newer than the command line git provided with two of the Linux operating systems supported by the Jenkins project.

          The current supported operating systems that deliver a CLI git version older than git 2.27:

          If the feature is added, it needs to be safeguarded against failing on older versions of command line git that do not support the --filter argument.

          If the feature is added, it needs to include automated tests that confirm it works as expected so that I can run those tests in my wide range of test configurations.

          I think that you could add credentials to checkout without compatibility concerns, though I don't plan to add credentials to checkout myself.

          Mark Waite added a comment - - edited My earlier objection to the --filter argument continues because (revised quote based on current status) The --filter argument seems to only have been added to git 2.27 and later. The git plugin needs to support command line git versions on supported Linux operating systems. Command line git 2.27 is newer than the command line git provided with two of the Linux operating systems supported by the Jenkins project. The current supported operating systems that deliver a CLI git version older than git 2.27: Debian 10 - end of public support 30 Jun 2024 Ubuntu 20 - end of public support 2 Apr 2025 If the feature is added, it needs to be safeguarded against failing on older versions of command line git that do not support the --filter argument. If the feature is added, it needs to include automated tests that confirm it works as expected so that I can run those tests in my wide range of test configurations. I think that you could add credentials to checkout without compatibility concerns, though I don't plan to add credentials to checkout myself.

            Unassigned Unassigned
            viceice Michael Kriese
            Votes:
            5 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: