Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-14572

Git plug-in fetches all tags even when refspec is provided

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • git-plugin
    • None
    • Windows Server 2008 Master, Jenkins ver. 1.447.2; Linux Jenkins 1.597, Git plugin 2.2.12.

      When a repository is configured to fetch a specific refspec from the upstream repository, all tags are fetched.

      The documentation specifically states that when a refspec is specified, only that refspec will be fetched.

      NOTE: This is not a duplicate of JENKINS-6124. That issue is requesting this to be optional for all cases. This issue is requesting only that the behavior in this specific scenario behave as documented. (i.e. fix the bug)

          [JENKINS-14572] Git plug-in fetches all tags even when refspec is provided

          Copied the following comment from JENKINS-6124 as it is relevant here as well....

          For a standard source code repository, this is not significant performance wise and we have not noticed this issue on our source code repositories.

          However, in general, it is not good that it is fetching references and therefore objects that are not required by the build. It is just inefficient. However, as stated, for source code repositories it is not significant enough that we have noticed.

          I have created a new issue that targets the scenario for our specific issue JENKINS-14572.

          Where we have noticed it is in the Git repositories we are using to deploy our web applications which are large repositories of binary files.

          We create each deployment image as an orphan commit, tag it, and push the tag not the branch back to our Git server. This way we can delete the tags after a holding period causing those commits to be unreferenced and eligible for garbage collection.

          We have the Jenkins plug-in configured to pull down an 'empty' branch by specifying a refspec. When all of the tags are fetched, then a large number of unneeded objects are pulled down from the repository greatly increasing the size of the repositories on the Jenkins slaves.

          Having said all of that, if you want to build a specific tag, why not specify it in the refspec? The refspec can use build parameters also. Why fetch all tags when you only need one?

          Douglas Beatty added a comment - Copied the following comment from JENKINS-6124 as it is relevant here as well.... For a standard source code repository, this is not significant performance wise and we have not noticed this issue on our source code repositories. However, in general, it is not good that it is fetching references and therefore objects that are not required by the build. It is just inefficient. However, as stated, for source code repositories it is not significant enough that we have noticed. I have created a new issue that targets the scenario for our specific issue JENKINS-14572 . Where we have noticed it is in the Git repositories we are using to deploy our web applications which are large repositories of binary files. We create each deployment image as an orphan commit, tag it, and push the tag not the branch back to our Git server. This way we can delete the tags after a holding period causing those commits to be unreferenced and eligible for garbage collection. We have the Jenkins plug-in configured to pull down an 'empty' branch by specifying a refspec. When all of the tags are fetched, then a large number of unneeded objects are pulled down from the repository greatly increasing the size of the repositories on the Jenkins slaves. Having said all of that, if you want to build a specific tag, why not specify it in the refspec? The refspec can use build parameters also. Why fetch all tags when you only need one?

          git repository isn't designed to hold large binary files, you're abusing git in this scenario.

          Nicolas De Loof added a comment - git repository isn't designed to hold large binary files, you're abusing git in this scenario.

          Git is handling our use case better than any other tool currently available to us, and it is doing so flawlessly (wish I could say the same for this plug-in).

          FWIW, I didn't say 'large binary files'. I said 'large repositories of binary files'. Individual file size was not specified. Your assumption is incorrect.

          Regardless, we have a work around for this. We are using Git directly via scripting rather than through this plug-in which in many cases is easier and more efficient anyway.

          Douglas Beatty added a comment - Git is handling our use case better than any other tool currently available to us, and it is doing so flawlessly (wish I could say the same for this plug-in). FWIW, I didn't say 'large binary files'. I said 'large repositories of binary files'. Individual file size was not specified. Your assumption is incorrect. Regardless, we have a work around for this. We are using Git directly via scripting rather than through this plug-in which in many cases is easier and more efficient anyway.

          I would like to reopen the issue.

          The documentation says:

          When do you want to modify this value? A good example is when you want to just retrieve one branch. For example, +refs/heads/master:refs/remotes/origin/master would only retrieve the master branch and nothing else.

          But when I set the refspecs to +refs/heads/master:refs/remotes/origin/master the repo in the workspace contains other references (branches) as well.

          As I try to learn how to use this plugin, I find this confusing. Please, either change the documentation, or the behavior.

          This is on linux, Jenkins 1.597, Git plugin 2.2.12.

          Martin d'Anjou added a comment - I would like to reopen the issue. The documentation says: When do you want to modify this value? A good example is when you want to just retrieve one branch. For example, +refs/heads/master:refs/remotes/origin/master would only retrieve the master branch and nothing else. But when I set the refspecs to +refs/heads/master:refs/remotes/origin/master the repo in the workspace contains other references (branches) as well. As I try to learn how to use this plugin, I find this confusing. Please, either change the documentation, or the behavior. This is on linux, Jenkins 1.597, Git plugin 2.2.12.

          Robert Moore added a comment -

          I'm seeing this behavior on Jenkins 1.598, Git-Plugin 2.3.5. It appears that git-plugin first fetches all branches with depth=1 (~500MB in our case), and then it fetches the specific change it needs (~20MB). I've tested locally and the second fetch is the only one needed for a build.

          > git init /path/to/.jenkins/workspace/Example-Project-Job
          Fetching upstream changes from ssh://user@example.com/Project.git
          > git --version
          using GIT_SSH to set credentials The Credentials for the user system user
          > git fetch --tags --progress ssh://user@example.com/Project.git +refs/heads/*:refs/remotes/origin/* --depth=1
          > git config remote.origin.url ssh://user@example.com/Project.git
          > git config remote.origin.fetch +refs/heads/*:refs/remotes/origin/*
          > git config remote.origin.url ssh://user@example.com/Project.git
          Fetching upstream changes from ssh://user@example.com/Project.git
          using GIT_SSH to set credentials The Credentials for the user system user
          > git fetch --tags --progress ssh://user@example.com/Project.git refs/changes/93/3393/2
          > git rev-parse 12345...123^

          {commit}

          Checking out Revision 12345...123 (mainline)
          > git config core.sparsecheckout
          > git checkout -f 12345...123

          Robert Moore added a comment - I'm seeing this behavior on Jenkins 1.598, Git-Plugin 2.3.5. It appears that git-plugin first fetches all branches with depth=1 (~500MB in our case), and then it fetches the specific change it needs (~20MB). I've tested locally and the second fetch is the only one needed for a build. > git init /path/to/.jenkins/workspace/Example-Project-Job Fetching upstream changes from ssh://user@example.com/Project.git > git --version using GIT_SSH to set credentials The Credentials for the user system user > git fetch --tags --progress ssh://user@example.com/Project.git +refs/heads/*:refs/remotes/origin/* --depth=1 > git config remote.origin.url ssh://user@example.com/Project.git > git config remote.origin.fetch +refs/heads/*:refs/remotes/origin/* > git config remote.origin.url ssh://user@example.com/Project.git Fetching upstream changes from ssh://user@example.com/Project.git using GIT_SSH to set credentials The Credentials for the user system user > git fetch --tags --progress ssh://user@example.com/Project.git refs/changes/93/3393/2 > git rev-parse 12345...123^ {commit} Checking out Revision 12345...123 (mainline) > git config core.sparsecheckout > git checkout -f 12345...123

          Zeus Minos added a comment -

          Is there any temp workaround ? Fetching all the tags in our case is just a waist of time.
          00:01:28.364 > C:\Program Files\Git\cmd\git.exe -c core.askpass=true fetch --tags --progress git@gitlab.dev.local:superman/b.git +refs/heads/:refs/remotes/origin/ --depth=1
          00:07:05.694 > C:\Program Files\Git\cmd\git.exe config remote.origin.url git@gitlab.dev.local:superman/b.git # timeout=10

          As you can see It takes about 7 min to execute the next cmd. This is just a waist of time :-/

          Please share with me any workaround

          Zeus Minos added a comment - Is there any temp workaround ? Fetching all the tags in our case is just a waist of time. 00:01:28.364 > C:\Program Files\Git\cmd\git.exe -c core.askpass=true fetch --tags --progress git@gitlab.dev.local:superman/b.git +refs/heads/ :refs/remotes/origin/ --depth=1 00:07:05.694 > C:\Program Files\Git\cmd\git.exe config remote.origin.url git@gitlab.dev.local:superman/b.git # timeout=10 As you can see It takes about 7 min to execute the next cmd. This is just a waist of time :-/ Please share with me any workaround

          Robert Moore added a comment - - edited

          We've been able to reduce cloning time by using a reference repo. We first created a bare clone in a shared space on each build slave, then use this as a reference repo in the build job. Now it clones from the reference, but it's a much faster FS operation rather than network. It's still not ideal, since we don't really need the repo to consume disk (or duplicated into the builds), but it's cut out ~2 minutes of the build job.

          In the project configuration, navigate to "Source code management" -> "Additional Behaviours" -> "Advanced Clone Behaviors". Check "Shallow Clone" and then the absolute path to the local cloned repo.

          EDIT: If you're not already using "Advanced Clone Behaviours", be sure to add it from the "Add" button.

          Robert Moore added a comment - - edited We've been able to reduce cloning time by using a reference repo. We first created a bare clone in a shared space on each build slave, then use this as a reference repo in the build job. Now it clones from the reference, but it's a much faster FS operation rather than network. It's still not ideal, since we don't really need the repo to consume disk (or duplicated into the builds), but it's cut out ~2 minutes of the build job. In the project configuration, navigate to "Source code management" -> "Additional Behaviours" -> "Advanced Clone Behaviors". Check "Shallow Clone" and then the absolute path to the local cloned repo. EDIT: If you're not already using "Advanced Clone Behaviours", be sure to add it from the "Add" button.

          Zeus Minos added a comment - - edited

          @Robert More: Thank you for your replay . Well Shallow Clone has been active since day 1 in our jobs and still 7 min. Guys can you please allow us to customize the git parameters in the coming versions. In first place I do not need or want to fetch any tags :-/

          Zeus Minos added a comment - - edited @Robert More: Thank you for your replay . Well Shallow Clone has been active since day 1 in our jobs and still 7 min. Guys can you please allow us to customize the git parameters in the coming versions. In first place I do not need or want to fetch any tags :-/

          Mark Waite added a comment -

          The plugin already has significant challenges the plugin maintaining compatibility among its many different use cases. I'd rather not provide even more opportunities for user surprises by allowing general purpose customization of the parameters passed to git. If we allow you to pass a custom parameter to a specific git command, it seems like that will then guarantee that the plugin must use that exact command and honor that exact parameter indefinitely.

          As an example, previously the plugin used "git clone" to populate the repository. When credentials support was added, that was switched to "git fetch". In the future, it may switch back to "git clone". Allowing you to pass arbitrary parameters to a subset of git commands used by the plugin seems like it will either make compatibility even more difficult, or make future changes even more difficult.

          Have you considered investigating the area of the plugin where the --tags argument is passed, seeking a way to remove that call to --tags without harming the rest of the plugin?

          Mark Waite added a comment - The plugin already has significant challenges the plugin maintaining compatibility among its many different use cases. I'd rather not provide even more opportunities for user surprises by allowing general purpose customization of the parameters passed to git. If we allow you to pass a custom parameter to a specific git command, it seems like that will then guarantee that the plugin must use that exact command and honor that exact parameter indefinitely. As an example, previously the plugin used "git clone" to populate the repository. When credentials support was added, that was switched to "git fetch". In the future, it may switch back to "git clone". Allowing you to pass arbitrary parameters to a subset of git commands used by the plugin seems like it will either make compatibility even more difficult, or make future changes even more difficult. Have you considered investigating the area of the plugin where the --tags argument is passed, seeking a way to remove that call to --tags without harming the rest of the plugin?

          Robert Moore added a comment -

          In our case, we just want a flag to disable the "initial" fetch of all tags. With Gerrit plugin, we're fetching a unique refspec afterward, so there's no need for us to do more than "git init". Would it be possible to have the option of disabling the initial fetch/clone? We don't need anything more complex than that.

          Robert Moore added a comment - In our case, we just want a flag to disable the "initial" fetch of all tags. With Gerrit plugin, we're fetching a unique refspec afterward, so there's no need for us to do more than "git init". Would it be possible to have the option of disabling the initial fetch/clone? We don't need anything more complex than that.

          Zeus Minos added a comment -

          @MArk Waite What do you mean by "Investigating the area of plugin where the --tag argument is passed?". Isn't such behavior hardcoded in the plugin? If not how shall I disabled it?

          Zeus Minos added a comment - @MArk Waite What do you mean by "Investigating the area of plugin where the --tag argument is passed?". Isn't such behavior hardcoded in the plugin? If not how shall I disabled it?

          Mark Waite added a comment -

          zeusminos, I mean that you can fork your own copy of the git client plugin source code and the git plugin source code on GitHub, compile them, locate the places in the source code where the --tag argument is passed to command line git, then you can experiment with modifying the behavior to better match what you want.

          That will allow you to modify and test your changes. If you find the changes are useful to you and don't break other use cases, you could provide unit tests which confirm the new behavior you've added, and can submit them as a pull request for possible inclusion in the plugin.

          Mark Waite added a comment - zeusminos , I mean that you can fork your own copy of the git client plugin source code and the git plugin source code on GitHub, compile them, locate the places in the source code where the --tag argument is passed to command line git, then you can experiment with modifying the behavior to better match what you want. That will allow you to modify and test your changes. If you find the changes are useful to you and don't break other use cases, you could provide unit tests which confirm the new behavior you've added, and can submit them as a pull request for possible inclusion in the plugin.

          Zeus Minos added a comment - - edited

          @Mark Waite I will really love to do it, but unfortunately my days are way too short to do that. This issue is not impacting just me. I am sure that there are other folks around here with better understanding of this plugin and they can do it in a blink of an eye. Worst case I will say bye bye to this plugin and I will handle that like in the old days "Executed Windows Batch cmd"

          Zeus Minos added a comment - - edited @Mark Waite I will really love to do it, but unfortunately my days are way too short to do that. This issue is not impacting just me . I am sure that there are other folks around here with better understanding of this plugin and they can do it in a blink of an eye. Worst case I will say bye bye to this plugin and I will handle that like in the old days "Executed Windows Batch cmd"

          Mark Waite added a comment -

          That's a fair approach as well.

          Mark Waite added a comment - That's a fair approach as well.

          Ronny Händel added a comment -

          We have a time and disk space problem with fetching all tags, too.

          It seems that git client plugin is already flexible enough to avoid fetching tags by setting the right option. I've added a check box "Do not fetch tags" to the advanced clone behaviors of the git plugin and made a pull request. This way a user can explicitly override the default behavior.

          Ronny Händel added a comment - We have a time and disk space problem with fetching all tags, too. It seems that git client plugin is already flexible enough to avoid fetching tags by setting the right option. I've added a check box "Do not fetch tags" to the advanced clone behaviors of the git plugin and made a pull request . This way a user can explicitly override the default behavior.

          Code changed in jenkins
          User: rhaendel
          Path:
          src/main/java/hudson/plugins/git/extensions/impl/CloneOption.java
          src/main/resources/hudson/plugins/git/extensions/impl/CloneOption/config.groovy
          src/main/resources/hudson/plugins/git/extensions/impl/CloneOption/help-noTags.html
          src/test/java/hudson/plugins/git/TestGitRepo.java
          src/test/java/hudson/plugins/git/extensions/impl/CloneOptionNoTagsTest.java
          http://jenkins-ci.org/commit/git-plugin/bfeda3e661531bd9baec937ec4a4944ac482692c
          Log:
          JENKINS-14572 Add "Do not fetch tags" advanced clone option

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: rhaendel Path: src/main/java/hudson/plugins/git/extensions/impl/CloneOption.java src/main/resources/hudson/plugins/git/extensions/impl/CloneOption/config.groovy src/main/resources/hudson/plugins/git/extensions/impl/CloneOption/help-noTags.html src/test/java/hudson/plugins/git/TestGitRepo.java src/test/java/hudson/plugins/git/extensions/impl/CloneOptionNoTagsTest.java http://jenkins-ci.org/commit/git-plugin/bfeda3e661531bd9baec937ec4a4944ac482692c Log: JENKINS-14572 Add "Do not fetch tags" advanced clone option

          Code changed in jenkins
          User: rhaendel
          Path:
          src/main/java/hudson/plugins/git/extensions/impl/CloneOption.java
          src/test/java/hudson/plugins/git/extensions/impl/CloneOptionShallowDefaultTagsTest.java
          http://jenkins-ci.org/commit/git-plugin/24fb7f3c441057ad09fc3ce6bbedffff797e92bc
          Log:
          JENKINS-14572 Add a test case for clone option's default 'fetch tags' behaviour

          Even with shallow clones "fetch all tags" is the default. Just added a
          missing test case and found a true/false bug in my previous commit which
          toggled the default to the not backwards compatible value.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: rhaendel Path: src/main/java/hudson/plugins/git/extensions/impl/CloneOption.java src/test/java/hudson/plugins/git/extensions/impl/CloneOptionShallowDefaultTagsTest.java http://jenkins-ci.org/commit/git-plugin/24fb7f3c441057ad09fc3ce6bbedffff797e92bc Log: JENKINS-14572 Add a test case for clone option's default 'fetch tags' behaviour Even with shallow clones "fetch all tags" is the default. Just added a missing test case and found a true/false bug in my previous commit which toggled the default to the not backwards compatible value.

          Code changed in jenkins
          User: Nicolas De loof
          Path:
          src/main/java/hudson/plugins/git/extensions/impl/CloneOption.java
          src/main/resources/hudson/plugins/git/extensions/impl/CloneOption/config.groovy
          src/main/resources/hudson/plugins/git/extensions/impl/CloneOption/help-noTags.html
          src/test/java/hudson/plugins/git/TestGitRepo.java
          src/test/java/hudson/plugins/git/extensions/impl/CloneOptionNoTagsTest.java
          src/test/java/hudson/plugins/git/extensions/impl/CloneOptionShallowDefaultTagsTest.java
          http://jenkins-ci.org/commit/git-plugin/5c7381cf59d58fe7e0b19a0012fa96144de66b46
          Log:
          Merge pull request #333 from rhaendel/master

          JENKINS-14572 Add "Do not fetch tags" advanced clone option

          Compare: https://github.com/jenkinsci/git-plugin/compare/6773c070cfff...5c7381cf59d5

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Nicolas De loof Path: src/main/java/hudson/plugins/git/extensions/impl/CloneOption.java src/main/resources/hudson/plugins/git/extensions/impl/CloneOption/config.groovy src/main/resources/hudson/plugins/git/extensions/impl/CloneOption/help-noTags.html src/test/java/hudson/plugins/git/TestGitRepo.java src/test/java/hudson/plugins/git/extensions/impl/CloneOptionNoTagsTest.java src/test/java/hudson/plugins/git/extensions/impl/CloneOptionShallowDefaultTagsTest.java http://jenkins-ci.org/commit/git-plugin/5c7381cf59d58fe7e0b19a0012fa96144de66b46 Log: Merge pull request #333 from rhaendel/master JENKINS-14572 Add "Do not fetch tags" advanced clone option Compare: https://github.com/jenkinsci/git-plugin/compare/6773c070cfff...5c7381cf59d5

          Mallik M added a comment -

          Is there any workaround for this issue?

          Mallik M added a comment - Is there any workaround for this issue?

          Ronny Händel added a comment -

          Since there has been no git-plugin release (not yet) after my pull request was merged, our workaround was to checkout the sources, build and deploy the plugin manually.

          Another possibility is scripting all git operations like dt25954 mentioned on 23/Oct/13 3:18 PM. Simply use "Execute shell" build steps for that. But this has one big drawback: This way, Jenkins cannot track the commits that are built and not send dedicated e-mails to the committers.

          Ronny Händel added a comment - Since there has been no git-plugin release (not yet) after my pull request was merged, our workaround was to checkout the sources , build and deploy the plugin manually. Another possibility is scripting all git operations like dt25954 mentioned on 23/Oct/13 3:18 PM. Simply use "Execute shell" build steps for that. But this has one big drawback: This way, Jenkins cannot track the commits that are built and not send dedicated e-mails to the committers.

          Mark Waite added a comment -

          Alternatives you can consider:

          • Install a pre-release build of the git plugin which includes the change
          • Use a reference repository in your job definition to reduce the data transfer and disc space use

          Mark Waite added a comment - Alternatives you can consider: Install a pre-release build of the git plugin which includes the change Use a reference repository in your job definition to reduce the data transfer and disc space use

          Mallik M added a comment -

          Thanks Mark, pre-release plugin worked for me and now plugin is not fetching tags.

          One more observation is "Shallow clone" seems not working always, I am able to point this by looking at console output of my project.
          Yet times I see "--depth 1" with command and not always.

          Is there a way to fix this issue?

          Mallik M added a comment - Thanks Mark, pre-release plugin worked for me and now plugin is not fetching tags. One more observation is "Shallow clone" seems not working always, I am able to point this by looking at console output of my project. Yet times I see "--depth 1" with command and not always. Is there a way to fix this issue?

          Mark Waite added a comment -

          mmlk if there is an issue with shallow clone not consistently remaining a shallow clone, please provide a separate bug report which shows the problem through a series of steps that others can duplicate. Console output is helpful, but is not sufficient evidence that there is a problem.

          Mark Waite added a comment - mmlk if there is an issue with shallow clone not consistently remaining a shallow clone, please provide a separate bug report which shows the problem through a series of steps that others can duplicate. Console output is helpful, but is not sufficient evidence that there is a problem.

          Mark Waite added a comment -

          Included in git plugin 2.4.1 released 26 May 2015

          Mark Waite added a comment - Included in git plugin 2.4.1 released 26 May 2015

          That issue is still there.
          It does work whith branches but not with tags.
          With branches I had to configure "Do not fetch tags" to stop fetching all tags.

          But when I do the same for '+refs/tags/ci-uat' it does not find any release for the build as no any tags fetched.
          If I remove ""Do not fetch tags"" it fetches all tags.

          Refspec: +refs/tags/ci-uat
          Honor refspec on initial clone: enabled

          Jenkins: 2.27
          Git plugin: 3.0.0
          Git client plugin: 2.0.0

          Alexey Ostrovsky added a comment - That issue is still there. It does work whith branches but not with tags. With branches I had to configure "Do not fetch tags" to stop fetching all tags. But when I do the same for '+refs/tags/ci-uat' it does not find any release for the build as no any tags fetched. If I remove ""Do not fetch tags"" it fetches all tags. Refspec: +refs/tags/ci-uat Honor refspec on initial clone: enabled Jenkins: 2.27 Git plugin: 3.0.0 Git client plugin: 2.0.0

          Git plugin does not respect 'Honor refspec on initial clone' in case if refs pointed to tags.
          All tags are fetched anyway.

          Alexey Ostrovsky added a comment - Git plugin does not respect 'Honor refspec on initial clone' in case if refs pointed to tags. All tags are fetched anyway.

            Unassigned Unassigned
            dt25954 Douglas Beatty
            Votes:
            6 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated: