-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Windows Server 2008 Master, Jenkins ver. 1.447.2; Linux Jenkins 1.597, Git plugin 2.2.12.
-
Powered by SuggestiMate
When a repository is configured to fetch a specific refspec from the upstream repository, all tags are fetched.
The documentation specifically states that when a refspec is specified, only that refspec will be fetched.
NOTE: This is not a duplicate of JENKINS-6124. That issue is requesting this to be optional for all cases. This issue is requesting only that the behavior in this specific scenario behave as documented. (i.e. fix the bug)
[JENKINS-14572] Git plug-in fetches all tags even when refspec is provided
git repository isn't designed to hold large binary files, you're abusing git in this scenario.
Git is handling our use case better than any other tool currently available to us, and it is doing so flawlessly (wish I could say the same for this plug-in).
FWIW, I didn't say 'large binary files'. I said 'large repositories of binary files'. Individual file size was not specified. Your assumption is incorrect.
Regardless, we have a work around for this. We are using Git directly via scripting rather than through this plug-in which in many cases is easier and more efficient anyway.
I would like to reopen the issue.
The documentation says:
When do you want to modify this value? A good example is when you want to just retrieve one branch. For example, +refs/heads/master:refs/remotes/origin/master would only retrieve the master branch and nothing else.
But when I set the refspecs to +refs/heads/master:refs/remotes/origin/master the repo in the workspace contains other references (branches) as well.
As I try to learn how to use this plugin, I find this confusing. Please, either change the documentation, or the behavior.
This is on linux, Jenkins 1.597, Git plugin 2.2.12.
I'm seeing this behavior on Jenkins 1.598, Git-Plugin 2.3.5. It appears that git-plugin first fetches all branches with depth=1 (~500MB in our case), and then it fetches the specific change it needs (~20MB). I've tested locally and the second fetch is the only one needed for a build.
> git init /path/to/.jenkins/workspace/Example-Project-Job
Fetching upstream changes from ssh://user@example.com/Project.git
> git --version
using GIT_SSH to set credentials The Credentials for the user system user
> git fetch --tags --progress ssh://user@example.com/Project.git +refs/heads/*:refs/remotes/origin/* --depth=1
> git config remote.origin.url ssh://user@example.com/Project.git
> git config remote.origin.fetch +refs/heads/*:refs/remotes/origin/*
> git config remote.origin.url ssh://user@example.com/Project.git
Fetching upstream changes from ssh://user@example.com/Project.git
using GIT_SSH to set credentials The Credentials for the user system user
> git fetch --tags --progress ssh://user@example.com/Project.git refs/changes/93/3393/2
> git rev-parse 12345...123^
Checking out Revision 12345...123 (mainline)
> git config core.sparsecheckout
> git checkout -f 12345...123
Is there any temp workaround ? Fetching all the tags in our case is just a waist of time.
00:01:28.364 > C:\Program Files\Git\cmd\git.exe -c core.askpass=true fetch --tags --progress git@gitlab.dev.local:superman/b.git +refs/heads/:refs/remotes/origin/ --depth=1
00:07:05.694 > C:\Program Files\Git\cmd\git.exe config remote.origin.url git@gitlab.dev.local:superman/b.git # timeout=10
As you can see It takes about 7 min to execute the next cmd. This is just a waist of time :-/
Please share with me any workaround
We've been able to reduce cloning time by using a reference repo. We first created a bare clone in a shared space on each build slave, then use this as a reference repo in the build job. Now it clones from the reference, but it's a much faster FS operation rather than network. It's still not ideal, since we don't really need the repo to consume disk (or duplicated into the builds), but it's cut out ~2 minutes of the build job.
In the project configuration, navigate to "Source code management" -> "Additional Behaviours" -> "Advanced Clone Behaviors". Check "Shallow Clone" and then the absolute path to the local cloned repo.
EDIT: If you're not already using "Advanced Clone Behaviours", be sure to add it from the "Add" button.
@Robert More: Thank you for your replay . Well Shallow Clone has been active since day 1 in our jobs and still 7 min. Guys can you please allow us to customize the git parameters in the coming versions. In first place I do not need or want to fetch any tags :-/
The plugin already has significant challenges the plugin maintaining compatibility among its many different use cases. I'd rather not provide even more opportunities for user surprises by allowing general purpose customization of the parameters passed to git. If we allow you to pass a custom parameter to a specific git command, it seems like that will then guarantee that the plugin must use that exact command and honor that exact parameter indefinitely.
As an example, previously the plugin used "git clone" to populate the repository. When credentials support was added, that was switched to "git fetch". In the future, it may switch back to "git clone". Allowing you to pass arbitrary parameters to a subset of git commands used by the plugin seems like it will either make compatibility even more difficult, or make future changes even more difficult.
Have you considered investigating the area of the plugin where the --tags argument is passed, seeking a way to remove that call to --tags without harming the rest of the plugin?
In our case, we just want a flag to disable the "initial" fetch of all tags. With Gerrit plugin, we're fetching a unique refspec afterward, so there's no need for us to do more than "git init". Would it be possible to have the option of disabling the initial fetch/clone? We don't need anything more complex than that.
@MArk Waite What do you mean by "Investigating the area of plugin where the --tag argument is passed?". Isn't such behavior hardcoded in the plugin? If not how shall I disabled it?
zeusminos, I mean that you can fork your own copy of the git client plugin source code and the git plugin source code on GitHub, compile them, locate the places in the source code where the --tag argument is passed to command line git, then you can experiment with modifying the behavior to better match what you want.
That will allow you to modify and test your changes. If you find the changes are useful to you and don't break other use cases, you could provide unit tests which confirm the new behavior you've added, and can submit them as a pull request for possible inclusion in the plugin.
@Mark Waite I will really love to do it, but unfortunately my days are way too short to do that. This issue is not impacting just me. I am sure that there are other folks around here with better understanding of this plugin and they can do it in a blink of an eye. Worst case I will say bye bye to this plugin and I will handle that like in the old days "Executed Windows Batch cmd"
We have a time and disk space problem with fetching all tags, too.
It seems that git client plugin is already flexible enough to avoid fetching tags by setting the right option. I've added a check box "Do not fetch tags" to the advanced clone behaviors of the git plugin and made a pull request. This way a user can explicitly override the default behavior.
Code changed in jenkins
User: rhaendel
Path:
src/main/java/hudson/plugins/git/extensions/impl/CloneOption.java
src/main/resources/hudson/plugins/git/extensions/impl/CloneOption/config.groovy
src/main/resources/hudson/plugins/git/extensions/impl/CloneOption/help-noTags.html
src/test/java/hudson/plugins/git/TestGitRepo.java
src/test/java/hudson/plugins/git/extensions/impl/CloneOptionNoTagsTest.java
http://jenkins-ci.org/commit/git-plugin/bfeda3e661531bd9baec937ec4a4944ac482692c
Log:
JENKINS-14572 Add "Do not fetch tags" advanced clone option
Code changed in jenkins
User: rhaendel
Path:
src/main/java/hudson/plugins/git/extensions/impl/CloneOption.java
src/test/java/hudson/plugins/git/extensions/impl/CloneOptionShallowDefaultTagsTest.java
http://jenkins-ci.org/commit/git-plugin/24fb7f3c441057ad09fc3ce6bbedffff797e92bc
Log:
JENKINS-14572 Add a test case for clone option's default 'fetch tags' behaviour
Even with shallow clones "fetch all tags" is the default. Just added a
missing test case and found a true/false bug in my previous commit which
toggled the default to the not backwards compatible value.
Code changed in jenkins
User: Nicolas De loof
Path:
src/main/java/hudson/plugins/git/extensions/impl/CloneOption.java
src/main/resources/hudson/plugins/git/extensions/impl/CloneOption/config.groovy
src/main/resources/hudson/plugins/git/extensions/impl/CloneOption/help-noTags.html
src/test/java/hudson/plugins/git/TestGitRepo.java
src/test/java/hudson/plugins/git/extensions/impl/CloneOptionNoTagsTest.java
src/test/java/hudson/plugins/git/extensions/impl/CloneOptionShallowDefaultTagsTest.java
http://jenkins-ci.org/commit/git-plugin/5c7381cf59d58fe7e0b19a0012fa96144de66b46
Log:
Merge pull request #333 from rhaendel/master
JENKINS-14572 Add "Do not fetch tags" advanced clone option
Compare: https://github.com/jenkinsci/git-plugin/compare/6773c070cfff...5c7381cf59d5
Since there has been no git-plugin release (not yet) after my pull request was merged, our workaround was to checkout the sources, build and deploy the plugin manually.
Another possibility is scripting all git operations like dt25954 mentioned on 23/Oct/13 3:18 PM. Simply use "Execute shell" build steps for that. But this has one big drawback: This way, Jenkins cannot track the commits that are built and not send dedicated e-mails to the committers.
Alternatives you can consider:
- Install a pre-release build of the git plugin which includes the change
- Use a reference repository in your job definition to reduce the data transfer and disc space use
Thanks Mark, pre-release plugin worked for me and now plugin is not fetching tags.
One more observation is "Shallow clone" seems not working always, I am able to point this by looking at console output of my project.
Yet times I see "--depth 1" with command and not always.
Is there a way to fix this issue?
mmlk if there is an issue with shallow clone not consistently remaining a shallow clone, please provide a separate bug report which shows the problem through a series of steps that others can duplicate. Console output is helpful, but is not sufficient evidence that there is a problem.
That issue is still there.
It does work whith branches but not with tags.
With branches I had to configure "Do not fetch tags" to stop fetching all tags.
But when I do the same for '+refs/tags/ci-uat' it does not find any release for the build as no any tags fetched.
If I remove ""Do not fetch tags"" it fetches all tags.
Refspec: +refs/tags/ci-uat
Honor refspec on initial clone: enabled
Jenkins: 2.27
Git plugin: 3.0.0
Git client plugin: 2.0.0
Git plugin does not respect 'Honor refspec on initial clone' in case if refs pointed to tags.
All tags are fetched anyway.
Copied the following comment from
JENKINS-6124as it is relevant here as well....For a standard source code repository, this is not significant performance wise and we have not noticed this issue on our source code repositories.
However, in general, it is not good that it is fetching references and therefore objects that are not required by the build. It is just inefficient. However, as stated, for source code repositories it is not significant enough that we have noticed.
I have created a new issue that targets the scenario for our specific issue JENKINS-14572.
Where we have noticed it is in the Git repositories we are using to deploy our web applications which are large repositories of binary files.
We create each deployment image as an orphan commit, tag it, and push the tag not the branch back to our Git server. This way we can delete the tags after a holding period causing those commits to be unreferenced and eligible for garbage collection.
We have the Jenkins plug-in configured to pull down an 'empty' branch by specifying a refspec. When all of the tags are fetched, then a large number of unneeded objects are pulled down from the repository greatly increasing the size of the repositories on the Jenkins slaves.
Having said all of that, if you want to build a specific tag, why not specify it in the refspec? The refspec can use build parameters also. Why fetch all tags when you only need one?