-
Bug
-
Resolution: Fixed
-
Major
-
Introduced in Git Plugin 2.4.1
History
This was originally discussed in the comments of JENKINS-26587 as it was first identified as a regression when testing that issue. As Git Plugin 2.4.1 has how been release w/o resolving this regression, it has escaped and I am opening a separate ticket.
Description of Problem
Previously in (and correctly?) when using the "Prune stale remote-tracking branches" behavior, in every build, after the git checkout -f but before the git reset --hard there was a git rev-parse --verify HEAD command. Now in place of the single rev-parse command there is a git rev-parse /remotes/origin/... command for every remote branch that exists.
Steps to reproduce.
- Install a baseline Jenkins Master, which will include the Git Plugin v 2.4.1. (We used the 1.625.3 LTS version.)
- Setup credentials so that it can authenticate to a remote repo. (We use Atlasssian Stas.h.
- Create a freestyle job to pull from a remote git repo that has multiple (preferably many) branches.
- branch specifier origin/master
- Additional Behaviors: Prune stale remote-tracking branches.
Run the job and observe the output. It will be something similar to the following:
Building on master in workspace /var/lib/jenkins/workspace/git-test > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url ssh://git@stash:7999/TEST/test.git # timeout=10 Pruning obsolete local branches Fetching upstream changes from ssh://git@stash:7999/TEST/test.git > git --version # timeout=10 using GIT_SSH to set credentials Credential ID: jenkins-stash-ssh > git fetch --tags --progress ssh://git@stash:7999/TEST/test.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision 38b70706154d37132bb67b82af7317e2ed85f9aa (refs/remotes/origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 38b70706154d37132bb67b82af7317e2ed85f9aa First time build. Skipping changelog. > git branch -a # timeout=10 > git rev-parse remotes/origin/CMOBL-4832-create-separate-onarrival-api^{commit} # timeout=10 > git rev-parse remotes/origin/CROSSTAB-5545^{commit} # timeout=10 > git rev-parse remotes/origin/CROSSTAB-5802^{commit} # timeout=10 > git rev-parse remotes/origin/CROSSTAB-5803^{commit} # timeout=10 ... several thousand ref-parses later ... > git rev-parse remotes/origin/webformtomvc-part1^{commit} # timeout=10 > git rev-parse remotes/origin/wip-csn-6200^{commit} # timeout=10 > git rev-parse remotes/origin/wip-prod-20347^{commit} # timeout=10 Finished: SUCCESS
With 17K+ remote branches in our repository, this added fifteen minutes to builds that previously took three to eight minutes total. Obviously we could stand to clean up some old branches but this behavior is not acceptable.
Workarounds
- Remove the prune stale branches behavior.
- Downgrade to Git Plugin 2.4.0
[JENKINS-32258] Excessive Git --Rev-Parse invocation when using Prune
I've submitted a pull request to the git client plugin which I believe resolves this bug report without altering the algorithm that was added to reduce the size of BuildData. Until build expires, you can download git-client.hpi from the CloudBees Jenkins pull request builder.
The pull request may also be worth reviewing, in case you see some problem that I missed. I'm most likely the one who will merge the pull request, but I have requested a review from others in case there is some mistake in the code.
I did a preliminary test of this on a new Jenkins box and the issue appears to be resolved. Has anyone else reviewed your pull request? I would like some more eyes on this before we put it on our production box. Any chance of getting a Git Client Plugin release soon?
There's been no feedback on the pull request either from ndeloof or from the larger request I made to the Cloudbees team and the recently created jenkins reviewers team.
Since the request for code review has been unanswered for 5 days, it may be time for me to merge the change to the master branch and release a new version of the plugin.
I can release a new version of the git client plugin which would only contain that fix, and am confident it will resolve the issue. The pull request describes some of the techniques I used to verify the fix.
Code changed in jenkins
User: Mark Waite
Path:
src/test/java/org/jenkinsci/plugins/gitclient/GitAPITestCase.java
http://jenkins-ci.org/commit/git-client-plugin/3a8d219569652fbe0531f6ffefa441818acaca06
Log:
Test for JENKINS-32258
Code changed in jenkins
User: Mark Waite
Path:
src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
http://jenkins-ci.org/commit/git-client-plugin/1a450a5ee993209cc9f27f2a4bfb31bc08a33257
Log:
[Fix JENKINS-32258] remove excess rev-parse calls
The BuildData cleanup change to remove dead branches from BuildData
needs the list of current branches. Generating the list of current
branches called rev-parse for each branch. With small branch counts,
that isn't a huge problem. Large branch counts make it painfully slow.
One of the problem repositories has 17000 branches.
Since the BuildData cleanup step is introducing extra work, this may
still be slower than earlier versions, since larger sized BuildDData
was accepted prior to git plugin 2.4.1 and git client 1.19.1.
Code changed in jenkins
User: Mark Waite
Path:
src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
src/test/java/org/jenkinsci/plugins/gitclient/GitAPITestCase.java
http://jenkins-ci.org/commit/git-client-plugin/89ab085182b7a4da6ae575fa4e154a3f3bf0add0
Log:
Merge pull request #196 from MarkEWaite/master-JENKINS-32258-many-rev-parse-calls
Fix JENKINS-32258 - many rev-parse calls pruning remote tracking branches
Compare: https://github.com/jenkinsci/git-client-plugin/compare/3139a94ab5b8...89ab085182b7
Fixed in git client plugin 1.19.2. Release to artifactory failed due to credentials error.
I've written a test which detects the failure and used `git bisect` to isolate the break to a BuildData related change from ndeloof.
I hope to investigate alternatives for a fix soon, or you can use that test and the commit information to experiment with possible fixes.