Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45447

Git checkout with many tags is much slower with 3.3.0 release if pre-build merge is used

      We have 62 thousands of tags in our git repo.
      After upgrade of git plugin from 3.1.0 to 3.3.0 checkout of the repo with PreBuildMerge option takes about an hour.

      14:54:44  > git rev-parse HEAD^{commit} # timeout=10
      14:54:45 Seen branch in repository origin/10098_update_header_link_and_my_account_menu
      ...
      ...
      14:54:45 Seen 1,592 remote branches
      14:54:45  > git tag -l # timeout=10
      14:54:46  > git rev-parse refs/tags/test01wxunit/20160617_094816^{commit} # timeout=10
      ...
      ...
      15:49:57  > git rev-parse refs/tags/b2bAATpipeline/tag/2015-03-06_12-08-07^{commit} # timeout=10
      15:49:57 Checking out Revision ab591971099f42107cbd7915120ee1381a206047 (origin/hybris551aws)
      

      It looks like the root cause of the issue is commit 31fedce9c41c9006c886835e03c9fe825d49aba6.

          [JENKINS-45447] Git checkout with many tags is much slower with 3.3.0 release if pre-build merge is used

          Mark Waite added a comment - - edited

          splatteredbits and yehorov are you able to test the git client plugin and git plugin pull requests that I have created? 

          They are working as expected for me.  I'd like someone directly impacted by the problem to confirm it is resolved in their case, in addition to my verification.

          Mark Waite added a comment - - edited splatteredbits  and yehorov are you able to test the git client plugin and git plugin pull requests that I have created?  They are working as expected for me.  I'd like someone directly impacted by the problem to confirm it is resolved in their case, in addition to my verification.

          Aaron Jensen added a comment -

          markewaite I'm hoping to get this installed and tested sometime this week.

          Aaron Jensen added a comment - markewaite I'm hoping to get this installed and tested sometime this week.

          I can verify this fixed the issue for us. We actually had edited the git plugin to remove that commit. Glad it can be fixed in the main branch. 

          Samuel Gabriel added a comment - I can verify this fixed the issue for us. We actually had edited the git plugin to remove that commit. Glad it can be fixed in the main branch. 

          Mark Waite added a comment -

          sgabriel thanks very much!  I'm still looking forward to verification from at least one other user, and to a code review.  If code reviewers are too busy, I'll likely merge it early next week for inclusion in a release.  We have some other interesting fixes that seem to be approaching "done" that would be worth releasing a new plugin version.

          Mark Waite added a comment - sgabriel thanks very much!  I'm still looking forward to verification from at least one other user, and to a code review.  If code reviewers are too busy, I'll likely merge it early next week for inclusion in a release.  We have some other interesting fixes that seem to be approaching "done" that would be worth releasing a new plugin version.

          Filipe Roque added a comment -

          I have tested this and everything seems OK

          Filipe Roque added a comment - I have tested this and everything seems OK

          I have tested pre-release plugins.
          Number of git rev-parse calls decreased from 71665 to 5.

          Mykhaylo Yehorov added a comment - I have tested pre-release plugins. Number of git rev-parse calls decreased from 71665 to 5.

          Aaron Jensen added a comment -

          I have tested the pre-release plugins. It fixes the issue for us. (Dozens of builds polling every minute.)

          Aaron Jensen added a comment - I have tested the pre-release plugins. It fixes the issue for us. (Dozens of builds polling every minute.)

          Code changed in jenkins
          User: Mark Waite
          Path:
          pom.xml
          src/main/java/hudson/plugins/git/Branch.java
          src/main/java/hudson/plugins/git/GitObject.java
          src/main/java/hudson/plugins/git/Tag.java
          src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
          src/main/java/org/jenkinsci/plugins/gitclient/GitClient.java
          src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java
          src/main/java/org/jenkinsci/plugins/gitclient/RemoteGitImpl.java
          src/test/java/hudson/plugins/git/BranchTest.java
          src/test/java/hudson/plugins/git/GitObjectTest.java
          src/test/java/hudson/plugins/git/TagTest.java
          src/test/java/org/jenkinsci/plugins/gitclient/GitClientTest.java
          http://jenkins-ci.org/commit/git-client-plugin/4dfc9368a7ce65bbb91294c22a0cd439fa7f77ea
          Log:
          Merge pull request #288 from MarkEWaite/add-getTags

          JENKINS-45447 Add GitClient.getTags()

          Compare: https://github.com/jenkinsci/git-client-plugin/compare/5526a61878c1...4dfc9368a7ce

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: pom.xml src/main/java/hudson/plugins/git/Branch.java src/main/java/hudson/plugins/git/GitObject.java src/main/java/hudson/plugins/git/Tag.java src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java src/main/java/org/jenkinsci/plugins/gitclient/GitClient.java src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java src/main/java/org/jenkinsci/plugins/gitclient/RemoteGitImpl.java src/test/java/hudson/plugins/git/BranchTest.java src/test/java/hudson/plugins/git/GitObjectTest.java src/test/java/hudson/plugins/git/TagTest.java src/test/java/org/jenkinsci/plugins/gitclient/GitClientTest.java http://jenkins-ci.org/commit/git-client-plugin/4dfc9368a7ce65bbb91294c22a0cd439fa7f77ea Log: Merge pull request #288 from MarkEWaite/add-getTags JENKINS-45447 Add GitClient.getTags() Compare: https://github.com/jenkinsci/git-client-plugin/compare/5526a61878c1...4dfc9368a7ce

          Code changed in jenkins
          User: Mark Waite
          Path:
          src/main/java/hudson/plugins/git/util/GitUtils.java
          http://jenkins-ci.org/commit/git-plugin/18bc3dc8ab0fa590725f09cd31443e0aee298bbf
          Log:
          JENKINS-45447 Improve poll and checkout performance for repos with many tags

          JENKINS-45447 reports that checkout of a repository with many tags
          from a freestyle job using a wildcard in the branch name is
          dramatically slower than earlier releases. Many other cases which use
          the advanced branch selection mechanism show the same performance
          problem.

          31fedce9c added tag checks to the evaluation loop for branch
          names. Unfortunately, tag evaluation needs both the tag name and the
          SHA1 of the commit identified by the tag. The original; implementation
          called revParse() to compute that SHA1 for each tag. With many tags in
          the repository (bug report example was 60,000 tags), the checkout time
          increased dramatically.

          I found that even the number of tags in the git plugin repository
          could add as much as 5 seconds for the computation of SHA1 hashes.
          Those computed SHA1 hashes were then immediately discarded because
          they did not satisfy the branch name selection criteria.

          Calling GitClient.getTags() returns more information than
          GitClient.getTagNames(), so it is possible that this change is still
          not fast enough.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: src/main/java/hudson/plugins/git/util/GitUtils.java http://jenkins-ci.org/commit/git-plugin/18bc3dc8ab0fa590725f09cd31443e0aee298bbf Log: JENKINS-45447 Improve poll and checkout performance for repos with many tags JENKINS-45447 reports that checkout of a repository with many tags from a freestyle job using a wildcard in the branch name is dramatically slower than earlier releases. Many other cases which use the advanced branch selection mechanism show the same performance problem. 31fedce9c added tag checks to the evaluation loop for branch names. Unfortunately, tag evaluation needs both the tag name and the SHA1 of the commit identified by the tag. The original; implementation called revParse() to compute that SHA1 for each tag. With many tags in the repository (bug report example was 60,000 tags), the checkout time increased dramatically. I found that even the number of tags in the git plugin repository could add as much as 5 seconds for the computation of SHA1 hashes. Those computed SHA1 hashes were then immediately discarded because they did not satisfy the branch name selection criteria. Calling GitClient.getTags() returns more information than GitClient.getTagNames(), so it is possible that this change is still not fast enough.

          Mark Waite added a comment - - edited

          Fixed in git client plugin 2.7.0 and git plugin 3.7.0, released 21 Dec 2017.

          Mark Waite added a comment - - edited Fixed in git client plugin 2.7.0 and git plugin 3.7.0, released 21 Dec 2017.

            markewaite Mark Waite
            yehorov Mykhaylo Yehorov
            Votes:
            6 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: