Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46487

Git Plugin only scans refs/heads on multibranch scan

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • git-plugin
    • None
    • Jenkins 2.75
      git-plugin 3.5.1

      Because this was completely breaking my build environment, I was able to track down the issue to the following file: https://github.com/jenkinsci/git-plugin/blob/git-3.5.1/src/main/java/jenkins/plugins/git/AbstractGitSCMSource.java

      Broken on sha a4febc060a23740816d660915b9fe0a7a6ffdcf6

      Working on sha d7c3ece2553cf0494a56c766f62a6c9f340bbe4c

      To get it to work in my environment, I reverted this one file to the previous sha and built the plugin locally. Installed on my Jenkins instance and the issue was fixed.

       

      Issue:

      For multibranch projects- not tested on single branch projects

      Basically, when listing remotes during branch discovery, it forces the '-h' option for git ls-remotes and ignores any refspec given.

      For example, I give the refspec '+refs/collab/:refs/remotes/origin/collab/' but it only lists (and therefore filters) remotes for '+refs/heads/:refs/remotes/origin/heads/'

      Our org needs the custom refs location because that is how we automate code reviews.  On version 3.4.1 of the plugin, this refspec worked just fine.  On version 3.5.0, it broke and deleted all the collab branches.

      Please let me know if you need any additional details.  I would submit a PR but, my Java is weak.  Thank you, kindly!

       

       

       

          [JENKINS-46487] Git Plugin only scans refs/heads on multibranch scan

          Stephen Catt created issue -

          Mark Waite added a comment -

          @stephenc I think this one needs your insights

          Mark Waite added a comment - @stephenc I think this one needs your insights
          Mark Waite made changes -
          Assignee Original: Mark Waite [ markewaite ] New: Stephen Connolly [ stephenc ]

          Konstantin Ryakhovskiy added a comment - - edited

          stephenc is there any workaround how we can build e.g. tags?

          we were using following refspec:

          +refs/tags/*:refs/remotes/origin/tags/*

          as you reported, the refspec functionality is broken now.

          can you maybe build a snapshot version before all your PRs are merged?

          Thanks in advance

          Konstantin Ryakhovskiy added a comment - - edited stephenc is there any workaround how we can build e.g. tags? we were using following refspec: +refs/tags/*:refs/remotes/origin/tags/* as you reported, the refspec functionality is broken now. can you maybe build a snapshot version before all your PRs are merged? Thanks in advance

          Stephen Catt added a comment -

          Hello stephenconnolly :

           

          Any movement or analysis on this ticket?  Should I just submit a PR with the aforementioned file reverted as that seems to fix this issue?

           

          Thanks,

          Steve Catt

          Stephen Catt added a comment - Hello stephenconnolly :   Any movement or analysis on this ticket?  Should I just submit a PR with the aforementioned file reverted as that seems to fix this issue?   Thanks, Steve Catt

          Mark Waite added a comment -

          I believe the plan is that those who want to discover tags (and other special case refspecs) will do that by adding a plugin which adds a trait to discover those additional refspecs.  The branch source base use case was not intended to handle tags, and rather than adding more special cases to the plugin (with the resulting complexity in the plugin), additional plugins will be created which add those cases.

          The tag discovery trait pull request was accepted into the git plugin today.

          Don't submit a pull request reverting the change mentioned in the description of this bug.  It will be rejected.  That is an intentional change as far as I can tell.

          Mark Waite added a comment - I believe the plan is that those who want to discover tags (and other special case refspecs) will do that by adding a plugin which adds a trait to discover those additional refspecs.  The branch source base use case was not intended to handle tags, and rather than adding more special cases to the plugin (with the resulting complexity in the plugin), additional plugins will be created which add those cases. The tag discovery trait pull request was accepted into the git plugin today. Don't submit a pull request reverting the change mentioned in the description of this bug.  It will be rejected.  That is an intentional change as far as I can tell.

          Stephen Catt added a comment -

          markewaite Thank you for the response.  For our repo, we use custom heads rather than tags (in our case refs/collab/<branch name>).  My issue is that the branch discovery function on custom heads prior to version 3.5.0 worked perfectly (branches rather than tags).  But, 3.5.0 introduced always including the "-h" option during branch discovery, so now it only retrieves "refs/heads/<branch name>".

           

          If this was an intended change, you may want to indicate in the changelogs that it is a breaking change for anyone using custom heads in multibranch discovery.  We lost and had to restore a significant amount of build history because of this change.

          Thank you and have a nice weekend!

          Stephen Catt added a comment - markewaite Thank you for the response.  For our repo, we use custom heads rather than tags (in our case refs/collab/<branch name>).  My issue is that the branch discovery function on custom heads prior to version 3.5.0 worked perfectly (branches rather than tags).  But, 3.5.0 introduced always including the "-h" option during branch discovery, so now it only retrieves "refs/heads/<branch name>".   If this was an intended change, you may want to indicate in the changelogs that it is a breaking change for anyone using custom heads in multibranch discovery.  We lost and had to restore a significant amount of build history because of this change. Thank you and have a nice weekend!

          So IIUC you can resolve this on the Jenkins side by importing these heads slightly differently, e.g. +refs/heads/collab/:refs/remotes/origin/collab/ (and if your build relies on those heads being also at refs/collab it will cost nothing to fetch them there at the same time with a second refspec

          The reason for the change was to resolve inconsistencies in identifying heads. We need a deterministic single namespace. Currently what we have in the Branch Sources is a priority of something like:

          1. If there is a ref/heads it owns the name ahead of PRs and tags
          2. If there is a PR it owns the name (always in the format PR-nnn) ahead of tags
          3. Tags only own the name if nobody else has claimed it first

          The Git CLI has the luxury of being able to call out "ambiguous ref" if you try to reference something with a short name and there are multiple candidates.

          Unfortunately for the SCMSource implementations, in the time between listing all the SCMHeads and fetching a revision for a manually triggered build, somebody could have created an ambiguous name between your refs/collab and refs/heads and Jenkins has no context within which to resolve those.

          What we really want is some sort of "discover other heads" trait, that would return not a plain SCMHead but rather a subclass that stored the ref section from whence it came. Without that information, we would be unable to correctly discover the revision outside of a full index process.

          An alternative would be to refactor the plugin not to use the SCMHead class directly for branches, but instead use a sub-class, e.g. BranchSCMHead. That subclass could have an advertised property that would be defaulting to R_HEADS for legacy instances and could be the refs that we pull from for non-legacy. This would allow the +refs/collab/:refs/remotes/origin/collab/ that you had before.

          I took a look at seeing if we could hack things and just prefix any non-refs/heads refs with the bit after the refs/ so that refs/heads/master would be master and refs/collab/master would be collab/master but the problem is that you get conflicts with refs/heads/collab/master and there are a number of places in the code where we would be unable to reconstruct the ref to query the revision of from just collab/master.

          Given the plugins that depend on Git using SCMHead the non-hack fix will likely have to wait for a 4.0.0 release of the Git plugin.

          If you can confirm that my suggested workaround resolves your issue for now then I will create the RFE Epic to move to a sub-class of SCMHead

          Stephen Connolly added a comment - So IIUC you can resolve this on the Jenkins side by importing these heads slightly differently, e.g. +refs/heads/collab/:refs/remotes/origin/collab/ (and if your build relies on those heads being also at refs/collab it will cost nothing to fetch them there at the same time with a second refspec The reason for the change was to resolve inconsistencies in identifying heads. We need a deterministic single namespace. Currently what we have in the Branch Sources is a priority of something like: If there is a ref/heads it owns the name ahead of PRs and tags If there is a PR it owns the name (always in the format PR-nnn) ahead of tags Tags only own the name if nobody else has claimed it first The Git CLI has the luxury of being able to call out "ambiguous ref" if you try to reference something with a short name and there are multiple candidates. Unfortunately for the SCMSource implementations, in the time between listing all the SCMHeads and fetching a revision for a manually triggered build, somebody could have created an ambiguous name between your refs/collab and refs/heads and Jenkins has no context within which to resolve those. What we really want is some sort of "discover other heads" trait, that would return not a plain SCMHead but rather a subclass that stored the ref section from whence it came. Without that information, we would be unable to correctly discover the revision outside of a full index process. An alternative would be to refactor the plugin not to use the SCMHead class directly for branches, but instead use a sub-class, e.g. BranchSCMHead. That subclass could have an advertised property that would be defaulting to R_HEADS for legacy instances and could be the refs that we pull from for non-legacy. This would allow the +refs/collab/:refs/remotes/origin/collab/ that you had before. I took a look at seeing if we could hack things and just prefix any non- refs/heads refs with the bit after the refs/ so that refs/heads/master would be master and refs/collab/master would be collab/master but the problem is that you get conflicts with refs/heads/collab/master and there are a number of places in the code where we would be unable to reconstruct the ref to query the revision of from just collab/master . Given the plugins that depend on Git using SCMHead the non-hack fix will likely have to wait for a 4.0.0 release of the Git plugin. If you can confirm that my suggested workaround resolves your issue for now then I will create the RFE Epic to move to a sub-class of SCMHead

          Stephen Catt added a comment - - edited

          stephenconnolly,

          Sorry for the delay.  I had to wait until the weekend to try this out so I didn't interrupt dev builds.  Unfortunately, using the refspec you suggested did not work. I tried several different iterations of it to no avail.  I'll continue to use my custom modified plugin until I have the time to create a branch discovery plugin for my collab branches (or to remove using that ref, as it is legacy for us).

           

          Also, to modify this without breaking things, I didn't revert the entire file.  I just added the old discoverBranches method into the AbstractGitSCMSource retrieve method as an override. So, when it's looking for branches, it uses the GitClient.getRemoteBranches() method instead of getRemoteReferences. Tags still uses the new discoverBranches private method. It works for now.  I've attached the edited file so you can diff it against the current repo version.

           

          Thanks,

          Steve Catt

          Stephen Catt added a comment - - edited stephenconnolly , Sorry for the delay.  I had to wait until the weekend to try this out so I didn't interrupt dev builds.  Unfortunately, using the refspec you suggested did not work. I tried several different iterations of it to no avail.  I'll continue to use my custom modified plugin until I have the time to create a branch discovery plugin for my collab branches (or to remove using that ref, as it is legacy for us).   Also, to modify this without breaking things, I didn't revert the entire file.  I just added the old discoverBranches method into the AbstractGitSCMSource retrieve method as an override. So, when it's looking for branches, it uses the GitClient.getRemoteBranches() method instead of getRemoteReferences. Tags still uses the new discoverBranches private method. It works for now.  I've attached the edited file so you can diff it against the current repo version.   Thanks, Steve Catt
          Stephen Catt made changes -
          Attachment New: AbstractGitSCMSource.java [ 39968 ]

            Unassigned Unassigned
            scatt Stephen Catt
            Votes:
            6 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated: