Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46487

Git Plugin only scans refs/heads on multibranch scan

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • git-plugin
    • None
    • Jenkins 2.75
      git-plugin 3.5.1

      Because this was completely breaking my build environment, I was able to track down the issue to the following file: https://github.com/jenkinsci/git-plugin/blob/git-3.5.1/src/main/java/jenkins/plugins/git/AbstractGitSCMSource.java

      Broken on sha a4febc060a23740816d660915b9fe0a7a6ffdcf6

      Working on sha d7c3ece2553cf0494a56c766f62a6c9f340bbe4c

      To get it to work in my environment, I reverted this one file to the previous sha and built the plugin locally. Installed on my Jenkins instance and the issue was fixed.

       

      Issue:

      For multibranch projects- not tested on single branch projects

      Basically, when listing remotes during branch discovery, it forces the '-h' option for git ls-remotes and ignores any refspec given.

      For example, I give the refspec '+refs/collab/:refs/remotes/origin/collab/' but it only lists (and therefore filters) remotes for '+refs/heads/:refs/remotes/origin/heads/'

      Our org needs the custom refs location because that is how we automate code reviews.  On version 3.4.1 of the plugin, this refspec worked just fine.  On version 3.5.0, it broke and deleted all the collab branches.

      Please let me know if you need any additional details.  I would submit a PR but, my Java is weak.  Thank you, kindly!

       

       

       

          [JENKINS-46487] Git Plugin only scans refs/heads on multibranch scan

          Stephen Catt added a comment -

          Hello stephenconnolly :

           

          Any movement or analysis on this ticket?  Should I just submit a PR with the aforementioned file reverted as that seems to fix this issue?

           

          Thanks,

          Steve Catt

          Stephen Catt added a comment - Hello stephenconnolly :   Any movement or analysis on this ticket?  Should I just submit a PR with the aforementioned file reverted as that seems to fix this issue?   Thanks, Steve Catt

          Mark Waite added a comment -

          I believe the plan is that those who want to discover tags (and other special case refspecs) will do that by adding a plugin which adds a trait to discover those additional refspecs.  The branch source base use case was not intended to handle tags, and rather than adding more special cases to the plugin (with the resulting complexity in the plugin), additional plugins will be created which add those cases.

          The tag discovery trait pull request was accepted into the git plugin today.

          Don't submit a pull request reverting the change mentioned in the description of this bug.  It will be rejected.  That is an intentional change as far as I can tell.

          Mark Waite added a comment - I believe the plan is that those who want to discover tags (and other special case refspecs) will do that by adding a plugin which adds a trait to discover those additional refspecs.  The branch source base use case was not intended to handle tags, and rather than adding more special cases to the plugin (with the resulting complexity in the plugin), additional plugins will be created which add those cases. The tag discovery trait pull request was accepted into the git plugin today. Don't submit a pull request reverting the change mentioned in the description of this bug.  It will be rejected.  That is an intentional change as far as I can tell.

          Stephen Catt added a comment -

          markewaite Thank you for the response.  For our repo, we use custom heads rather than tags (in our case refs/collab/<branch name>).  My issue is that the branch discovery function on custom heads prior to version 3.5.0 worked perfectly (branches rather than tags).  But, 3.5.0 introduced always including the "-h" option during branch discovery, so now it only retrieves "refs/heads/<branch name>".

           

          If this was an intended change, you may want to indicate in the changelogs that it is a breaking change for anyone using custom heads in multibranch discovery.  We lost and had to restore a significant amount of build history because of this change.

          Thank you and have a nice weekend!

          Stephen Catt added a comment - markewaite Thank you for the response.  For our repo, we use custom heads rather than tags (in our case refs/collab/<branch name>).  My issue is that the branch discovery function on custom heads prior to version 3.5.0 worked perfectly (branches rather than tags).  But, 3.5.0 introduced always including the "-h" option during branch discovery, so now it only retrieves "refs/heads/<branch name>".   If this was an intended change, you may want to indicate in the changelogs that it is a breaking change for anyone using custom heads in multibranch discovery.  We lost and had to restore a significant amount of build history because of this change. Thank you and have a nice weekend!

          So IIUC you can resolve this on the Jenkins side by importing these heads slightly differently, e.g. +refs/heads/collab/:refs/remotes/origin/collab/ (and if your build relies on those heads being also at refs/collab it will cost nothing to fetch them there at the same time with a second refspec

          The reason for the change was to resolve inconsistencies in identifying heads. We need a deterministic single namespace. Currently what we have in the Branch Sources is a priority of something like:

          1. If there is a ref/heads it owns the name ahead of PRs and tags
          2. If there is a PR it owns the name (always in the format PR-nnn) ahead of tags
          3. Tags only own the name if nobody else has claimed it first

          The Git CLI has the luxury of being able to call out "ambiguous ref" if you try to reference something with a short name and there are multiple candidates.

          Unfortunately for the SCMSource implementations, in the time between listing all the SCMHeads and fetching a revision for a manually triggered build, somebody could have created an ambiguous name between your refs/collab and refs/heads and Jenkins has no context within which to resolve those.

          What we really want is some sort of "discover other heads" trait, that would return not a plain SCMHead but rather a subclass that stored the ref section from whence it came. Without that information, we would be unable to correctly discover the revision outside of a full index process.

          An alternative would be to refactor the plugin not to use the SCMHead class directly for branches, but instead use a sub-class, e.g. BranchSCMHead. That subclass could have an advertised property that would be defaulting to R_HEADS for legacy instances and could be the refs that we pull from for non-legacy. This would allow the +refs/collab/:refs/remotes/origin/collab/ that you had before.

          I took a look at seeing if we could hack things and just prefix any non-refs/heads refs with the bit after the refs/ so that refs/heads/master would be master and refs/collab/master would be collab/master but the problem is that you get conflicts with refs/heads/collab/master and there are a number of places in the code where we would be unable to reconstruct the ref to query the revision of from just collab/master.

          Given the plugins that depend on Git using SCMHead the non-hack fix will likely have to wait for a 4.0.0 release of the Git plugin.

          If you can confirm that my suggested workaround resolves your issue for now then I will create the RFE Epic to move to a sub-class of SCMHead

          Stephen Connolly added a comment - So IIUC you can resolve this on the Jenkins side by importing these heads slightly differently, e.g. +refs/heads/collab/:refs/remotes/origin/collab/ (and if your build relies on those heads being also at refs/collab it will cost nothing to fetch them there at the same time with a second refspec The reason for the change was to resolve inconsistencies in identifying heads. We need a deterministic single namespace. Currently what we have in the Branch Sources is a priority of something like: If there is a ref/heads it owns the name ahead of PRs and tags If there is a PR it owns the name (always in the format PR-nnn) ahead of tags Tags only own the name if nobody else has claimed it first The Git CLI has the luxury of being able to call out "ambiguous ref" if you try to reference something with a short name and there are multiple candidates. Unfortunately for the SCMSource implementations, in the time between listing all the SCMHeads and fetching a revision for a manually triggered build, somebody could have created an ambiguous name between your refs/collab and refs/heads and Jenkins has no context within which to resolve those. What we really want is some sort of "discover other heads" trait, that would return not a plain SCMHead but rather a subclass that stored the ref section from whence it came. Without that information, we would be unable to correctly discover the revision outside of a full index process. An alternative would be to refactor the plugin not to use the SCMHead class directly for branches, but instead use a sub-class, e.g. BranchSCMHead. That subclass could have an advertised property that would be defaulting to R_HEADS for legacy instances and could be the refs that we pull from for non-legacy. This would allow the +refs/collab/:refs/remotes/origin/collab/ that you had before. I took a look at seeing if we could hack things and just prefix any non- refs/heads refs with the bit after the refs/ so that refs/heads/master would be master and refs/collab/master would be collab/master but the problem is that you get conflicts with refs/heads/collab/master and there are a number of places in the code where we would be unable to reconstruct the ref to query the revision of from just collab/master . Given the plugins that depend on Git using SCMHead the non-hack fix will likely have to wait for a 4.0.0 release of the Git plugin. If you can confirm that my suggested workaround resolves your issue for now then I will create the RFE Epic to move to a sub-class of SCMHead

          Stephen Catt added a comment - - edited

          stephenconnolly,

          Sorry for the delay.  I had to wait until the weekend to try this out so I didn't interrupt dev builds.  Unfortunately, using the refspec you suggested did not work. I tried several different iterations of it to no avail.  I'll continue to use my custom modified plugin until I have the time to create a branch discovery plugin for my collab branches (or to remove using that ref, as it is legacy for us).

           

          Also, to modify this without breaking things, I didn't revert the entire file.  I just added the old discoverBranches method into the AbstractGitSCMSource retrieve method as an override. So, when it's looking for branches, it uses the GitClient.getRemoteBranches() method instead of getRemoteReferences. Tags still uses the new discoverBranches private method. It works for now.  I've attached the edited file so you can diff it against the current repo version.

           

          Thanks,

          Steve Catt

          Stephen Catt added a comment - - edited stephenconnolly , Sorry for the delay.  I had to wait until the weekend to try this out so I didn't interrupt dev builds.  Unfortunately, using the refspec you suggested did not work. I tried several different iterations of it to no avail.  I'll continue to use my custom modified plugin until I have the time to create a branch discovery plugin for my collab branches (or to remove using that ref, as it is legacy for us).   Also, to modify this without breaking things, I didn't revert the entire file.  I just added the old discoverBranches method into the AbstractGitSCMSource retrieve method as an override. So, when it's looking for branches, it uses the GitClient.getRemoteBranches() method instead of getRemoteReferences. Tags still uses the new discoverBranches private method. It works for now.  I've attached the edited file so you can diff it against the current repo version.   Thanks, Steve Catt

          Hello, seems that we are stuck with the exact same issue. 

          We build pull-requests that are stored under "refs/pull-requests/*/merge" with our Bitbucket server. Now we use 3.4.1 plugin version and cannot update to 3.7.0 where the flow is broken. Is there any possible workaround or solution for that? 

          Thanks.

          Dmitry Yunitsky added a comment - Hello, seems that we are stuck with the exact same issue.  We build pull-requests that are stored under "refs/pull-requests/*/merge" with our Bitbucket server. Now we use 3.4.1 plugin version and cannot update to 3.7.0 where the flow is broken. Is there any possible workaround or solution for that?  Thanks.

          yunikkk, scatt, kr for me this issue is fixed with git-plugin 3.9.0

          There is a new option called "Discover other refs".

           

          I've already closed #45990 which sounded very similar to me. Could you please confirm that the issue is fixed for you as well?

          Florian Bäuerle added a comment - yunikkk , scatt , kr  for me this issue is fixed with git-plugin 3.9.0 There is a new option called "Discover other refs".   I've already closed #45990 which sounded very similar to me. Could you please confirm that the issue is fixed for you as well?

          Stephen Catt added a comment -

          fbaeuerle,

          I installed the update over the weekend.  Though the "discover other refs" plugin will now successfully scan for the branches, it won't build them.  It kept throwing an error that it could not find a revision to build.  I was looking through the git plugin repo and it looks like they have not implemented telescoping support yet, which may be the issue.  From what I could tell, this i tracked in JENKINS-51134

          So, hopefully, the next version will fix it.

           

          Stephen Catt added a comment - fbaeuerle , I installed the update over the weekend.  Though the "discover other refs" plugin will now successfully scan for the branches, it won't build them.  It kept throwing an error that it could not find a revision to build.  I was looking through the git plugin repo and it looks like they have not implemented telescoping support yet, which may be the issue.  From what I could tell, this i tracked in JENKINS-51134 So, hopefully, the next version will fix it.  

          scatt

          Thanks. Correct - looks like I was a bit too overzealously closing the other Issue.

          I get this message:
          ERROR: Could not determine exact tip revision of pull/2212/merge; falling back to nondeterministic checkout

          In the end, the correct commit is checked out but as soon as the actual pipeline script is executed on a node, the job fails with java.nio.file.NoSuchFileException. The Dockerfile cannot be found, which makes sense, because there is absolutetly nothing created on the slave (not even a workdir is being created).

          Florian Bäuerle added a comment - scatt Thanks. Correct - looks like I was a bit too overzealously closing the other Issue. I get this message: ERROR: Could not determine exact tip revision of pull/2212/merge; falling back to nondeterministic checkout In the end, the correct commit is checked out but as soon as the actual pipeline script is executed on a node, the job fails with java.nio.file.NoSuchFileException. The Dockerfile cannot be found, which makes sense, because there is absolutetly nothing created on the slave (not even a workdir is being created).

          Jan Nijtmans added a comment -

          Just encountered this bug in our environment as well.  What I see in our log is:

              git rev-parse refs/pull-requests-2^{commit} # timeout=10

          which should have been:

              git rev-parse refs/pull-requests/2/merge^{commit} # timeout=10

          So it looks like the "rev-parse" command is constructed wrong: The translated name is used in stead of the internal GIT name. for the ref.

           

          Hope this helps,

                    Jan Nijtmans

          Jan Nijtmans added a comment - Just encountered this bug in our environment as well.  What I see in our log is:     git rev-parse refs/pull-requests-2^{commit} # timeout=10 which should have been:     git rev-parse refs/pull-requests/2/merge^{commit} # timeout=10 So it looks like the "rev-parse" command is constructed wrong: The translated name is used in stead of the internal GIT name. for the ref.   Hope this helps,           Jan Nijtmans

            Unassigned Unassigned
            scatt Stephen Catt
            Votes:
            6 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated: