Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-72731

Refspec does not work for Multibranch Pipelines

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • git-plugin
    • None

      PS: Im not sure if it is only for Multibranch Pipelines, but thats my current workflow, and the only one Im aware of.

       

      Setting Refspec on the Configuration level:

       

      Making sure that the refspec is honored:

       

      But still when the branch runs, tags are fetched, but not the declared refspec:

       

       

      I believe this a regression, since this has worked for me in the past.

          [JENKINS-72731] Refspec does not work for Multibranch Pipelines

          Mark Waite added a comment - - edited

          I place the refspec configuration inside the Jenkinsfile rather than placing it on the multibranch Pipeline definition in my test jobs. The refspec inside the Jenkinsfile works as expected. Here are some examples of that configuration:

          I assume that "origin/develop" is a single branch on the remote repository. If it is a single branch, then I don't see how a multibranch Pipeline provides any benefit. It will find at most a single branch named "develop" and that single branch will result in a single job named "develop" in the multibranch Pipeline folder.

          If "origin/develop" is not a single branch on the remote repository, then I believe you need a different syntax to match all branches that are beneath the "develop" folder. The syntax for all branches in the develop folder is

          +refs/heads/develop/*:refs/remotes/origin/develop/*
          

          Mark Waite added a comment - - edited I place the refspec configuration inside the Jenkinsfile rather than placing it on the multibranch Pipeline definition in my test jobs. The refspec inside the Jenkinsfile works as expected. Here are some examples of that configuration: Multibranch Pipeline with JGit Multibranch Pipeline with command line git I assume that "origin/develop" is a single branch on the remote repository. If it is a single branch, then I don't see how a multibranch Pipeline provides any benefit. It will find at most a single branch named "develop" and that single branch will result in a single job named "develop" in the multibranch Pipeline folder. If "origin/develop" is not a single branch on the remote repository, then I believe you need a different syntax to match all branches that are beneath the "develop" folder. The syntax for all branches in the develop folder is +refs/heads/develop/*:refs/remotes/origin/develop/*

          Ruan added a comment -

          Well, for a bit more context, I need Jenkins to be able to determine the commits between this branch (that is being built via the Multibranch Pipeline) and develop. I use this to parse Jira tickets from commit messages. 

           

          Thus, I need git to know about develop. This used to work before, but now there is no ref to develop. Ie the error in the last picture

          Ruan added a comment - Well, for a bit more context, I need Jenkins to be able to determine the commits between this branch (that is being built via the Multibranch Pipeline) and develop. I use this to parse Jira tickets from commit messages.    Thus, I need git to know about develop. This used to work before, but now there is no ref to develop. Ie the error in the last picture

          Mark Waite added a comment -

          I need Jenkins to be able to determine the commits between this branch (that is being built via the Multibranch Pipeline) and develop.

          You need a refspec that includes the branch you are building and the develop branch. The refspec you've defined does not include the branch you are building. The refspec definition can be multiple strings. Add a refspec definition (with a trailing '*') to match the branches you are building.

          I need git to know about develop. This used to work before, but now there is no ref to develop. Ie the error in the last picture

          That would have worked with that refspec in the past (by a happy accident) if you had not enabled the "honor refspec during clone" setting. In that case, the initial clone copied all the refspecs from the remote repository. The "honor refspec during clone" setting was added a while ago so that the initial clone did not make the mistake of always cloning all refspecs.

          Mark Waite added a comment - I need Jenkins to be able to determine the commits between this branch (that is being built via the Multibranch Pipeline) and develop. You need a refspec that includes the branch you are building and the develop branch. The refspec you've defined does not include the branch you are building. The refspec definition can be multiple strings. Add a refspec definition (with a trailing '*') to match the branches you are building. I need git to know about develop. This used to work before, but now there is no ref to develop. Ie the error in the last picture That would have worked with that refspec in the past (by a happy accident) if you had not enabled the "honor refspec during clone" setting. In that case, the initial clone copied all the refspecs from the remote repository. The "honor refspec during clone" setting was added a while ago so that the initial clone did not make the mistake of always cloning all refspecs.

          Ruan added a comment -

          Super, Ill give it a go and come back to this

          Ruan added a comment - Super, Ill give it a go and come back to this

          Ruan added a comment - - edited

          Actually, no luck. Seems like the honouring doesn't work for Multibranch pipelines as it should

          Previously, when this worked, my refspec was defined as:

          +refs/heads/develop:refs/remotes/origin/develop +refs/heads/master:refs/remotes/origin/master 

          This can be seen in the historical jobs that passed with this configuration (this is from the logs):

           > git fetch --no-tags --force --progress -- https://bitbucket.org/xxx/xxx.git +refs/heads/feat/playwright-e2e:refs/remotes/origin/feat/playwright-e2e +refs/heads/develop:refs/remotes/origin/develop +refs/heads/master:refs/remotes/origin/master # timeout=10

           

          But now

          My refspec is defined as (as per the comment above):

           

          +refs/heads/master:refs/remotes/origin/master +refs/heads/develop:refs/remotes/origin/develop +refs/heads/*:refs/remotes/@{remote}/* 

          This doesnt influence the fetch at all. Yes, I do have the honor ref spec checked.

          The fetch looks like this

          > git fetch --no-tags --force --progress – https://xxx@bitbucket.org/xxx/xxx.git +refs/heads/FixDeployment:refs/remotes/origin/FixDeployment # timeout=10 

          With Honor Refspec off

          This works as expected, as you mentioned, since it adds all branches.

          > git fetch --no-tags --force --progress -- git@bitbucket.org:xxx/xxx.git +refs/heads/*:refs/remotes/origin/* # timeout=10 

          This works for now, but this is not a long term solution. The refspec should work for multibranch pipelines like before

          Ruan added a comment - - edited Actually, no luck. Seems like the honouring doesn't work for Multibranch pipelines as it should Previously , when this worked, my refspec was defined as: +refs/heads/develop:refs/remotes/origin/develop +refs/heads/master:refs/remotes/origin/master This can be seen in the historical jobs that passed with this configuration (this is from the logs):  > git fetch --no-tags --force --progress -- https: //bitbucket.org/xxx/xxx.git +refs/heads/feat/playwright-e2e:refs/remotes/origin/feat/playwright-e2e +refs/heads/develop:refs/remotes/origin/develop +refs/heads/master:refs/remotes/origin/master # timeout=10   But now My refspec is defined as (as per the comment above):   +refs/heads/master:refs/remotes/origin/master +refs/heads/develop:refs/remotes/origin/develop +refs/heads/*:refs/remotes/@{remote}/* This doesnt influence the fetch at all. Yes, I do have the honor ref spec checked. The fetch looks like this > git fetch --no-tags --force --progress – https: //xxx@bitbucket.org/xxx/xxx.git +refs/heads/FixDeployment:refs/remotes/origin/FixDeployment # timeout=10 With Honor Refspec off This works as expected, as you mentioned, since it adds all branches. > git fetch --no-tags --force --progress -- git@bitbucket.org:xxx/xxx.git +refs/heads/*:refs/remotes/origin/* # timeout=10 This works for now, but this is not a long term solution. The refspec should work for multibranch pipelines like before

          Ruan added a comment - - edited

          As a note, its not possible to specificy the current branch in the refspec, for the configuration on a Multibranch Pipeline. (This makes sense, since it can/will change often)

           

          It seems the plugin used to append the refspec previously, when honour was turned off, as can be seen by my historical logs. So this quote is false

          That would have worked with that refspec in the past (by a happy accident) if you had not enabled the "honor refspec during clone" setting. In that case, the initial clone copied all the refspecs from the remote repository. The "honor refspec during clone" setting was added a while ago so that the initial clone did not make the mistake of always cloning all refspecs.

           

          Now the current behaviour seems to be:

          • Honor Refspec off
            • Fetch all branches
          • Honor Refspec on
            • Just fetch the single branch. I think this means it does nothing?

          Regardless of what i fill in the refspec, for a multibranch pipeline, it will always just fetch the single branch if Honor Refspec is on

           

          ---

          PS: It seems that the default behaviour is Honor Refspec On. At least, I have to explicitly add Advanced clone behaviours to turn it off

          Ruan added a comment - - edited As a note, its not possible to specificy the current branch in the refspec, for the configuration on a Multibranch Pipeline. (This makes sense, since it can/will change often)   It seems the plugin used to append the refspec previously, when honour was turned off, as can be seen by my historical logs. So this quote is false That would have worked with that refspec in the past (by a happy accident) if you had not enabled the "honor refspec during clone" setting. In that case, the initial clone copied all the refspecs from the remote repository. The "honor refspec during clone" setting was added a while ago so that the initial clone did not make the mistake of always cloning all refspecs.   Now the current behaviour seems to be: Honor Refspec off Fetch all branches Honor Refspec on Just fetch the single branch. I think this means it does nothing? Regardless of what i fill in the refspec, for a multibranch pipeline, it will always just fetch the single branch if Honor Refspec is on   --- PS: It seems that the default behaviour is Honor Refspec On . At least, I have to explicitly add Advanced clone behaviours  to turn it off

          Mark Waite added a comment -

          As a note, its not possible to specify the current branch in the refspec, for the configuration on a Multibranch Pipeline. (This makes sense, since it can/will change often)

          I've confirmed in my test environment that a branch specific refspec can be used in a multibranch Pipeline. The multibranch Pipeline provides a variable BRANCH_NAME that is the current branch being used in the multibranch Pipeline. That variable can be used to define a refspec that matches a single branch. My sample Jenkinsfile shows that technique. That Jenkinsfile is a small branch that is part of a very large repository. When that multibranch job runs, it only downloads the specific refspec for the branch.

          I've confirmed that I can specify multiple refspecs in the checkout command for a multibranch Pipeline and that the multiple refspecs are honored.

          Mark Waite added a comment - As a note, its not possible to specify the current branch in the refspec, for the configuration on a Multibranch Pipeline. (This makes sense, since it can/will change often) I've confirmed in my test environment that a branch specific refspec can be used in a multibranch Pipeline. The multibranch Pipeline provides a variable BRANCH_NAME that is the current branch being used in the multibranch Pipeline. That variable can be used to define a refspec that matches a single branch. My sample Jenkinsfile shows that technique. That Jenkinsfile is a small branch that is part of a very large repository . When that multibranch job runs, it only downloads the specific refspec for the branch. I've confirmed that I can specify multiple refspecs in the checkout command for a multibranch Pipeline and that the multiple refspecs are honored.

          Ruan added a comment - - edited

          Just had time to try out using 

          ```
          +refs/heads/${BRANCH_NAME}:refs/remotes/origin/${BRANCH_NAME}
          ```

          That doesn't seem to work via the GUI setup (still just refs the built branch). I didn't try via a pipeline file, but my assumption is that it would just work like you specified. 

          Slight side note:
          What is strange, is that I would just expect some sort of error to occur. 
          There seems to be a behaviour in Multibranch pipelines, where it just fails, and then defaults to the default refspec (only the single branch that it is building)

           

          In any-case, are you aiming to drop support for the GUI setup? Because then this issue can be closed, as I can probably just do the setup via jenkinsfile. 

          The documentation would then just need updating as well

          Ruan added a comment - - edited Just had time to try out using  ``` +refs/heads/${BRANCH_NAME}:refs/remotes/origin/${BRANCH_NAME} ``` That doesn't seem to work via the GUI setup (still just refs the built branch). I didn't try via a pipeline file, but my assumption is that it would just work like you specified.  Slight side note: What is strange, is that I would just expect some sort of error to occur.  There seems to be a behaviour in Multibranch pipelines, where it just fails, and then defaults to the default refspec (only the single branch that it is building)   In any-case, are you aiming to drop support for the GUI setup? Because then this issue can be closed, as I can probably just do the setup via jenkinsfile.  The documentation would then just need updating as well

          Mark Waite added a comment -

          That doesn't seem to work via the GUI setup (still just refs the built branch).

          I think that you may be mixing two different cases here, the definition of the parent multibranch Pipeline job and the definition of the Pipeline job for each branch that is detected.

          The parent multibranch Pipeline job (which is what I think you mean when you say "GUI setup") must be able to detect all branches. It cannot create branches that it does not detect and if the refspec limits to a specific branch name, it will only detect for that branch name.

          The Pipeline job for each branch that is detected can use BRANCH_NAME in the refspec of the checkout step of the job. That allows the job created for that specific branch to be limited to a single refspec.

          Mark Waite added a comment - That doesn't seem to work via the GUI setup (still just refs the built branch). I think that you may be mixing two different cases here, the definition of the parent multibranch Pipeline job and the definition of the Pipeline job for each branch that is detected. The parent multibranch Pipeline job (which is what I think you mean when you say "GUI setup") must be able to detect all branches. It cannot create branches that it does not detect and if the refspec limits to a specific branch name, it will only detect for that branch name. The Pipeline job for each branch that is detected can use BRANCH_NAME in the refspec of the checkout step of the job. That allows the job created for that specific branch to be limited to a single refspec.

          Ruan added a comment -

          Oh, yeah I understand that.

          I think the issue here is, that the refspec on the parent branch used to be passed down to the child branch. Those branches were added to the refspec of every single child. And it used to work just fine. That's how I built all my branches for months. This can be seen by my playwright branch log in a previous message. I do a diff between the branch being built and develop.

          But, when I recently upgraded the git plugin, it started failing. That's because my pipeline no longer had a reference to the branches set in the parent pipeline setup. The only way to get my old branches to work, is to turn off Honor refspec. This will add all branches to my refpec, in the child pipeline. This is not what I want, nor is it what the behaviour was before.

          Perhaps this is correct, and intended. And then this issue can be closed, and I would just open a feature request for a method to enforce a base refspec on all child branches.

          Ruan added a comment - Oh, yeah I understand that. I think the issue here is, that the refspec on the parent branch  used to be passed down to the child branch. Those branches were added to the refspec of every single child . And it used to work just fine. That's how I built all my branches for months. This can be seen by my playwright branch log in a previous message. I do a diff between the branch being built and develop. But, when I recently upgraded the git plugin, it started failing. That's because my pipeline no longer had a reference to the branches set in the parent pipeline setup. The only way to get my old branches to work, is to turn off  Honor refspec.  This will add all branches to my refpec, in the child pipeline. This is not what I want, nor is it what the behaviour was before. Perhaps this is correct, and intended. And then this issue can be closed, and I would just open a feature request for a method to enforce a base refspec on all child branches.

            Unassigned Unassigned
            rheynsza Ruan
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: