Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63785

Git plugin uses wrong branch unless second fetch is preserved

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • git-plugin
    • None
    • Git Plugin 4.4.3
      Jenkins 2.249.1
      CentOS 7

      Hello,

      First, thank you for maintaining and developing this nice Git plugin!

      Recently, I got some issues with some jobs using a wrong refspec to build. This only happened recently for jobs doing a "frech clone" and I suspect, only after an update of the Git Plugin.

      I don't have the issue when enabling "Preserve second fetch during checkout" option from PR-845 (JENKINS-49757).

      Maybe what is particular on my side is that there are multiple branches in my repo and my refspec contains 'refs/heads' ("refs/heads/mptcp_trunk/master"). From what I see in PR-845, that's why the second fetch is not done but I don't know why it is an issue to add 'refs/heads' in the refspec. When I look at what is done by Jenkins, it looks like I could only add the name of the branch but if I have not to specified 'refs/heads', I am confused with the option name and the description
      Also, I didn't set anything related to "Advanced Clone Option" as mentioned in the doc because I didn't find the corresponding option in the config (and to be honest, I don't know why I would like to do the clone and the fetch later differently ).

      (Note: I added "origin/" in the branch name just to try but it didn't help. Just to explain why we see a reference to "origin/xxx" in the logs)

      Here are the logs without "Preserve second fetch during checkout" option:

       > git fetch --tags --progress ssh://gerrit:29418/repo.git +refs/heads/*:refs/remotes/origin/* # timeout=10
       > git config remote.origin.url ssh://gerrit:29418/repo.git # timeout=10
       > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
      Avoid second fetch
       > git rev-parse FETCH_HEAD^{commit} # timeout=10
      Checking out Revision 5ac0ebcbb890f2a7852bca357641163837a219bd (origin/mptcp_trunk/master)
       > git config core.sparsecheckout # timeout=10
       > git checkout -f 5ac0ebcbb890f2a7852bca357641163837a219bd # timeout=10
      Commit message: "old commit"
      

      Here, you can see that all branches are fetches but then FETCH_HEAD is used with git checkout which is not correct: FETCH_HEAD corresponds to the first branch by alphabetical order on the remote (because we asked for refs/heads/*).

      Now here is the output with "Preserve second fetch during checkout" on:

       > git fetch --tags --progress ssh://gerrit:29418/repo.git +refs/heads/*:refs/remotes/origin/* # timeout=10
       > git config remote.origin.url ssh://gerrit:29418/repo.git # timeout=10
       > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
       > git config remote.origin.url ssh://gerrit:29418/repo.git # timeout=10
      Fetching upstream changes from ssh://gerrit:29418/repo.git
      using GIT_SSH to set credentials 
       > git fetch --tags --progress ssh://gerrit:29418/repo.git refs/heads/mptcp_trunk/master +refs/heads/mptcp_trunk/master:refs/remotes/origin/mptcp_trunk/master # timeout=10
       > git rev-parse FETCH_HEAD^{commit} # timeout=10
      Checking out Revision 21ec2388799b5bd7e18f3f2bd4aa7f0e2be9bb23 (origin/mptcp_trunk/master)
       > git config core.sparsecheckout # timeout=10
       > git checkout -f 21ec2388799b5bd7e18f3f2bd4aa7f0e2be9bb23 # timeout=10
      Commit message: "new commit"
      

      FETCH_HEAD is now corresponding to refs/heads/mptcp_trunk/master, what I want: then everything is OK.

      I hope this will help finding the issue not to have to enable "Preserve second fetch during checkout" option.

          [JENKINS-63785] Git plugin uses wrong branch unless second fetch is preserved

          Mark Waite added a comment -

          Thanks for reporting the issue. It looks like you've found a case that is missed in our logic to decide if the second fetch is required. Could you upload the config.xml for your job definition and / or the checkout step that is being performed (if this is a Pipeline job).

          I suspect that the job is configured with multiple repositories or with multiple refspecs and the plugin is not handling that correctly. We'll need the job details to assure that we can duplicate the problem and provide a fix. In this case, I expect that there is a way that the job can be redefined so that it does not require the second fetch. Whether there is or not, this is a case where the second fetch is required and the plugin did not detect that the second fetch is required. Sorry we missed that!

          Mark Waite added a comment - Thanks for reporting the issue. It looks like you've found a case that is missed in our logic to decide if the second fetch is required. Could you upload the config.xml for your job definition and / or the checkout step that is being performed (if this is a Pipeline job). I suspect that the job is configured with multiple repositories or with multiple refspecs and the plugin is not handling that correctly. We'll need the job details to assure that we can duplicate the problem and provide a fix. In this case, I expect that there is a way that the job can be redefined so that it does not require the second fetch. Whether there is or not, this is a case where the second fetch is required and the plugin did not detect that the second fetch is required. Sorry we missed that!

          Hi Mark,

          Thank you for your quick reply!

          Could you upload the config.xml for your job definition and / or the checkout step that is being performed (if this is a Pipeline job).

          I suspect that the job is configured with multiple repositories or with multiple refspecs and the plugin is not handling that correctly. We'll need the job details to assure that we can duplicate the problem and provide a fix.

          This is not a Pipeline job, there is no multiple repositories but multiple refspecs:

           

            <scmCheckoutRetryCount>3</scmCheckoutRetryCount>
            <properties>
              <hudson.model.ParametersDefinitionProperty>
                <parameterDefinitions>
                  <hudson.model.StringParameterDefinition>
                    <name>GERRIT_PROJECT</name>
                    <description>GERRIT_PROJECT parameter if not given by trigger.</description>
                    <defaultValue>repo</defaultValue>
                  </hudson.model.StringParameterDefinition>
                  <hudson.model.StringParameterDefinition>
                    <name>GERRIT_BRANCH</name>
                    <description>JJB configured GERRIT_BRANCH parameter.</description>
                    <defaultValue>mptcp_trunk/master</defaultValue>
                  </hudson.model.StringParameterDefinition>
                  <hudson.model.StringParameterDefinition>
                    <name>GERRIT_REFSPEC</name>
                    <description>GERRIT_REFSPEC parameter if not given by trigger. DO NOT CHANGE THIS: if the build triggered by "pollscm": pollscm will follow the last refspec modified here instead of the default value.</description>
                    <defaultValue>refs/heads/mptcp_trunk/master</defaultValue>
                  </hudson.model.StringParameterDefinition>
                </parameterDefinitions>
              </hudson.model.ParametersDefinitionProperty>
            </properties>
            <scm class="hudson.plugins.git.GitSCM">
              <configVersion>2</configVersion>
              <userRemoteConfigs>
                <hudson.plugins.git.UserRemoteConfig>
                  <name>origin</name>
                  <refspec>$GERRIT_REFSPEC +refs/heads/$GERRIT_BRANCH:refs/remotes/origin/$GERRIT_BRANCH</refspec>
                  <url>ssh://gerrit:29418/repo.git</url>
                  <credentialsId>REMOVED</credentialsId>
                </hudson.plugins.git.UserRemoteConfig>
              </userRemoteConfigs>
              <branches>
                <hudson.plugins.git.BranchSpec>
                  <name>origin/$GERRIT_BRANCH</name>
                </hudson.plugins.git.BranchSpec>
              </branches>
              <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>
              <remotePoll>false</remotePoll>
              <gitTool>Default</gitTool>
              <submoduleCfg class="list"/>
              <reference/>
              <gitConfigName>NAME</gitConfigName>
              <gitConfigEmail>EMAIL</gitConfigEmail>
              <extensions>
                <hudson.plugins.git.extensions.impl.BuildChooserSetting>
                  <buildChooser class="com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser"/>
                </hudson.plugins.git.extensions.impl.BuildChooserSetting>
                <hudson.plugins.git.extensions.impl.SubmoduleOption>
                  <disableSubmodules>true</disableSubmodules>
                  <recursiveSubmodules>false</recursiveSubmodules>
                  <trackingSubmodules>false</trackingSubmodules>
                  <parentCredentials>false</parentCredentials>
                  <reference/>
                  <timeout>10</timeout>
                </hudson.plugins.git.extensions.impl.SubmoduleOption>
              </extensions>
              <browser class="hudson.plugins.git.browser.GitWeb">
                <url>https://gerrit/gitweb?p=repo.git</url>
              </browser>
            </scm>
          

          Do you need more than the properties and scm sections? (If yes, I might have to strip some info: emails address, scripts, etc. because this example is coming from a "private" instance, I didn't have issues with OpenSource projects because they didn't do a clone recently )

          In this case, I expect that there is a way that the job can be redefined so that it does not require the second fetch. Whether there is or not, this is a case where the second fetch is required and the plugin did not detect that the second fetch is required.

          In my case, the first fetch getting all branches is not needed. It doesn't really hurt but I don't need all branches.

          But if for some reasons, all branches have to be fetched, a second fetch to the same repo is not needed if the plugin can use the first refspec in the config list instead of FETCH_HEAD. But I don't know if other users rely on FETCH_HEAD later. I guess they should use HEAD instead anyway.

          Sorry we missed that!

          All good, this happens

          Cheers,
          Matt

           

          Matthieu Baerts added a comment - Hi Mark, Thank you for your quick reply! Could you upload the config.xml for your job definition and / or the checkout step that is being performed (if this is a Pipeline job). I suspect that the job is configured with multiple repositories or with multiple refspecs and the plugin is not handling that correctly. We'll need the job details to assure that we can duplicate the problem and provide a fix. This is not a Pipeline job, there is no multiple repositories but multiple refspecs:   <scmCheckoutRetryCount>3</scmCheckoutRetryCount> <properties> <hudson.model.ParametersDefinitionProperty> <parameterDefinitions> <hudson.model.StringParameterDefinition> <name>GERRIT_PROJECT</name> <description>GERRIT_PROJECT parameter if not given by trigger.</description> <defaultValue>repo</defaultValue> </hudson.model.StringParameterDefinition> <hudson.model.StringParameterDefinition> <name>GERRIT_BRANCH</name> <description>JJB configured GERRIT_BRANCH parameter.</description> <defaultValue>mptcp_trunk/master</defaultValue> </hudson.model.StringParameterDefinition> <hudson.model.StringParameterDefinition> <name>GERRIT_REFSPEC</name> <description>GERRIT_REFSPEC parameter if not given by trigger. DO NOT CHANGE THIS: if the build triggered by "pollscm" : pollscm will follow the last refspec modified here instead of the default value.</description> <defaultValue>refs/heads/mptcp_trunk/master</defaultValue> </hudson.model.StringParameterDefinition> </parameterDefinitions> </hudson.model.ParametersDefinitionProperty> </properties> <scm class= "hudson.plugins.git.GitSCM" > <configVersion>2</configVersion> <userRemoteConfigs> <hudson.plugins.git.UserRemoteConfig> <name>origin</name> <refspec>$GERRIT_REFSPEC +refs/heads/$GERRIT_BRANCH:refs/remotes/origin/$GERRIT_BRANCH</refspec> <url>ssh: //gerrit:29418/repo.git</url> <credentialsId>REMOVED</credentialsId> </hudson.plugins.git.UserRemoteConfig> </userRemoteConfigs> <branches> <hudson.plugins.git.BranchSpec> <name>origin/$GERRIT_BRANCH</name> </hudson.plugins.git.BranchSpec> </branches> <doGenerateSubmoduleConfigurations> false </doGenerateSubmoduleConfigurations> <remotePoll> false </remotePoll> <gitTool>Default</gitTool> <submoduleCfg class= "list" /> <reference/> <gitConfigName>NAME</gitConfigName> <gitConfigEmail>EMAIL</gitConfigEmail> <extensions> <hudson.plugins.git.extensions.impl.BuildChooserSetting> <buildChooser class= "com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser" /> </hudson.plugins.git.extensions.impl.BuildChooserSetting> <hudson.plugins.git.extensions.impl.SubmoduleOption> <disableSubmodules> true </disableSubmodules> <recursiveSubmodules> false </recursiveSubmodules> <trackingSubmodules> false </trackingSubmodules> <parentCredentials> false </parentCredentials> <reference/> <timeout>10</timeout> </hudson.plugins.git.extensions.impl.SubmoduleOption> </extensions> <browser class= "hudson.plugins.git.browser.GitWeb" > <url>https: //gerrit/gitweb?p=repo.git</url> </browser> </scm> Do you need more than the properties and scm sections? (If yes, I might have to strip some info: emails address, scripts, etc. because this example is coming from a "private" instance, I didn't have issues with OpenSource projects because they didn't do a clone recently ) In this case, I expect that there is a way that the job can be redefined so that it does not require the second fetch. Whether there is or not, this is a case where the second fetch is required and the plugin did not detect that the second fetch is required. In my case, the first fetch getting all branches is not needed. It doesn't really hurt but I don't need all branches. But if for some reasons, all branches have to be fetched, a second fetch to the same repo is not needed if the plugin can use the first refspec in the config list instead of FETCH_HEAD. But I don't know if other users rely on FETCH_HEAD later. I guess they should use HEAD instead anyway. Sorry we missed that! All good, this happens Cheers, Matt  

          Mark Waite added a comment -

          matttbe that should be enough. Your freestyle job is missing the clone extension "Honor refspec on initial clone". That option will allow the git plugin to use your defined refspec on the first fetch instead of using its default wildcard refspec. See https://plugins.jenkins.io/git/#clone-extensions for the description of that option.

          That should resolve your specific issue, but there is still a bug to fix in the git plugin because it should have detected this case and correctly realized that the second fetch is required.

          Mark Waite added a comment - matttbe that should be enough. Your freestyle job is missing the clone extension "Honor refspec on initial clone". That option will allow the git plugin to use your defined refspec on the first fetch instead of using its default wildcard refspec. See https://plugins.jenkins.io/git/#clone-extensions for the description of that option. That should resolve your specific issue, but there is still a bug to fix in the git plugin because it should have detected this case and correctly realized that the second fetch is required.

          Your freestyle job is missing the clone extension "Honor refspec on initial clone". That option will allow the git plugin to use your defined refspec on the first fetch instead of using its default wildcard refspec. See https://plugins.jenkins.io/git/#clone-extensions for the description of that option.

          Oh, I just found the option, I missed the Add button, more used to change settings in jenkins-job-builder

          Thank you for the tip!

          Matthieu Baerts added a comment - Your freestyle job is missing the clone extension "Honor refspec on initial clone". That option will allow the git plugin to use your defined refspec on the first fetch instead of using its default wildcard refspec. See https://plugins.jenkins.io/git/#clone-extensions for the description of that option. Oh, I just found the option, I missed the Add button, more used to change settings in jenkins-job-builder Thank you for the tip!

          Mark Waite added a comment -

          matttbe I'm not confident that I'm duplicating the exact conditions that you're seeing, but I suspect that you'll need to continue using "Preserve second fetch on checkout".

          I've created a job that uses a parameterized refspec in the userRemoteConfig. Unfortunately, due to a bug in the git plugin, the parameter placed in the refspec is only expanded by CLI git and not by JGit on the first fetch. It is expanded on the second fetch. That's a bug. Because of that bug, you may need to continue preserving the second fetch in your Jenkins installation.

          You don't see that bug in your current configuration because the first fetch is using the wildcard refspec rather than honoring the refspec that was set in the userRemoteConfig. When you enable "Honor refspec in initial checkout", I expect it will fail because the parameter will not be expanded in the first call to fetch if JGit is the selected implementation.

          Mark Waite added a comment - matttbe I'm not confident that I'm duplicating the exact conditions that you're seeing, but I suspect that you'll need to continue using "Preserve second fetch on checkout". I've created a job that uses a parameterized refspec in the userRemoteConfig. Unfortunately, due to a bug in the git plugin, the parameter placed in the refspec is only expanded by CLI git and not by JGit on the first fetch. It is expanded on the second fetch. That's a bug. Because of that bug, you may need to continue preserving the second fetch in your Jenkins installation. You don't see that bug in your current configuration because the first fetch is using the wildcard refspec rather than honoring the refspec that was set in the userRemoteConfig. When you enable "Honor refspec in initial checkout", I expect it will fail because the parameter will not be expanded in the first call to fetch if JGit is the selected implementation.

          Claes Buckwalter added a comment - - edited

          Hi. We think we are encountering this problem too for our Jenkins instances and will provide details.

          I realize that eliminating a second fetch is a nice performance optimization, but given the risks (building the wrong source code!), it would have been nice if this was a JVM system property to explicitly opt-in. Or alternatively a JVM system property to opt-out by default for all jobs. Now we have to update 1000s of Jenkins jobs (check "Preserve second fetch on checkout") to go back to the previous behavior. We don't want to risk hitting any more edge cases.

          Claes Buckwalter added a comment - - edited Hi. We think we are encountering this problem too for our Jenkins instances and will provide details. I realize that eliminating a second fetch is a nice performance optimization, but given the risks (building the wrong source code!), it would have been nice if this was a JVM system property to explicitly opt-in. Or alternatively a JVM system property to opt-out by default for all jobs. Now we have to update 1000s of Jenkins jobs (check "Preserve second fetch on checkout") to go back to the previous behavior. We don't want to risk hitting any more edge cases.

          Oops. We discovered the old behavior can be preserved for all jobs by default via the global settings of Jenkins, so please ignore my previous comment  Thank you for this escape hatch!

          Claes Buckwalter added a comment - Oops. We discovered the old behavior can be preserved for all jobs by default via the global settings of Jenkins, so please ignore my previous comment  Thank you for this escape hatch!

            Unassigned Unassigned
            matttbe Matthieu Baerts
            Votes:
            2 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: