Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-36507

git plugin 2.5.2 breaks builds which use a narrow refspec but expect all refs were fetched

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • git-plugin
    • None
    • Ubuntu, Jenkins 1.656, VSphere plugin 2.53, Git plugin 2.5.1 or 2.5.2

      The git plugin is unable to find revisions to build in environments where the current user is not Jenkins.
      This problem was introduced in git 2.5.1, 2 releases, and was not present in the 2.5.0 release.

      On builds where the build user is not Jenkins (the slave is running as a different user), the job aborts due to git errors.
      e.g:
      14:48:47 Started by user jmellor
      14:48:47 Starting limited count build: 1
      14:48:47 [EnvInject] - Loading node environment variables.
      14:48:47 Building remotely on robot-testbuilder-1 (robot-testbuilder) in workspace /var/lib/jenkins/workspace/soc-dashboard_dev_nightly
      14:48:50 Wiping out workspace first.
      14:48:50 Cloning the remote Git repository
      14:48:50 Cloning repository ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git
      14:48:50 > /usr/bin/git init /var/lib/jenkins/workspace/soc-dashboard_dev_nightly # timeout=10
      14:48:50 Fetching upstream changes from ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git
      14:48:50 > /usr/bin/git --version # timeout=10
      14:48:50 > /usr/bin/git -c core.askpass=true fetch --tags --progress ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git refs/heads/develop
      14:49:13 > /usr/bin/git config remote.origin.url ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git # timeout=10
      14:49:14 > /usr/bin/git config --add remote.origin.fetch refs/heads/develop # timeout=10
      14:49:14 > /usr/bin/git config remote.origin.url ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git # timeout=10
      14:49:14 Fetching upstream changes from ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git
      14:49:14 > /usr/bin/git -c core.askpass=true fetch --tags --progress ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git refs/heads/develop
      14:49:14 > /usr/bin/git rev-parse origin/develop^

      {commit} # timeout=10
      14:49:14 > /usr/bin/git rev-parse develop^{commit}

      # timeout=10
      14:49:14 ERROR: Couldn't find any revision to build. Verify the repository and branch configuration for this job.
      14:49:14 Retrying after 10 seconds
      14:49:24 Wiping out workspace first.
      14:49:24 Cloning the remote Git repository
      14:49:24 Cloning repository ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git
      14:49:24 > /usr/bin/git init /var/lib/jenkins/workspace/soc-dashboard_dev_nightly # timeout=10
      14:49:24 Fetching upstream changes from ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git
      14:49:24 > /usr/bin/git --version # timeout=10
      14:49:24 > /usr/bin/git -c core.askpass=true fetch --tags --progress ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git refs/heads/develop
      14:49:46 > /usr/bin/git config remote.origin.url ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git # timeout=10
      14:49:46 > /usr/bin/git config --add remote.origin.fetch refs/heads/develop # timeout=10
      14:49:46 > /usr/bin/git config remote.origin.url ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git # timeout=10
      14:49:46 Fetching upstream changes from ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git
      14:49:46 > /usr/bin/git -c core.askpass=true fetch --tags --progress ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git refs/heads/develop
      14:49:46 > /usr/bin/git rev-parse origin/develop^

      {commit} # timeout=10
      14:49:46 > /usr/bin/git rev-parse develop^{commit}

      # timeout=10
      14:49:46 ERROR: Couldn't find any revision to build. Verify the repository and branch configuration for this job.

      My current workaround is to revert all the way back to git plugin 2.4.4 (which I had laying around), and the fault does not exhibit anymore. This build is also known to have been working for months with the git plugin 2.5.0 release.

        1. gerrit-config.PNG
          gerrit-config.PNG
          14 kB
        2. Parameters.PNG
          Parameters.PNG
          13 kB
        3. slave-config.PNG
          slave-config.PNG
          26 kB
        4. slave-config.PNG
          slave-config.PNG
          26 kB

          [JENKINS-36507] git plugin 2.5.2 breaks builds which use a narrow refspec but expect all refs were fetched

          Mark Waite added a comment - - edited

          I'm not understanding enough of the use case yet (sorry about that). You said, "... only trigger this build when Gerrit sees a change on develop branch..." The refspec you provided (refs/heads/develop) should have only detected changes on the remote branch named "develop". Unfortunately, the refspec you've defined, "refs/heads/develop" seems to mean (with git 2.9.0 on Ubuntu) "fetch the blobs on the remote branch named 'develop', but don't create any local branches to track that remote branch".

          Could you try a different experiment? This should fit your use case for both git plugin 2.4.4 and later. Instead of the refspec "ref/heads/develop", use the refspec "ref/heads/develop:refs/remotes/origin/develop". That should cause a remote tracking branch to be created named "origin/develop" which will then satisfy the request for rev-parse origin/develop.

          Mark Waite added a comment - - edited I'm not understanding enough of the use case yet (sorry about that). You said, "... only trigger this build when Gerrit sees a change on develop branch..." The refspec you provided (refs/heads/develop) should have only detected changes on the remote branch named "develop". Unfortunately, the refspec you've defined, "refs/heads/develop" seems to mean (with git 2.9.0 on Ubuntu) "fetch the blobs on the remote branch named 'develop', but don't create any local branches to track that remote branch". Could you try a different experiment? This should fit your use case for both git plugin 2.4.4 and later. Instead of the refspec "ref/heads/develop", use the refspec " ref/heads/develop:refs/remotes/origin/develop ". That should cause a remote tracking branch to be created named "origin/develop" which will then satisfy the request for rev-parse origin/develop.

          Mark Waite added a comment - - edited

          On your earlier question, "can the fix be reverted?", the answer is certainly yes, but then people (like me) in environments with very large repos need to find other solutions, when the git refspec provides a well-designed solution to limit the amount of information fetched from the remote repository.

          My earlier suggestion

          Another way to retain compatibility with the old "fetch everything initially" behavior would be to require that users who want the fix for JENKINS-31393 must add the "Additional Behaviour" for "Advanced clone behaviours". In that "advanced behaviors" there would be a check box added to "Honor refspec on initial clone". If a large number of users are affected by this same condition, that may be the preferred approach.

          is a revert of the fix, with an option for users who want the new behavior to select that new behavior through an optional choice in the "Additional Behaviours"

          Before I would add the complexity of the additional checkbox, I'd want to know that many more users are affected by it, or that there is no alternative which will work in cases like yours.

          Mark Waite added a comment - - edited On your earlier question, "can the fix be reverted?", the answer is certainly yes, but then people (like me) in environments with very large repos need to find other solutions, when the git refspec provides a well-designed solution to limit the amount of information fetched from the remote repository. My earlier suggestion Another way to retain compatibility with the old "fetch everything initially" behavior would be to require that users who want the fix for JENKINS-31393 must add the "Additional Behaviour" for "Advanced clone behaviours". In that "advanced behaviors" there would be a check box added to "Honor refspec on initial clone". If a large number of users are affected by this same condition, that may be the preferred approach. is a revert of the fix, with an option for users who want the new behavior to select that new behavior through an optional choice in the "Additional Behaviours" Before I would add the complexity of the additional checkbox, I'd want to know that many more users are affected by it, or that there is no alternative which will work in cases like yours.

          John Mellor added a comment -

          Mark requested:
          > Could you try a different experiment?

          I should note that because of the Gerrit plugin, I have to specify the refspec as $GERRIT_REFSPEC parameter to the build, and this is sometimes overloaded by the Gerrit plugin when building a verification build, which would then change $GERRIT_REFSPEC to refs/for/<some branch> where it sees a change. I have a check in my build scripting to differentiate between a Gerrit verification build and a build on the branch that this build is watching, in order to prevent attempts to promote verification builds, etc. The integration between Gerrit and Git plugins is poorly done, but that's a different problem.

          Using Git plugin 2.4.4, that one does not work either:

          12:59:11 > /usr/bin/git init /var/lib/jenkins/workspace/soc-dashboard_dev_nightly # timeout=10
          12:59:11 Fetching upstream changes from ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git
          12:59:11 > /usr/bin/git --version # timeout=10
          12:59:11 > /usr/bin/git -c core.askpass=true fetch --tags --progress ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git +refs/heads/:refs/remotes/origin/
          12:59:34 > /usr/bin/git config remote.origin.url ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git # timeout=10
          12:59:34 > /usr/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
          12:59:34 > /usr/bin/git config remote.origin.url ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git # timeout=10
          12:59:34 Fetching upstream changes from ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git
          12:59:34 > /usr/bin/git -c core.askpass=true fetch --tags --progress ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git ref/heads/develop:refs/remotes/origin/develop
          12:59:34 ERROR: Error fetching remote repo 'origin'

          John Mellor added a comment - Mark requested: > Could you try a different experiment? I should note that because of the Gerrit plugin, I have to specify the refspec as $GERRIT_REFSPEC parameter to the build, and this is sometimes overloaded by the Gerrit plugin when building a verification build, which would then change $GERRIT_REFSPEC to refs/for/<some branch> where it sees a change. I have a check in my build scripting to differentiate between a Gerrit verification build and a build on the branch that this build is watching, in order to prevent attempts to promote verification builds, etc. The integration between Gerrit and Git plugins is poorly done, but that's a different problem. Using Git plugin 2.4.4, that one does not work either: 12:59:11 > /usr/bin/git init /var/lib/jenkins/workspace/soc-dashboard_dev_nightly # timeout=10 12:59:11 Fetching upstream changes from ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git 12:59:11 > /usr/bin/git --version # timeout=10 12:59:11 > /usr/bin/git -c core.askpass=true fetch --tags --progress ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git +refs/heads/ :refs/remotes/origin/ 12:59:34 > /usr/bin/git config remote.origin.url ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git # timeout=10 12:59:34 > /usr/bin/git config --add remote.origin.fetch +refs/heads/ :refs/remotes/origin/ # timeout=10 12:59:34 > /usr/bin/git config remote.origin.url ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git # timeout=10 12:59:34 Fetching upstream changes from ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git 12:59:34 > /usr/bin/git -c core.askpass=true fetch --tags --progress ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git ref/heads/develop:refs/remotes/origin/develop 12:59:34 ERROR: Error fetching remote repo 'origin'

          Mark Waite added a comment -

          Thanks for the clarification. It sounds like the only path forward is to make the default behavior of the plugin match the 2.5.0 (and 2.4.4) refspec handling, with an option to honor the refspec in the initial fetch.

          I will probably be a week or two before that change is ready and released. You'll know it is in the release when this bug is updated to say that the git plugin release included the change.

          Mark Waite added a comment - Thanks for the clarification. It sounds like the only path forward is to make the default behavior of the plugin match the 2.5.0 (and 2.4.4) refspec handling, with an option to honor the refspec in the initial fetch. I will probably be a week or two before that change is ready and released. You'll know it is in the release when this bug is updated to say that the git plugin release included the change.

          John Mellor added a comment -

          Mark asked:
          > I'm not understanding enough of the use case yet

          When integrating with the Gerrit plugin, it is normally necessary to trigger a "verification" build to check that every commit actually builds and unit-tests. The Gerrit verification builds normally build special refs/for/<branch> branches, which may be merged into the <branch> in git at a future date after code review is complete. In this particular case, <build> is "develop", but gerrit will trigger a build for any branch in the refs/for/* tree as required with the introduction of short-lived feature and bugfix branches. These refs/for branches are deleted by Gerrit upon successful merge to the <branch>, and either a pushbutton build or a merge trigger from Gerrit will trigger the <branch> build.

          Make sense now?

          John Mellor added a comment - Mark asked: > I'm not understanding enough of the use case yet When integrating with the Gerrit plugin, it is normally necessary to trigger a "verification" build to check that every commit actually builds and unit-tests. The Gerrit verification builds normally build special refs/for/<branch> branches, which may be merged into the <branch> in git at a future date after code review is complete. In this particular case, <build> is "develop", but gerrit will trigger a build for any branch in the refs/for/* tree as required with the introduction of short-lived feature and bugfix branches. These refs/for branches are deleted by Gerrit upon successful merge to the <branch>, and either a pushbutton build or a merge trigger from Gerrit will trigger the <branch> build. Make sense now?

          Mark Waite added a comment - - edited

          jmellor I was able to submit a pull request which I believe meets the use case you have (retain the 2.5.0 and prior behavior of pulling all references on initial clone) while giving users who need the flexibility to honor the refspec in initial clone. For a few days (until the job output expires from the jenkinsci infrastructure), you can download a prototype build.

          If it meets your use case, then I'll merge it to the master branch and be able to release it as an official fix for this bug.

          After posting that pull request, I performed some more detailed integration testing using a Docker image and many different bug verification jobs. Several of those jobs failed in unexpected ways. Those failures will need to be investigated and resolved before this pull request will be ready to merge to master for a release.

          Mark Waite added a comment - - edited jmellor I was able to submit a pull request which I believe meets the use case you have (retain the 2.5.0 and prior behavior of pulling all references on initial clone) while giving users who need the flexibility to honor the refspec in initial clone. For a few days (until the job output expires from the jenkinsci infrastructure), you can download a prototype build . If it meets your use case, then I'll merge it to the master branch and be able to release it as an official fix for this bug. After posting that pull request, I performed some more detailed integration testing using a Docker image and many different bug verification jobs. Several of those jobs failed in unexpected ways. Those failures will need to be investigated and resolved before this pull request will be ready to merge to master for a release.

          Mark Waite added a comment -

          The pull request has been through some additional interactive tests and has been merged to the master branch. A prototype build is available for you to test.

          Mark Waite added a comment - The pull request has been through some additional interactive tests and has been merged to the master branch. A prototype build is available for you to test.

          John Mellor added a comment -

          I’ve installed the prototype build, and come up with 4 use-case combinations that test the behaviour change.

          I believe that the prototype is working, as all 4 use-cases work as intended. Thumbs up!!!

          John Mellor added a comment - I’ve installed the prototype build, and come up with 4 use-case combinations that test the behaviour change. I believe that the prototype is working, as all 4 use-cases work as intended. Thumbs up!!!

          Mark Waite added a comment -

          alt_jmellor thanks very much.

          Mark Waite added a comment - alt_jmellor thanks very much.

          Mark Waite added a comment -

          Fixed in git plugin 2.5.3 released 30 July 2016

          Mark Waite added a comment - Fixed in git plugin 2.5.3 released 30 July 2016

            Unassigned Unassigned
            alt_jmellor John Mellor
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: