It appears that your build is defined with a narrow refspec ("refs/heads/develop"), but your build then assumes that the entire repository was fetched, even though a narrow refspec was provided. I believe that means your build depends on bug JENKINS-31393 in the git plugin.

JENKINS-31393 was fixed in git plugin 2.5.1. Prior to git plugin 2.5.1, the refspec provided by the user was ignored for the initial fetch. That meant that no matter what refspec was provided by the user, the entire repository was fetched. Users with large repositories didn't like that the refspec they constructed was ignored on initial fetch, since it increased the size of the cloned repository and took longer to clone that repository.

With the fix of JENKINS-31393, the refspec is honored on the initial fetch. One alternative to handle that case is to add an additional refspec to your job definition which will include all the commits. For example, your current refspec seems to be "refs/heads/develop". The new refspec could be

refs/heads/develop +refs/heads/*:refs/remotes/origin/*

I'm not sure how your build could have been working for months with git plugin 2.5.0, since git plugin 2.5.0 released less than a month ago, 19 Jun 2016. Were you using one of the 2.5.0 beta versions prior to 19 Jun 2016, or was it that you were using git plugin 2.4.4 successfully for months?

There is a note in the changelog under 2.5.1 describing the risk of this happening. It says "some users may depend on the old, poor behavior that the plugin fetched all refspecs even though the user had specified a narrower refspec. Those users can delete their refspec or modify it to be as wide as they need". Unfortunately, I didn't see a way to preserve the old behavior (fetch everything) and fix JENKINS-31393.

Another way to retain compatibility with the old "fetch everything initially" behavior would be to require that users who want the fix for JENKINS-31393 must add the "Additional Behaviour" for "Advanced clone behaviours". In that "advanced behaviors" there would be a check box added to "Honor refspec on initial clone". If a large number of users are affected by this same condition, that may be the preferred approach.

I tried your suggested fix, and it breaks the git fetch pretty spectacularly using the git plugin 2.4.4 plugin version:

09:57:18 Building remotely on docker-trusty2 (dockerhost docker-2) in workspace /var/lib/jenkins/workspace/soc-dashboard_develop
09:57:18 Wiping out workspace first.
09:57:18 Cloning the remote Git repository
09:57:18 Cloning repository ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git
09:57:18 > /usr/bin/git init /var/lib/jenkins/workspace/soc-dashboard_develop # timeout=10
09:57:18 Fetching upstream changes from ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git
09:57:18 > /usr/bin/git --version # timeout=10
09:57:18 > /usr/bin/git -c core.askpass=true fetch --tags --progress ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git +refs/heads/:refs/remotes/origin/
09:57:30 > /usr/bin/git config remote.origin.url ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git # timeout=10
09:57:30 > /usr/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
09:57:30 > /usr/bin/git config remote.origin.url ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git # timeout=10
09:57:30 Fetching upstream changes from ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git
09:57:30 > /usr/bin/git -c core.askpass=true fetch --tags --progress ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git refs/heads/develop +refs/heads/:refs/remotes/origin/
09:57:30 ERROR: Error fetching remote repo 'origin'

Screenshots of the parameters, git config and gerrit config settings for this build attached:

I have separate builds for each production branch for this kind of project. What need to happen in this case is to only trigger this build when Gerrit sees a change on develop branch, and not on just any branch.

How should this be configured to thread the needle? Does your suggested fix only work with git plugin 2.5.2, or should it also work with the older and functioning plugin? Can the "fix" be reverted, and the people with poorly-constructed huge git repos maybe suck it up?

I'm not understanding enough of the use case yet (sorry about that). You said, "... only trigger this build when Gerrit sees a change on develop branch..." The refspec you provided (refs/heads/develop) should have only detected changes on the remote branch named "develop". Unfortunately, the refspec you've defined, "refs/heads/develop" seems to mean (with git 2.9.0 on Ubuntu) "fetch the blobs on the remote branch named 'develop', but don't create any local branches to track that remote branch".

Could you try a different experiment? This should fit your use case for both git plugin 2.4.4 and later. Instead of the refspec "ref/heads/develop", use the refspec "ref/heads/develop:refs/remotes/origin/develop". That should cause a remote tracking branch to be created named "origin/develop" which will then satisfy the request for rev-parse origin/develop.

On your earlier question, "can the fix be reverted?", the answer is certainly yes, but then people (like me) in environments with very large repos need to find other solutions, when the git refspec provides a well-designed solution to limit the amount of information fetched from the remote repository.

My earlier suggestion

Another way to retain compatibility with the old "fetch everything initially" behavior would be to require that users who want the fix for JENKINS-31393 must add the "Additional Behaviour" for "Advanced clone behaviours". In that "advanced behaviors" there would be a check box added to "Honor refspec on initial clone". If a large number of users are affected by this same condition, that may be the preferred approach.

is a revert of the fix, with an option for users who want the new behavior to select that new behavior through an optional choice in the "Additional Behaviours"

Before I would add the complexity of the additional checkbox, I'd want to know that many more users are affected by it, or that there is no alternative which will work in cases like yours.

Mark requested:
> Could you try a different experiment?

I should note that because of the Gerrit plugin, I have to specify the refspec as $GERRIT_REFSPEC parameter to the build, and this is sometimes overloaded by the Gerrit plugin when building a verification build, which would then change $GERRIT_REFSPEC to refs/for/<some branch> where it sees a change. I have a check in my build scripting to differentiate between a Gerrit verification build and a build on the branch that this build is watching, in order to prevent attempts to promote verification builds, etc. The integration between Gerrit and Git plugins is poorly done, but that's a different problem.

Using Git plugin 2.4.4, that one does not work either:

12:59:11 > /usr/bin/git init /var/lib/jenkins/workspace/soc-dashboard_dev_nightly # timeout=10
12:59:11 Fetching upstream changes from ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git
12:59:11 > /usr/bin/git --version # timeout=10
12:59:11 > /usr/bin/git -c core.askpass=true fetch --tags --progress ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git +refs/heads/:refs/remotes/origin/
12:59:34 > /usr/bin/git config remote.origin.url ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git # timeout=10
12:59:34 > /usr/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
12:59:34 > /usr/bin/git config remote.origin.url ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git # timeout=10
12:59:34 Fetching upstream changes from ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git
12:59:34 > /usr/bin/git -c core.askpass=true fetch --tags --progress ssh://jmellor@gerrit.internal:29418/socdash/soc-dashboard.git ref/heads/develop:refs/remotes/origin/develop
12:59:34 ERROR: Error fetching remote repo 'origin'

Thanks for the clarification. It sounds like the only path forward is to make the default behavior of the plugin match the 2.5.0 (and 2.4.4) refspec handling, with an option to honor the refspec in the initial fetch.

I will probably be a week or two before that change is ready and released. You'll know it is in the release when this bug is updated to say that the git plugin release included the change.