-
Bug
-
Resolution: Fixed
-
Major
-
None
-
git-plugin 2.3.5
git-client-plugin 1.16.1
jenkins 1.580.3
We were running jenkins 1.580.2, git-plugin 2.3.3, git-client-plugin 1.15.0, before upgrading to jenkins 1.580.3, git-plugin 2.3.5, git-client-plugin 1.16.1.
After the upgrade, some our SCM poll jobs started to behave erratically, as shown by following poll log:
Started on Feb 23, 2015 1:33:00 PM Using strategy: Default [poll] Last Built Revision: Revision a74f1d1204a5c892466b52ac68ee6443c1e459d7 (refs/remotes/origin/linux-3.14.y) > /usr/bin/git --version # timeout=10 > /usr/bin/git -c core.askpass=true ls-remote -h git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git # timeout=10 [poll] Latest remote head revision on origin/linux-3.14.y is: a74f1d1204a5c892466b52ac68ee6443c1e459d7 Done. Took 0.69 sec Changes found
So, as the log above shows, "Last Built Revision" and "Latest remote head revision" are the same, and yet "Changes found". And this "Changes found" happened at each poll point (every 5 mins in our case), with build triggering, building that revision, and then doing it over again.
After closer look at by job owners, it turned out that after upgrade, this job started to use optimized workspace-less polling (using git ls-remote), whereas previously, it used workspace-ful polling.
Looking at https://github.com/jenkinsci/git-plugin/blob/master/src/main/java/hudson/plugins/git/GitSCM.java#L590 , turns out that while poll log talks about "Last Built Revision" and "Latest remote head revision", the actual logic for detecting whether change has happened is different. So, "Latest remote head revision" is taken, then some kind of data structure is queried for a build number associated with that change, and if there's no such build, it is triggered. And in our case, apparently exactly this querying part what failed, because otherwise revision values were ok.
Summing up: 2.3.5/1.16.1 appear to have made WS-less polling optimization more aggressive, which I don't really see noted in changelog. That's not bad on its own, but apparently that mode has some bugs. In this particular case, the problems can be avoided by adding an extra stop-gap check of "Last Built Revision" and "Latest remote head revision" being equal - if they're, then there're for sure no changes, regardless of presence of detailed revision-to-build mapping.
I would recommend plugin maintainers to add such a condition, to make plugin more robust.
Thanks!
I'm totally confused now. I may not be able to reproduce the problem
I went back and upgraded all my jenkins servers to git plugin 2.3.5
and did a
apt-get install jenkins --reinstall
to get the latest 1.604 jenksin
and now they seem to all be polling correctly and not detecting changes (because the branch didn't change)
They all have that "already built by .." message ..that says what the prior build was, which is good
Git Polling Log
Started on Mar 16, 2015 6:20:00 AM
Using strategy: Default
[poll] Last Built Revision: Revision 1dad640e7ec50cea3965522efa8c797a3ea9b3bc (origin/seb_jenkins)
> git ls-remote -h https://github.com/h2oai/h2o-dev.git # timeout=10
[poll] Latest remote head revision on origin/seb_jenkins is: 1dad640e7ec50cea3965522efa8c797a3ea9b3bc - already built by 44
Done. Took 1.7 sec
No changes