Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
None
-
git-plugin 2.3.5
git-client-plugin 1.16.1
jenkins 1.580.3
Description
We were running jenkins 1.580.2, git-plugin 2.3.3, git-client-plugin 1.15.0, before upgrading to jenkins 1.580.3, git-plugin 2.3.5, git-client-plugin 1.16.1.
After the upgrade, some our SCM poll jobs started to behave erratically, as shown by following poll log:
Started on Feb 23, 2015 1:33:00 PM Using strategy: Default [poll] Last Built Revision: Revision a74f1d1204a5c892466b52ac68ee6443c1e459d7 (refs/remotes/origin/linux-3.14.y) > /usr/bin/git --version # timeout=10 > /usr/bin/git -c core.askpass=true ls-remote -h git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git # timeout=10 [poll] Latest remote head revision on origin/linux-3.14.y is: a74f1d1204a5c892466b52ac68ee6443c1e459d7 Done. Took 0.69 sec Changes found
So, as the log above shows, "Last Built Revision" and "Latest remote head revision" are the same, and yet "Changes found". And this "Changes found" happened at each poll point (every 5 mins in our case), with build triggering, building that revision, and then doing it over again.
After closer look at by job owners, it turned out that after upgrade, this job started to use optimized workspace-less polling (using git ls-remote), whereas previously, it used workspace-ful polling.
Looking at https://github.com/jenkinsci/git-plugin/blob/master/src/main/java/hudson/plugins/git/GitSCM.java#L590 , turns out that while poll log talks about "Last Built Revision" and "Latest remote head revision", the actual logic for detecting whether change has happened is different. So, "Latest remote head revision" is taken, then some kind of data structure is queried for a build number associated with that change, and if there's no such build, it is triggered. And in our case, apparently exactly this querying part what failed, because otherwise revision values were ok.
Summing up: 2.3.5/1.16.1 appear to have made WS-less polling optimization more aggressive, which I don't really see noted in changelog. That's not bad on its own, but apparently that mode has some bugs. In this particular case, the problems can be avoided by adding an extra stop-gap check of "Last Built Revision" and "Latest remote head revision" being equal - if they're, then there're for sure no changes, regardless of presence of detailed revision-to-build mapping.
I would recommend plugin maintainers to add such a condition, to make plugin more robust.
Thanks!
Hi Mohamed.
As I noted above, the problem I had disappeared as I was messing with things.
what sort of system do you have jenkins running on?
I reinstalled jenkins on my ubuntu system with apt-get install jenkins --reinstall
that may have had a correlation
I can show some things from my current jobs.
Git client plugin version 1.16.1
Git plugin version 2.3.5
Github API plugin 1.63
Github plugin 1.11
Jenkins 1.607
I have a String Parameter on the build called BRANCH
Under Advanced in Source Code Management, the Refspec is
+refs/heads/${BRANCH}:refs/remotes/origin/${BRANCH}
Branches to Build (Branch Specifier) is
${BRANCH}
I'm using Poll SCM
it currently is working, with Git Polling Log giving results like this when no change is needed
Started on Apr 6, 2015 3:55:00 PM
Using strategy: Default
[poll] Last Built Revision: Revision 4a8e9987b4d2ca94a966abe88a590809e3109e4c (origin/cliffc_jenkins)
> git ls-remote -h https://github.com/h2oai/h2o-dev.git # timeout=10
[poll] Latest remote head revision on origin/cliffc_jenkins is: 4a8e9987b4d2ca94a966abe88a590809e3109e4c - already built by 1032
Done. Took 3.5 sec
No changes