Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
None
-
git-plugin 2.3.5
git-client-plugin 1.16.1
jenkins 1.580.3
Description
We were running jenkins 1.580.2, git-plugin 2.3.3, git-client-plugin 1.15.0, before upgrading to jenkins 1.580.3, git-plugin 2.3.5, git-client-plugin 1.16.1.
After the upgrade, some our SCM poll jobs started to behave erratically, as shown by following poll log:
Started on Feb 23, 2015 1:33:00 PM Using strategy: Default [poll] Last Built Revision: Revision a74f1d1204a5c892466b52ac68ee6443c1e459d7 (refs/remotes/origin/linux-3.14.y) > /usr/bin/git --version # timeout=10 > /usr/bin/git -c core.askpass=true ls-remote -h git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git # timeout=10 [poll] Latest remote head revision on origin/linux-3.14.y is: a74f1d1204a5c892466b52ac68ee6443c1e459d7 Done. Took 0.69 sec Changes found
So, as the log above shows, "Last Built Revision" and "Latest remote head revision" are the same, and yet "Changes found". And this "Changes found" happened at each poll point (every 5 mins in our case), with build triggering, building that revision, and then doing it over again.
After closer look at by job owners, it turned out that after upgrade, this job started to use optimized workspace-less polling (using git ls-remote), whereas previously, it used workspace-ful polling.
Looking at https://github.com/jenkinsci/git-plugin/blob/master/src/main/java/hudson/plugins/git/GitSCM.java#L590 , turns out that while poll log talks about "Last Built Revision" and "Latest remote head revision", the actual logic for detecting whether change has happened is different. So, "Latest remote head revision" is taken, then some kind of data structure is queried for a build number associated with that change, and if there's no such build, it is triggered. And in our case, apparently exactly this querying part what failed, because otherwise revision values were ok.
Summing up: 2.3.5/1.16.1 appear to have made WS-less polling optimization more aggressive, which I don't really see noted in changelog. That's not bad on its own, but apparently that mode has some bugs. In this particular case, the problems can be avoided by adding an extra stop-gap check of "Last Built Revision" and "Latest remote head revision" being equal - if they're, then there're for sure no changes, regardless of presence of detailed revision-to-build mapping.
I would recommend plugin maintainers to add such a condition, to make plugin more robust.
Thanks!
Attachments
Activity
The git plugin 2.3.6 pre-release and git client plugin 1.18.0 pre-release are being tested in hopes of releasing new versions before the end of June. If you're willing to assist with the testing, please download and install a pre-release build of the git client plugin and the git plugin. Problems detected in the pre-release should be e-mailed to MarkEWaite and ndeloof.
I wrote some test ideas if you would like suggestions of areas that need testing. The git plugin supports many different use cases and its automated tests only evaluate a very few of those use cases.
Mohamed Galal El-Din
added a comment - I've reported JENKINS-27769 to track my issue. Thanks Kevin. Mohamed Galal El-Din
added a comment - Hi Kevin,
I'm using Ubuntu 14.04. And originally installed Jenkins using the apt-get method as explained here
My Jenkins server version is 1.607. And my git repository is hosted on BitBucket.
My issue seems to be a different than this one. I'll file a separate JIRA.
Mohamed Galal El-Din
added a comment - Hi Kevin,
I'm using Ubuntu 14.04. And originally installed Jenkins using the apt-get method as explained here
My Jenkins server version is 1.607. And my git repository is hosted on BitBucket.
My issue seems to be a different than this one. I'll file a separate JIRA. 
Kevin Normoyle
added a comment - Hi Mohamed.
As I noted above, the problem I had disappeared as I was messing with things.
what sort of system do you have jenkins running on?
I reinstalled jenkins on my ubuntu system with apt-get install jenkins --reinstall
that may have had a correlation
I can show some things from my current jobs.
Git client plugin version 1.16.1
Git plugin version 2.3.5
Github API plugin 1.63
Github plugin 1.11
Jenkins 1.607
I have a String Parameter on the build called BRANCH
Under Advanced in Source Code Management, the Refspec is
+refs/heads/${BRANCH}:refs/remotes/origin/${BRANCH}
Branches to Build (Branch Specifier) is
${BRANCH}
I'm using Poll SCM
it currently is working, with Git Polling Log giving results like this when no change is needed
Started on Apr 6, 2015 3:55:00 PM
Using strategy: Default
[poll] Last Built Revision: Revision 4a8e9987b4d2ca94a966abe88a590809e3109e4c (origin/cliffc_jenkins)
> git ls-remote -h https://github.com/h2oai/h2o-dev.git # timeout=10
[poll] Latest remote head revision on origin/cliffc_jenkins is: 4a8e9987b4d2ca94a966abe88a590809e3109e4c - already built by 1032
Done. Took 3.5 sec
No changes
Kevin Normoyle
added a comment - Hi Mohamed.
As I noted above, the problem I had disappeared as I was messing with things.
what sort of system do you have jenkins running on?
I reinstalled jenkins on my ubuntu system with apt-get install jenkins --reinstall
that may have had a correlation
I can show some things from my current jobs.
Git client plugin version 1.16.1
Git plugin version 2.3.5
Github API plugin 1.63
Github plugin 1.11
Jenkins 1.607
I have a String Parameter on the build called BRANCH
Under Advanced in Source Code Management, the Refspec is
+refs/heads/${BRANCH}:refs/remotes/origin/${BRANCH}
Branches to Build (Branch Specifier) is
${BRANCH}
I'm using Poll SCM
it currently is working, with Git Polling Log giving results like this when no change is needed
Started on Apr 6, 2015 3:55:00 PM
Using strategy: Default
[poll] Last Built Revision: Revision 4a8e9987b4d2ca94a966abe88a590809e3109e4c (origin/cliffc_jenkins)
> git ls-remote -h https://github.com/h2oai/h2o-dev.git # timeout=10
[poll] Latest remote head revision on origin/cliffc_jenkins is: 4a8e9987b4d2ca94a966abe88a590809e3109e4c - already built by 1032
Done. Took 3.5 sec
No changes
Fix included in git plugin 2.4.0 released 18 July 2015