-
Bug
-
Resolution: Fixed
-
Major
-
None
-
git-plugin 2.3.5
git-client-plugin 1.16.1
jenkins 1.580.3
We were running jenkins 1.580.2, git-plugin 2.3.3, git-client-plugin 1.15.0, before upgrading to jenkins 1.580.3, git-plugin 2.3.5, git-client-plugin 1.16.1.
After the upgrade, some our SCM poll jobs started to behave erratically, as shown by following poll log:
Started on Feb 23, 2015 1:33:00 PM Using strategy: Default [poll] Last Built Revision: Revision a74f1d1204a5c892466b52ac68ee6443c1e459d7 (refs/remotes/origin/linux-3.14.y) > /usr/bin/git --version # timeout=10 > /usr/bin/git -c core.askpass=true ls-remote -h git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git # timeout=10 [poll] Latest remote head revision on origin/linux-3.14.y is: a74f1d1204a5c892466b52ac68ee6443c1e459d7 Done. Took 0.69 sec Changes found
So, as the log above shows, "Last Built Revision" and "Latest remote head revision" are the same, and yet "Changes found". And this "Changes found" happened at each poll point (every 5 mins in our case), with build triggering, building that revision, and then doing it over again.
After closer look at by job owners, it turned out that after upgrade, this job started to use optimized workspace-less polling (using git ls-remote), whereas previously, it used workspace-ful polling.
Looking at https://github.com/jenkinsci/git-plugin/blob/master/src/main/java/hudson/plugins/git/GitSCM.java#L590 , turns out that while poll log talks about "Last Built Revision" and "Latest remote head revision", the actual logic for detecting whether change has happened is different. So, "Latest remote head revision" is taken, then some kind of data structure is queried for a build number associated with that change, and if there's no such build, it is triggered. And in our case, apparently exactly this querying part what failed, because otherwise revision values were ok.
Summing up: 2.3.5/1.16.1 appear to have made WS-less polling optimization more aggressive, which I don't really see noted in changelog. That's not bad on its own, but apparently that mode has some bugs. In this particular case, the problems can be avoided by adding an extra stop-gap check of "Last Built Revision" and "Latest remote head revision" being equal - if they're, then there're for sure no changes, regardless of presence of detailed revision-to-build mapping.
I would recommend plugin maintainers to add such a condition, to make plugin more robust.
Thanks!
[JENKINS-27093] Spurious gits scm poll change detection
Paul Sokolovsky
created issue -
Code changed in jenkins
User: Nicolas De Loof
Path:
src/main/java/hudson/plugins/git/util/BuildData.java
http://jenkins-ci.org/commit/git-plugin/aa31af020633ef2d414e3e30d3c4bc3137d0bc01
Log:
JENKINS-27093 quick check for lastBuilt.revision
Code changed in jenkins
User: Nicolas De Loof
Path:
src/main/java/hudson/plugins/git/util/BuildData.java
http://jenkins-ci.org/commit/git-plugin/cfa9180ad4d9c71fd0b06fb1bae47a9d14489957
Log:
JENKINS-27093 quick check for lastBuilt.revision
Hi, Paul, could you provide two consecutive build.xml files (you can cut any private info) that were triggered and the result of manual run for git command from polling log?
Paul Sokolovsky
added a comment - Thanks for the replies and commits already made. So, our situation is that we have pretty busy production Jenkins system, so there's only limited experimentation can be done there, and in limited time I had to look into this issue, I traced it just to the source line in the original description above. This is also 1st time I looked into SCM polling, so I don't know how related data is stored and other details.
But job we had issues to is actually public: https://ci.linaro.org/job/trigger-linux-ltsi/ . As you can see (note: builds in question may expire in a week or two), from build #132 Feb 23, 2015 11:33:10 AM to build #161 Feb 23, 2015 1:58:09 PM , build triggered every 5 mins, which is SCM polling period for that job. But typical poll log looks like: https://ci.linaro.org/job/trigger-linux-ltsi/160/pollingLog/ (content quoted above in the original description).
Kanstantsin: The repo is public, running git command from poll log, I get the expected output (I also thought that maybe there's stray space or something gets parsed, but I confirmed it all looks ok). Will try to look up those build.xml's as a next step.
$ /usr/bin/git -c core.askpass=true ls-remote -h git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git 8e63197ffe7750c94c8ea9d159ce3e46a76bfcf2 refs/heads/linux-2.6.11.y d04a37911968d919fa842ad40fa9e9ff1dd10904 refs/heads/linux-2.6.12.y 816e9c6c226227c4862b2067aace0f450cc92635 refs/heads/linux-2.6.13.y 789f444285aedfb04af7aa3748aa52e99ac4bd8f refs/heads/linux-2.6.14.y f70602f4f6248735a02c61a1323c9151a33a3775 refs/heads/linux-2.6.15.y 6b0daf99dd2392b024bdca05530e4e761bc3cdae refs/heads/linux-2.6.16.y 78ace17e51d4968ed2355e8f708d233d1cc37f6d refs/heads/linux-2.6.17.y 299a2479bca6211f845158761920ec480f35a229 refs/heads/linux-2.6.18.y 09780ab3b26507776671900e0ed7920f297498ed refs/heads/linux-2.6.19.y f3815da6b4fd508cc3574399248e2e15cb8a617f refs/heads/linux-2.6.20.y a31a9035702124423c3aa5aa848937f165753a4f refs/heads/linux-2.6.21.y 37579d1574f6c18f1f648201c6b0850ac94094cd refs/heads/linux-2.6.22.y 6531868a73a6c91bf0e3e60ded7d1440ee24dfa8 refs/heads/linux-2.6.23.y 928bb8c418b5f9e96dbccc8d7eafb6635ae81548 refs/heads/linux-2.6.24.y 00935daeb04cd54a67b66c9e3babc23389251a98 refs/heads/linux-2.6.25.y 63e0e67b17dc233f93f709610971bbfadc97f75e refs/heads/linux-2.6.26.y bc4e1a77b06519a01e7aed1125695598e27ddeb2 refs/heads/linux-2.6.27.y 5861c853a3f529b9c6a338dd7c4a7afec397ea7a refs/heads/linux-2.6.28.y 12010107aaf417503b7e413d84f2554080aebfe2 refs/heads/linux-2.6.29.y 0d0675cf44c85bd3c0d891845aa02f9249cd7c68 refs/heads/linux-2.6.30.y a389e98d2c6e1900f035befe215f541436bcb0b2 refs/heads/linux-2.6.31.y 3bd0d1ad14c3566107e391732e4df658b005ad67 refs/heads/linux-2.6.32.y 86a705267a2a502a3d62ef0797e449677b25835f refs/heads/linux-2.6.33.y 5878e067ecac4bd2320e933ec485c01190a5b881 refs/heads/linux-2.6.34.y 675f7660ffb0e1880011f6b3c4f9ac241491e3cd refs/heads/linux-2.6.35.y 69ad303ab8321656d6144d13b2444a5595bb6581 refs/heads/linux-2.6.36.y e396c9d8699c95d52b2abcc2d4d5f9616e839734 refs/heads/linux-2.6.37.y 4b7a6d2528bfb625cc359d89ac16439b0ec744ea refs/heads/linux-2.6.38.y ea0dc0dc1c1dca25e50384e300a528db57ee7de5 refs/heads/linux-2.6.39.y 5dba9ddd98cbc7ad319d687887981a0ea0062c75 refs/heads/linux-3.0.y 9bb1282f6a7754955c18be912fbc2b55d133f1b9 refs/heads/linux-3.1.y 5cfc71ce138e79ceb6250f78137dd05ba52e9d34 refs/heads/linux-3.10.y 5ee54f38171b9b3541c5e9cf9c3a9e53455fd8b4 refs/heads/linux-3.11.y 22ccf8f1a5450ac5a6bc2bb519699838017ce1ea refs/heads/linux-3.12.y 2d20120bba8475c963a8d28dd0ffa13637fa3ad7 refs/heads/linux-3.13.y a74f1d1204a5c892466b52ac68ee6443c1e459d7 refs/heads/linux-3.14.y f35b5e46feabab668a44df5b33f3558629f94dfc refs/heads/linux-3.15.y d0335e4feea0d3f7a8af3116c5dc166239da7521 refs/heads/linux-3.16.y bc15d4627aa8f562a1c5ec9d84076b8db25bab31 refs/heads/linux-3.17.y a17f9bf1f7cd3412b9920577a7c0ec34cb81b233 refs/heads/linux-3.18.y bfa76d49576599a4b9f9b7a71f23d73d6dcff735 refs/heads/linux-3.19.y fd623507bdcee1f7a387ae86adb7a66b431dd056 refs/heads/linux-3.2.y 845720650c557a75262b629b0bc228fffcf64638 refs/heads/linux-3.3.y 28895317f9a7d726cd13fc9b5447cb5dcb5cd22c refs/heads/linux-3.4.y f2b152564afdf9c9917c17d1c41c1082c82067bd refs/heads/linux-3.5.y b2824f4e0990716407b0c0e7acee75bb6353febf refs/heads/linux-3.6.y 356d8c6fb2a7cf49e836742738a8b9a47e77cfea refs/heads/linux-3.7.y dbf932a9b316d5b29b3e220e5a30e7a165ad2992 refs/heads/linux-3.8.y 896f5009ed1fbaec43f360c4ebf022639cd61d5f refs/heads/linux-3.9.y c517d838eb7d07bbe9507871fab3931deccff539 refs/heads/master
Paul Sokolovsky
added a comment - Thanks for the replies and commits already made. So, our situation is that we have pretty busy production Jenkins system, so there's only limited experimentation can be done there, and in limited time I had to look into this issue, I traced it just to the source line in the original description above. This is also 1st time I looked into SCM polling, so I don't know how related data is stored and other details.
But job we had issues to is actually public: https://ci.linaro.org/job/trigger-linux-ltsi/ . As you can see (note: builds in question may expire in a week or two), from build #132 Feb 23, 2015 11:33:10 AM to build #161 Feb 23, 2015 1:58:09 PM , build triggered every 5 mins, which is SCM polling period for that job. But typical poll log looks like: https://ci.linaro.org/job/trigger-linux-ltsi/160/pollingLog/ (content quoted above in the original description).
Kanstantsin: The repo is public, running git command from poll log, I get the expected output (I also thought that maybe there's stray space or something gets parsed, but I confirmed it all looks ok). Will try to look up those build.xml's as a next step.
$ /usr/bin/git -c core.askpass= true ls-remote -h git: //git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
8e63197ffe7750c94c8ea9d159ce3e46a76bfcf2 refs/heads/linux-2.6.11.y
d04a37911968d919fa842ad40fa9e9ff1dd10904 refs/heads/linux-2.6.12.y
816e9c6c226227c4862b2067aace0f450cc92635 refs/heads/linux-2.6.13.y
789f444285aedfb04af7aa3748aa52e99ac4bd8f refs/heads/linux-2.6.14.y
f70602f4f6248735a02c61a1323c9151a33a3775 refs/heads/linux-2.6.15.y
6b0daf99dd2392b024bdca05530e4e761bc3cdae refs/heads/linux-2.6.16.y
78ace17e51d4968ed2355e8f708d233d1cc37f6d refs/heads/linux-2.6.17.y
299a2479bca6211f845158761920ec480f35a229 refs/heads/linux-2.6.18.y
09780ab3b26507776671900e0ed7920f297498ed refs/heads/linux-2.6.19.y
f3815da6b4fd508cc3574399248e2e15cb8a617f refs/heads/linux-2.6.20.y
a31a9035702124423c3aa5aa848937f165753a4f refs/heads/linux-2.6.21.y
37579d1574f6c18f1f648201c6b0850ac94094cd refs/heads/linux-2.6.22.y
6531868a73a6c91bf0e3e60ded7d1440ee24dfa8 refs/heads/linux-2.6.23.y
928bb8c418b5f9e96dbccc8d7eafb6635ae81548 refs/heads/linux-2.6.24.y
00935daeb04cd54a67b66c9e3babc23389251a98 refs/heads/linux-2.6.25.y
63e0e67b17dc233f93f709610971bbfadc97f75e refs/heads/linux-2.6.26.y
bc4e1a77b06519a01e7aed1125695598e27ddeb2 refs/heads/linux-2.6.27.y
5861c853a3f529b9c6a338dd7c4a7afec397ea7a refs/heads/linux-2.6.28.y
12010107aaf417503b7e413d84f2554080aebfe2 refs/heads/linux-2.6.29.y
0d0675cf44c85bd3c0d891845aa02f9249cd7c68 refs/heads/linux-2.6.30.y
a389e98d2c6e1900f035befe215f541436bcb0b2 refs/heads/linux-2.6.31.y
3bd0d1ad14c3566107e391732e4df658b005ad67 refs/heads/linux-2.6.32.y
86a705267a2a502a3d62ef0797e449677b25835f refs/heads/linux-2.6.33.y
5878e067ecac4bd2320e933ec485c01190a5b881 refs/heads/linux-2.6.34.y
675f7660ffb0e1880011f6b3c4f9ac241491e3cd refs/heads/linux-2.6.35.y
69ad303ab8321656d6144d13b2444a5595bb6581 refs/heads/linux-2.6.36.y
e396c9d8699c95d52b2abcc2d4d5f9616e839734 refs/heads/linux-2.6.37.y
4b7a6d2528bfb625cc359d89ac16439b0ec744ea refs/heads/linux-2.6.38.y
ea0dc0dc1c1dca25e50384e300a528db57ee7de5 refs/heads/linux-2.6.39.y
5dba9ddd98cbc7ad319d687887981a0ea0062c75 refs/heads/linux-3.0.y
9bb1282f6a7754955c18be912fbc2b55d133f1b9 refs/heads/linux-3.1.y
5cfc71ce138e79ceb6250f78137dd05ba52e9d34 refs/heads/linux-3.10.y
5ee54f38171b9b3541c5e9cf9c3a9e53455fd8b4 refs/heads/linux-3.11.y
22ccf8f1a5450ac5a6bc2bb519699838017ce1ea refs/heads/linux-3.12.y
2d20120bba8475c963a8d28dd0ffa13637fa3ad7 refs/heads/linux-3.13.y
a74f1d1204a5c892466b52ac68ee6443c1e459d7 refs/heads/linux-3.14.y
f35b5e46feabab668a44df5b33f3558629f94dfc refs/heads/linux-3.15.y
d0335e4feea0d3f7a8af3116c5dc166239da7521 refs/heads/linux-3.16.y
bc15d4627aa8f562a1c5ec9d84076b8db25bab31 refs/heads/linux-3.17.y
a17f9bf1f7cd3412b9920577a7c0ec34cb81b233 refs/heads/linux-3.18.y
bfa76d49576599a4b9f9b7a71f23d73d6dcff735 refs/heads/linux-3.19.y
fd623507bdcee1f7a387ae86adb7a66b431dd056 refs/heads/linux-3.2.y
845720650c557a75262b629b0bc228fffcf64638 refs/heads/linux-3.3.y
28895317f9a7d726cd13fc9b5447cb5dcb5cd22c refs/heads/linux-3.4.y
f2b152564afdf9c9917c17d1c41c1082c82067bd refs/heads/linux-3.5.y
b2824f4e0990716407b0c0e7acee75bb6353febf refs/heads/linux-3.6.y
356d8c6fb2a7cf49e836742738a8b9a47e77cfea refs/heads/linux-3.7.y
dbf932a9b316d5b29b3e220e5a30e7a165ad2992 refs/heads/linux-3.8.y
896f5009ed1fbaec43f360c4ebf022639cd61d5f refs/heads/linux-3.9.y
c517d838eb7d07bbe9507871fab3931deccff539 refs/heads/master
Paul Sokolovsky
added a comment - build.xml's from builds #159, #160 of the job above are attached. Paul Sokolovsky
made changes -
| Attachment | New: 159-build.xml [ 28658 ] | |
| Attachment | New: 160-build.xml [ 28659 ] |
| Environment |
New:
git-plugin 2.3.5 git-client-plugin 1.16.1 jenkins 1.580.3 |
|
| Summary | Original: Spurious gits scm poll change detection with git-plugin 2.3.5 & git-client-plugin 1.16.1 | New: Spurious gits scm poll change detection |
This looks like regression by some of ndeloof commits for 2.3.5. Will you be able to deploy patched git plugin with additional logging?
For sure such a check can be introduced, but would be interesting to understand why the last build based on commit sha1 isn't identified