• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • None
    • git-plugin 2.3.5
      git-client-plugin 1.16.1
      jenkins 1.580.3

      We were running jenkins 1.580.2, git-plugin 2.3.3, git-client-plugin 1.15.0, before upgrading to jenkins 1.580.3, git-plugin 2.3.5, git-client-plugin 1.16.1.

      After the upgrade, some our SCM poll jobs started to behave erratically, as shown by following poll log:

      Started on Feb 23, 2015 1:33:00 PM
      Using strategy: Default
      [poll] Last Built Revision: Revision a74f1d1204a5c892466b52ac68ee6443c1e459d7 (refs/remotes/origin/linux-3.14.y)
       > /usr/bin/git --version # timeout=10
       > /usr/bin/git -c core.askpass=true ls-remote -h git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git # timeout=10
      [poll] Latest remote head revision on origin/linux-3.14.y is: a74f1d1204a5c892466b52ac68ee6443c1e459d7
      Done. Took 0.69 sec
      Changes found
      

      So, as the log above shows, "Last Built Revision" and "Latest remote head revision" are the same, and yet "Changes found". And this "Changes found" happened at each poll point (every 5 mins in our case), with build triggering, building that revision, and then doing it over again.

      After closer look at by job owners, it turned out that after upgrade, this job started to use optimized workspace-less polling (using git ls-remote), whereas previously, it used workspace-ful polling.

      Looking at https://github.com/jenkinsci/git-plugin/blob/master/src/main/java/hudson/plugins/git/GitSCM.java#L590 , turns out that while poll log talks about "Last Built Revision" and "Latest remote head revision", the actual logic for detecting whether change has happened is different. So, "Latest remote head revision" is taken, then some kind of data structure is queried for a build number associated with that change, and if there's no such build, it is triggered. And in our case, apparently exactly this querying part what failed, because otherwise revision values were ok.

      Summing up: 2.3.5/1.16.1 appear to have made WS-less polling optimization more aggressive, which I don't really see noted in changelog. That's not bad on its own, but apparently that mode has some bugs. In this particular case, the problems can be avoided by adding an extra stop-gap check of "Last Built Revision" and "Latest remote head revision" being equal - if they're, then there're for sure no changes, regardless of presence of detailed revision-to-build mapping.

      I would recommend plugin maintainers to add such a condition, to make plugin more robust.

      Thanks!

          [JENKINS-27093] Spurious gits scm poll change detection

          I'm totally confused now. I may not be able to reproduce the problem

          I went back and upgraded all my jenkins servers to git plugin 2.3.5
          and did a
          apt-get install jenkins --reinstall
          to get the latest 1.604 jenksin

          and now they seem to all be polling correctly and not detecting changes (because the branch didn't change)

          They all have that "already built by .." message ..that says what the prior build was, which is good

          Git Polling Log

          Started on Mar 16, 2015 6:20:00 AM
          Using strategy: Default
          [poll] Last Built Revision: Revision 1dad640e7ec50cea3965522efa8c797a3ea9b3bc (origin/seb_jenkins)
          > git ls-remote -h https://github.com/h2oai/h2o-dev.git # timeout=10
          [poll] Latest remote head revision on origin/seb_jenkins is: 1dad640e7ec50cea3965522efa8c797a3ea9b3bc - already built by 44
          Done. Took 1.7 sec
          No changes

          Kevin Normoyle added a comment - I'm totally confused now. I may not be able to reproduce the problem I went back and upgraded all my jenkins servers to git plugin 2.3.5 and did a apt-get install jenkins --reinstall to get the latest 1.604 jenksin and now they seem to all be polling correctly and not detecting changes (because the branch didn't change) They all have that "already built by .." message ..that says what the prior build was, which is good Git Polling Log Started on Mar 16, 2015 6:20:00 AM Using strategy: Default [poll] Last Built Revision: Revision 1dad640e7ec50cea3965522efa8c797a3ea9b3bc (origin/seb_jenkins) > git ls-remote -h https://github.com/h2oai/h2o-dev.git # timeout=10 [poll] Latest remote head revision on origin/seb_jenkins is: 1dad640e7ec50cea3965522efa8c797a3ea9b3bc - already built by 44 Done. Took 1.7 sec No changes

          > I'm totally confused now. I may not be able to reproduce the problem

          Well, I personally got an impression that the issues we faced might have been related to the fact that between plugin versions upgrades, polling methods changed, and "new" method may have lacked some metadata, not generated by previous method. Something like that. Anyway, I suggested a short-circuit fix for this issue, and it was applied, so just need to wait for 2.36 release (we also have busy production systems, so building and testing debug builds is complicated). We're on Jenkins LTS releases btw (still on 1.580.3, waiting for maintenance window to upgrade to 1.596).

          Paul Sokolovsky added a comment - > I'm totally confused now. I may not be able to reproduce the problem Well, I personally got an impression that the issues we faced might have been related to the fact that between plugin versions upgrades, polling methods changed, and "new" method may have lacked some metadata, not generated by previous method. Something like that. Anyway, I suggested a short-circuit fix for this issue, and it was applied, so just need to wait for 2.36 release (we also have busy production systems, so building and testing debug builds is complicated). We're on Jenkins LTS releases btw (still on 1.580.3, waiting for maintenance window to upgrade to 1.596).

          Kevin Normoyle added a comment - - edited

          Hi thanks.
          Well, a couple of days has gone by with multiple jenkins using similar jobs (polling with parameterized branches). 6 jenkins masters each with 8 jobs. total of 12 branches. Seems to be working as desired with single branches resolved correctly and no unnecessary building due to changes to other branches.

          I've not tried to recreate the initial problem. It was weird since it affected every jenkins master I had created, and then it started working right as I noted above, after messing with reinstall of jenkins and the git-plugin. (I think that was what led it to working..although there could have been some racy thing. I was doing combined polling plus scheduled build. I don't think there was any race effect going on, but if jenkins was looking at "last build" state somehow, you never know.

          But looking at other open jiras in this area:

          I found another open jira in this area that is interesting. I use a parameter ${BRANCH} to specify the branch. While there have been issues in the past with that, I notice in this jira a very odd behavior, where the behavior depends on the branch being different in the last build.

          I was wondering if I copied jenkins setups with jobs pointing to master or a different branch, then modified the branch parameter in the job. (well yeah, I know that's how I setup all my different jenkins masters on different machines)

          I'm wondering if the issue mentioned in this jira could have affected things..i.e. jenkins deciding that a branch variable wasn't what it was supposed to be.

          BRANCH_TO_BUILD is equivalent to my use of BRANCH (${BRANCH}) in the jobs I was having problems with above.

          quoting from this link
          https://issues.jenkins-ci.org/browse/JENKINS-27349
          Git SCM-polling uses wrong parameter values for a parametrized branchspec

          to reproduce:
          A parametrized build with parameter BRANCH_TO_BUILD, with default value "branch1"
          Git SCM with branch specifier "${BRANCH_TO_BUILD}"
          SCM polling enabled
          Previous manual job launch with BRANCH_TO_BUILD=branch2

          Expected:

          When polling, BRANCH_TO_BUILD is set to branch1 (default value), so only updates to branch1 trigger a build.

          Actual:

          When polling, BRANCH_TO_BUILD is set to branch2 (value from last execution).

          also:
          https://issues.jenkins-ci.org/browse/JENKINS-27327
          When specifying the branch to poll as part of a parameterized build, git-plugin uses last polled branch instead

          another related one, but not what I was seeing:
          https://issues.jenkins-ci.org/browse/JENKINS-27332
          git-plugin no longer detects changes of branch with /

          The basic question of using a parameterized branch seems to have been fixed in Mar...I had been wondering if that was affecting me (since I'm using a branch parameter) ..but this says it's fixed
          https://issues.jenkins-ci.org/browse/JENKINS-27166
          CLONE - Git SCM-polling doesn't work when using a parametrized branch-name

          Kevin Normoyle added a comment - - edited Hi thanks. Well, a couple of days has gone by with multiple jenkins using similar jobs (polling with parameterized branches). 6 jenkins masters each with 8 jobs. total of 12 branches. Seems to be working as desired with single branches resolved correctly and no unnecessary building due to changes to other branches. I've not tried to recreate the initial problem. It was weird since it affected every jenkins master I had created, and then it started working right as I noted above, after messing with reinstall of jenkins and the git-plugin. (I think that was what led it to working..although there could have been some racy thing. I was doing combined polling plus scheduled build. I don't think there was any race effect going on, but if jenkins was looking at "last build" state somehow, you never know. But looking at other open jiras in this area: I found another open jira in this area that is interesting. I use a parameter ${BRANCH} to specify the branch. While there have been issues in the past with that, I notice in this jira a very odd behavior, where the behavior depends on the branch being different in the last build. I was wondering if I copied jenkins setups with jobs pointing to master or a different branch, then modified the branch parameter in the job. (well yeah, I know that's how I setup all my different jenkins masters on different machines) I'm wondering if the issue mentioned in this jira could have affected things..i.e. jenkins deciding that a branch variable wasn't what it was supposed to be. BRANCH_TO_BUILD is equivalent to my use of BRANCH (${BRANCH}) in the jobs I was having problems with above. quoting from this link https://issues.jenkins-ci.org/browse/JENKINS-27349 Git SCM-polling uses wrong parameter values for a parametrized branchspec to reproduce: A parametrized build with parameter BRANCH_TO_BUILD, with default value "branch1" Git SCM with branch specifier "${BRANCH_TO_BUILD}" SCM polling enabled Previous manual job launch with BRANCH_TO_BUILD=branch2 Expected: When polling, BRANCH_TO_BUILD is set to branch1 (default value), so only updates to branch1 trigger a build. Actual: When polling, BRANCH_TO_BUILD is set to branch2 (value from last execution). also: https://issues.jenkins-ci.org/browse/JENKINS-27327 When specifying the branch to poll as part of a parameterized build, git-plugin uses last polled branch instead another related one, but not what I was seeing: https://issues.jenkins-ci.org/browse/JENKINS-27332 git-plugin no longer detects changes of branch with / The basic question of using a parameterized branch seems to have been fixed in Mar...I had been wondering if that was affecting me (since I'm using a branch parameter) ..but this says it's fixed https://issues.jenkins-ci.org/browse/JENKINS-27166 CLONE - Git SCM-polling doesn't work when using a parametrized branch-name

          Kevin Normoyle: Thanks for this survey and cross-referencing other issues. Just for information, our job, for which this issue was originally reported (https://ci.linaro.org/job/trigger-linux-ltsi), doesn't have branch parametrized, it is set statically in the job config.

          Paul Sokolovsky added a comment - Kevin Normoyle: Thanks for this survey and cross-referencing other issues. Just for information, our job, for which this issue was originally reported ( https://ci.linaro.org/job/trigger-linux-ltsi ), doesn't have branch parametrized, it is set statically in the job config.

          I have a similar problem which might be related. Jenkins was working fine with me until I upgraded the git plugin on April 1st, 2015. After that date, Jenkins performs the polling as expected but it never find changes although there is.

          When I downgraded the git plugin from 2.3.5 to 2.3, everything started working again.

          Mohamed Galal El-Din added a comment - I have a similar problem which might be related. Jenkins was working fine with me until I upgraded the git plugin on April 1st, 2015. After that date, Jenkins performs the polling as expected but it never find changes although there is. When I downgraded the git plugin from 2.3.5 to 2.3, everything started working again.

          Hi Mohamed.
          As I noted above, the problem I had disappeared as I was messing with things.

          what sort of system do you have jenkins running on?
          I reinstalled jenkins on my ubuntu system with apt-get install jenkins --reinstall
          that may have had a correlation

          I can show some things from my current jobs.
          Git client plugin version 1.16.1
          Git plugin version 2.3.5
          Github API plugin 1.63
          Github plugin 1.11
          Jenkins 1.607

          I have a String Parameter on the build called BRANCH

          Under Advanced in Source Code Management, the Refspec is
          +refs/heads/${BRANCH}:refs/remotes/origin/${BRANCH}
          Branches to Build (Branch Specifier) is
          ${BRANCH}

          I'm using Poll SCM

          it currently is working, with Git Polling Log giving results like this when no change is needed
          Started on Apr 6, 2015 3:55:00 PM
          Using strategy: Default
          [poll] Last Built Revision: Revision 4a8e9987b4d2ca94a966abe88a590809e3109e4c (origin/cliffc_jenkins)
          > git ls-remote -h https://github.com/h2oai/h2o-dev.git # timeout=10
          [poll] Latest remote head revision on origin/cliffc_jenkins is: 4a8e9987b4d2ca94a966abe88a590809e3109e4c - already built by 1032
          Done. Took 3.5 sec
          No changes

          Kevin Normoyle added a comment - Hi Mohamed. As I noted above, the problem I had disappeared as I was messing with things. what sort of system do you have jenkins running on? I reinstalled jenkins on my ubuntu system with apt-get install jenkins --reinstall that may have had a correlation I can show some things from my current jobs. Git client plugin version 1.16.1 Git plugin version 2.3.5 Github API plugin 1.63 Github plugin 1.11 Jenkins 1.607 I have a String Parameter on the build called BRANCH Under Advanced in Source Code Management, the Refspec is +refs/heads/${BRANCH}:refs/remotes/origin/${BRANCH} Branches to Build (Branch Specifier) is ${BRANCH} I'm using Poll SCM it currently is working, with Git Polling Log giving results like this when no change is needed Started on Apr 6, 2015 3:55:00 PM Using strategy: Default [poll] Last Built Revision: Revision 4a8e9987b4d2ca94a966abe88a590809e3109e4c (origin/cliffc_jenkins) > git ls-remote -h https://github.com/h2oai/h2o-dev.git # timeout=10 [poll] Latest remote head revision on origin/cliffc_jenkins is: 4a8e9987b4d2ca94a966abe88a590809e3109e4c - already built by 1032 Done. Took 3.5 sec No changes

          Hi Kevin,

          I'm using Ubuntu 14.04. And originally installed Jenkins using the apt-get method as explained here

          My Jenkins server version is 1.607. And my git repository is hosted on BitBucket.

          My issue seems to be a different than this one. I'll file a separate JIRA.

          Mohamed Galal El-Din added a comment - Hi Kevin, I'm using Ubuntu 14.04. And originally installed Jenkins using the apt-get method as explained here My Jenkins server version is 1.607. And my git repository is hosted on BitBucket. My issue seems to be a different than this one. I'll file a separate JIRA.

          I've reported JENKINS-27769 to track my issue. Thanks Kevin.

          Mohamed Galal El-Din added a comment - I've reported JENKINS-27769 to track my issue. Thanks Kevin.

          Mark Waite added a comment -

          The git plugin 2.3.6 pre-release and git client plugin 1.18.0 pre-release are being tested in hopes of releasing new versions before the end of June. If you're willing to assist with the testing, please download and install a pre-release build of the git client plugin and the git plugin. Problems detected in the pre-release should be e-mailed to MarkEWaite and ndeloof.

          I wrote some test ideas if you would like suggestions of areas that need testing. The git plugin supports many different use cases and its automated tests only evaluate a very few of those use cases.

          Mark Waite added a comment - The git plugin 2.3.6 pre-release and git client plugin 1.18.0 pre-release are being tested in hopes of releasing new versions before the end of June. If you're willing to assist with the testing, please download and install a pre-release build of the git client plugin and the git plugin . Problems detected in the pre-release should be e-mailed to MarkEWaite and ndeloof . I wrote some test ideas if you would like suggestions of areas that need testing. The git plugin supports many different use cases and its automated tests only evaluate a very few of those use cases.

          Mark Waite added a comment -

          Fix included in git plugin 2.4.0 released 18 July 2015

          Mark Waite added a comment - Fix included in git plugin 2.4.0 released 18 July 2015

            ndeloof Nicolas De Loof
            pfalcon Paul Sokolovsky
            Votes:
            1 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated:
              Resolved: