Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55714

Git plugin scm polling trigger not working on first build

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • git-plugin
    • None
    • Jenkins v2.148
      Git plugin v3.9.1

      We have an odd problem with Git SCM polling on some of our jenkins projects. The projects in question only seem to be affected when they are built for the first time. These projects are of type "pipeline" and they always have multiple Git repositories being checked out.

      The build configurations are somewhat complex, but here are the variables that I think are contributing to the problem. First off, we have multiple stages for these builds, each of which checks out several of the same git repos. Further, to ensure that each stage checks out the same copy / revision of these repos we first check out the head revision from a given refspec on the first stage and then cache the revision / hash from that checkout, and then use that hash when checking out subsequent copies. In pseudo-code, the mechanics look something like this:

      def cached_rev = '' 
      stage ("stage 1") {
           node {
               checkout (scm: [class: '$GitSCM', branches: 'master'...)
               cached_rev = sh(returnStdout: true, script: 'git rev-parse HEAD').trim()
               // do some build stuff
           }
      }
      stage ("stage 2") {
           node {
               checkout (scm: [class: '$GitSCM', branches: cached_rev...])
               // do more build stuff
           }
      } 

      The mechanics of this configuration work as expected at built time (ie: the correct / same revisions get checked out for each stage), however the polling mechanics don't work. For some reason changes to repos configured in this way always detect changes in them which causes builds to run over and over and over again, in perpetuity.

      One other factor that may play a role here is that we also dynamically set the `changelog` attribute on each SCM checkout. We set the flag to `true` in the first stage, and then to `false` on all subsequent stages. We do this to try and prevent the change logs from these shared repositories from showing up many times in duplicate on the build summary page (ie: if we have 5 stages configured like this, all sharing the same repo configuration, then changes to this repo typically show up 5 times in the build summary page, which is annoying at best).

      One other interesting piece of the puzzle here as well is that we have tried setting the `poll: false` option on these common Git repos as well to try and circumvent this strange pollng behavior. However, interestingly enough, this seems to have no effect. It is almost like the poll flag is being ignored in certain cases or perhaps confusing the Git plugin in certain situations. 

      Another interesting piece of the puzzle is that the problematic behavior only seems to happen when we build a project for the first time. I've noticed the message "no polling baseline" several times in the polling logs for these builds, which seems to suggest to me that perhaps the Git plugin is doing something different when a job is first built, maybe because it has no pre-existing workspace to use as a reference for pulling Git revisions. It seems a little odd to me that Jenkins would be using the workspace for polling data seeing as how the workspace may exist on any number of build agents attached to the same build farm, some of which may be more recent than others. Also, should the build agent used for the most recent build go offline then that could introduce instability in the logic as well. I'm not 100% convinced this specific message contributes to the problem I'm reporting here. I'm just mentioning it in case it's relevant in some way.

      Finally, we've tried several things to "work around" this problem in our production environment. Killing duplicate builds that get triggered yields no effect. The next SCM trigger operation still picks up changes from earlier builds and flags them as "new". We've tried killing newly triggered builds from the queue before they start building, but subsequent SCM triggers still kick in. The only effective workaround we've found is if we disable the jobs that exhibit the problem (ie: using Groovy API calls), leaving the job disabled until the next build trigger kicks in, and then re-enable the job, the SCM triggers seem to correct themselves and the problem ceases to exist. Subsequent build operations proceed as normal.

      Any help anyone may have that could explain this problem or how to fix it would be greatly appreciated. 

       

          [JENKINS-55714] Git plugin scm polling trigger not working on first build

          Here is some sample output from some of our Git polling logs that seemed relevant. Each one of our builds that hit this problem have output that looks something like this:

          [poll] Latest remote head revision on refs/heads/master is: c3da10753cc0860f500df626fdb0721897ec5b05
          no polling baseline in /home/path/to/builds/IHNK35RKEDZLNTXWNWBKYMUY4CX6G6J764MFFEOAMVC574ZGNUAA on build-agent-35
          Using strategy: Default 

          Each affected build has the same message. The same refspec in each case. The same Git commit hash in each case. So even though build #1 obviously checks out that repo at that revision and built it, build #2 seems to miss that fact and thus it thinks the revision is new. This fact is corroberated by this other snippet from the polling log that appears at the very end of each polling operation:

           [poll] Latest remote head revision on refs/heads/master is: c3da10753cc0860f500df626fdb0721897ec5b05
          Done. Took 15 sec
          Changes found

          However, that being said we do have examples where builds that hit this problem have several Git repos each being checked out in the same way, but sometimes certain repos/branches do seem to detect when previous builds have used them. Usually the log output from the polling operation looks something like this:

          [poll] Latest remote head revision on refs/heads/master is: 3e04a485962dd3b3539ab8b5f214e9b8c52f9108 - already built by 2
          Using strategy: Default 

          So I am at a loss why sometimes the Git plugin can successfully resolve when a specific revision has been built before and when it hasn't. Sometimes we get different repos that work fairly consistently while others don't, but we also have cases where specific branches in a repo seem to detect changes properly while other branches don't. It's all very inconsistent.

          If anyone has any suggestions on how to narrow down the potential causes for this problem further do let me know.

          Kevin Phillips added a comment - Here is some sample output from some of our Git polling logs that seemed relevant. Each one of our builds that hit this problem have output that looks something like this: [poll] Latest remote head revision on refs/heads/master is: c3da10753cc0860f500df626fdb0721897ec5b05 no polling baseline in /home/path/to/builds/IHNK35RKEDZLNTXWNWBKYMUY4CX6G6J764MFFEOAMVC574ZGNUAA on build-agent-35 Using strategy: Default Each affected build has the same message. The same refspec in each case. The same Git commit hash in each case. So even though build #1 obviously checks out that repo at that revision and built it, build #2 seems to miss that fact and thus it thinks the revision is new. This fact is corroberated by this other snippet from the polling log that appears at the very end of each polling operation: [poll] Latest remote head revision on refs/heads/master is: c3da10753cc0860f500df626fdb0721897ec5b05 Done. Took 15 sec Changes found However, that being said we do have examples where builds that hit this problem have several Git repos each being checked out in the same way, but sometimes certain repos/branches do seem to detect when previous builds have used them. Usually the log output from the polling operation looks something like this: [poll] Latest remote head revision on refs/heads/master is: 3e04a485962dd3b3539ab8b5f214e9b8c52f9108 - already built by 2 Using strategy: Default So I am at a loss why sometimes the Git plugin can successfully resolve when a specific revision has been built before and when it hasn't. Sometimes we get different repos that work fairly consistently while others don't, but we also have cases where specific branches in a repo seem to detect changes properly while other branches don't. It's all very inconsistent. If anyone has any suggestions on how to narrow down the potential causes for this problem further do let me know.

          Hello Kevin,

          Please check https://issues.jenkins-ci.org/browse/JENKINS-38508 , may be relevant to your problem statement.

          Thanks,

          Nandlal

          Nandlal Deshmukh added a comment - Hello Kevin, Please check  https://issues.jenkins-ci.org/browse/JENKINS-38508  , may be relevant to your problem statement. Thanks, Nandlal

            Unassigned Unassigned
            leedega Kevin Phillips
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: