Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50556

git polling succeeds even when origin/master has nothing new

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • git-plugin
    • None
    • Git Plugin 3.3.2
      Jenkins 2.60.1

      I have a repository which is setup to perform git polling, and it always triggers, suggesting there are changes, even though there are not.

      In this case, the job has multiple checkouts using a pipeline, but I think it would only happen in the case of a single checkout as well.

      Specifically, the branch name is master, and it checks first all the "remote/branch" combinations, (in this case, "origin/master") and then fails to find anything new (as origin/master was already built). Then, it checks the branch name on its own, thinking that maybe it's a tag or some other ref.

      This results in reporting a separate commit (likely whatever master happens to be at when the polling was created), and thus reports a new commit to build. This causes the build to trigger every polling timeout, even though there aren't any real new changes.

      Here's a subset of the actual output with repository names and paths redacted. I have the VERBOSE logging enabled for more information.

      Started on Apr 3, 2018 4:01:00 PM
      Using strategy: Default
      [poll] Last Built Revision: Revision 72c4cea985bb5ff14813d27dac94f34887cbd841 (refs/remotes/origin/master)
      using GIT_SSH to set credentials 
       > git ls-remote -h <groovyLibURL> # timeout=10
      Found 1 remote heads on <groovyLibURL>
      [poll] Latest remote head revision on refs/heads/master is: 72c4cea985bb5ff14813d27dac94f34887cbd841 - already built by 1738
      Using strategy: Default
      [poll] Last Built Revision: Revision cc24044526e9d460c74a59fcdeb12e43a446cb7d (origin/master)
       > git rev-parse --is-inside-work-tree # timeout=10
      Fetching changes from the remote Git repositories
       > git config remote.origin.url <mainRepositoryURL> # timeout=10
      Fetching upstream changes from <mainRepositoryURL>
       > git --version # timeout=10
      using GIT_SSH to set credentials 
       > git fetch --no-tags --progress <mainRepositoryURL> +refs/heads/*:refs/remotes/origin/*
      Polling for changes in
      getCandidateRevisions(true,master,,,hudson.plugins.git.util.BuildData@3fcd3164[scmName=<mainRepositoryName>,remoteUrls=[<mainRepositoryURL>],buildsByBranchName={origin/master=Build #1738 of Revision cc24044526e9d460c74a59fcdeb12e43a446cb7d (origin/master)},lastBuild=Build #1738 of Revision cc24044526e9d460c74a59fcdeb12e43a446cb7d (origin/master)]) considering branches to build
      Qualifying master as a branch in repository origin -> origin/master
       > git rev-parse origin/master^{commit} # timeout=10
      rev-parse origin/master -> AnyObjectId[cc24044526e9d460c74a59fcdeb12e43a446cb7d]
      AnyObjectId[cc24044526e9d460c74a59fcdeb12e43a446cb7d] has already been built
       > git rev-parse master^{commit} # timeout=10
      Failed to rev-parse: master
      Using strategy: Default
      [poll] Last Built Revision: Revision d24e49e865522042434be7b1a46f592aae2608dc (origin/master)
       > git rev-parse --is-inside-work-tree # timeout=10
      Fetching changes from the remote Git repositories
       > git config remote.origin.url <firstRepositoryURL> # timeout=10
      Fetching upstream changes from <firstRepositoryURL>
       > git --version # timeout=10
      using GIT_SSH to set credentials 
       > git fetch --no-tags --progress <firstRepositoryURL> +refs/heads/*:refs/remotes/origin/*
      Polling for changes in
      getCandidateRevisions(true,master,,,hudson.plugins.git.util.BuildData@857d3e89[scmName=<firstRepositoryName>,remoteUrls=[<firstRepositoryURL>],buildsByBranchName={origin/master=Build #1738 of Revision d24e49e865522042434be7b1a46f592aae2608dc (origin/master)},lastBuild=Build #1738 of Revision d24e49e865522042434be7b1a46f592aae2608dc (origin/master)]) considering branches to build
      Qualifying master as a branch in repository origin -> origin/master
       > git rev-parse origin/master^{commit} # timeout=10
      rev-parse origin/master -> AnyObjectId[d24e49e865522042434be7b1a46f592aae2608dc]
      AnyObjectId[d24e49e865522042434be7b1a46f592aae2608dc] has already been built
       > git rev-parse master^{commit} # timeout=10
      rev-parse master -> AnyObjectId[d24e49e865522042434be7b1a46f592aae2608dc]
      AnyObjectId[d24e49e865522042434be7b1a46f592aae2608dc] has already been built
      Using strategy: Default
      [poll] Last Built Revision: Revision 93a6d3510777096a6fbd7f16ac22c28ef4bd872e (origin/master)
       > git rev-parse --is-inside-work-tree # timeout=10
      Fetching changes from the remote Git repositories
       > git config remote.origin.url <secondRepositoryURL> # timeout=10
      Fetching upstream changes from <secondRepositoryURL>
       > git --version # timeout=10
      using GIT_SSH to set credentials 
       > git fetch --no-tags --progress <secondRepositoryURL> +refs/heads/*:refs/remotes/origin/*
      Polling for changes in
      getCandidateRevisions(true,master,,,hudson.plugins.git.util.BuildData@2e191547[scmName=<secondRepositoryName>,remoteUrls=[<secondRepositoryURL>],buildsByBranchName={origin/master=Build #1738 of Revision 93a6d3510777096a6fbd7f16ac22c28ef4bd872e (origin/master)},lastBuild=Build #1738 of Revision 93a6d3510777096a6fbd7f16ac22c28ef4bd872e (origin/master)]) considering branches to build
      Qualifying master as a branch in repository origin -> origin/master
       > git rev-parse origin/master^{commit} # timeout=10
      rev-parse origin/master -> AnyObjectId[93a6d3510777096a6fbd7f16ac22c28ef4bd872e]
      AnyObjectId[93a6d3510777096a6fbd7f16ac22c28ef4bd872e] has already been built
       > git rev-parse master^{commit} # timeout=10
      rev-parse master -> AnyObjectId[4bf94e5146026ba4e3bebef3b2ebf8a795d0d0f8]
      Found a new commit AnyObjectId[4bf94e5146026ba4e3bebef3b2ebf8a795d0d0f8] to be built on master
      {0} seems to be a non-branch reference (tag?)
       > git log --full-history --no-abbrev --format=raw -M -m 93a6d3510777096a6fbd7f16ac22c28ef4bd872e..4bf94e5146026ba4e3bebef3b2ebf8a795d0d0f8 # timeout=10
      Using strategy: Default
      [poll] Last Built Revision: Revision a4db49e3433ef3304c7956e81369c2ed5fa69b5d (origin/master)
       > git rev-parse --is-inside-work-tree # timeout=10
      Fetching changes from the remote Git repositories
       > git config remote.origin.url <thirdRepositoryURL> # timeout=10
      Fetching upstream changes from <thirdRepositoryURL>
       > git --version # timeout=10
      using GIT_SSH to set credentials 
       > git fetch --no-tags --progress <thirdRepositoryURL> +refs/heads/*:refs/remotes/origin/*
      Polling for changes in
      getCandidateRevisions(true,master,,,hudson.plugins.git.util.BuildData@49f3b5e7[scmName=<thirdRepositoryName>,remoteUrls=[<thirdRepositoryURL>],buildsByBranchName={origin/master=Build #1738 of Revision a4db49e3433ef3304c7956e81369c2ed5fa69b5d (origin/master)},lastBuild=Build #1738 of Revision a4db49e3433ef3304c7956e81369c2ed5fa69b5d (origin/master)]) considering branches to build
      Qualifying master as a branch in repository origin -> origin/master
       > git rev-parse origin/master^{commit} # timeout=10
      rev-parse origin/master -> AnyObjectId[a4db49e3433ef3304c7956e81369c2ed5fa69b5d]
      AnyObjectId[a4db49e3433ef3304c7956e81369c2ed5fa69b5d] has already been built
       > git rev-parse master^{commit} # timeout=10
      rev-parse master -> AnyObjectId[00f059f63e272c44fb3f754be4e2f4a1b474c416]
      Found a new commit AnyObjectId[00f059f63e272c44fb3f754be4e2f4a1b474c416] to be built on master
      {0} seems to be a non-branch reference (tag?)
       > git log --full-history --no-abbrev --format=raw -M -m a4db49e3433ef3304c7956e81369c2ed5fa69b5d..00f059f63e272c44fb3f754be4e2f4a1b474c416 # timeout=10
      Using strategy: Default
      [poll] Last Built Revision: Revision 06188df26697ef38293a8533ce601934762c7d6c (origin/master)
       > git rev-parse --is-inside-work-tree # timeout=10
      Fetching changes from the remote Git repositories
       > git config remote.origin.url <fourthRepositoryURL> # timeout=10
      Fetching upstream changes from <fourthRepositoryURL>
       > git --version # timeout=10
      using GIT_SSH to set credentials 
       > git fetch --no-tags --progress <fourthRepositoryURL> +refs/heads/*:refs/remotes/origin/*
      Polling for changes in
      getCandidateRevisions(true,master,,,hudson.plugins.git.util.BuildData@dff532d1[scmName=<fourthRepositoryName>,remoteUrls=[<fourthRepositoryURL>],buildsByBranchName={origin/master=Build #1738 of Revision 06188df26697ef38293a8533ce601934762c7d6c (origin/master)},lastBuild=Build #1738 of Revision 06188df26697ef38293a8533ce601934762c7d6c (origin/master)]) considering branches to build
      Qualifying master as a branch in repository origin -> origin/master
       > git rev-parse origin/master^{commit} # timeout=10
      rev-parse origin/master -> AnyObjectId[06188df26697ef38293a8533ce601934762c7d6c]
      AnyObjectId[06188df26697ef38293a8533ce601934762c7d6c] has already been built
       > git rev-parse master^{commit} # timeout=10
      Failed to rev-parse: master
      Done. Took 6.5 sec
      Changes found
      

      I am highly suspect that the relevant commit related to why this breaks is:

      c7dae80b85c1 ("JENKINS-21952 Resolve tags with slashes", 2014-02-27)

      I believe what is happening is that the "remote/branch" output is failing to find any new changes, so we fall back to attempting to resolve the name like a tag, but it so happens that the polling setup actually does end up with the branch checked out, and it results in us finding the local master name and reporting it.

      I don't really know how to resolve this yet.

          [JENKINS-50556] git polling succeeds even when origin/master has nothing new

          Phil Adams added a comment - - edited

          I'm having the same issue.  My github enterprise repo has two branches - "master" and "0.0.1" - and I have two jenkins jobs defined (one for each branch... i.e. the branch specifier in each job definition is set to "*/<branchname>").  In my github repository, I've configured the webhook url (https://<jenkins-host>/git/notifyCommit?url=<github-repo-url>&branches=master,0.0.1).   When I hit the webhook URL (testing from github), jenkins receives the POST and pokes the correct jobs, but it incorrectly things there are changes on each of the branches, so it triggers a build for each one.

          While trying to get the git push triggering to work with the GitHub plugin, I see the same exact behavior.   A push to one branch triggers both builds to start, even though the push was to only one of the branches.   So it seems like the basic issue here is when the git plugin looks to see if there are changes on a branch, it always thinks there are changes even when no commits have been pushed since the last build.

          Edit: In case it matters, I'm using Jenkins v2.89.2 with the Git Plugin v3.8.0

          Edit: Just wanted clarify a couple things...  I think my scenario is simpler than the one described by the issue originator.   I temporarily changed one of my jobs (for the 0.0.1 branch) to use polling (every 2 minutes for testing purposes) rather than the webhook.   Without any changes being pushed to my 0.0.1 branch, the first time the polling ran it thought there were changes present so it triggered a build.  And subsequent to that, each time the polling is initiated (every 2 minutes), it thinks there is a change and triggers a build.   Is there anything I can do via configuration to fix this?   Also, if there is some additional server logging that I can enable to try to determine why this is broken, I'm happy to do that as well.

          Thanks!

          Phil Adams added a comment - - edited I'm having the same issue.  My github enterprise repo has two branches - "master" and "0.0.1" - and I have two jenkins jobs defined (one for each branch... i.e. the branch specifier in each job definition is set to "*/<branchname>").  In my github repository, I've configured the webhook url (https://<jenkins-host>/git/notifyCommit?url=<github-repo-url>&branches=master,0.0.1).   When I hit the webhook URL (testing from github), jenkins receives the POST and pokes the correct jobs, but it incorrectly things there are changes on each of the branches, so it triggers a build for each one. While trying to get the git push triggering to work with the GitHub plugin, I see the same exact behavior.   A push to one branch triggers both builds to start, even though the push was to only one of the branches.   So it seems like the basic issue here is when the git plugin looks to see if there are changes on a branch, it always thinks there are changes even when no commits have been pushed since the last build. Edit: In case it matters, I'm using Jenkins v2.89.2 with the Git Plugin v3.8.0 Edit: Just wanted clarify a couple things...  I think my scenario is simpler than the one described by the issue originator.   I temporarily changed one of my jobs (for the 0.0.1 branch) to use polling (every 2 minutes for testing purposes) rather than the webhook.   Without any changes being pushed to my 0.0.1 branch, the first time the polling ran it thought there were changes present so it triggered a build.  And subsequent to that, each time the polling is initiated (every 2 minutes), it thinks there is a change and triggers a build.   Is there anything I can do via configuration to fix this?   Also, if there is some additional server logging that I can enable to try to determine why this is broken, I'm happy to do that as well. Thanks!

          Phil Adams added a comment - - edited

          I just thought to check the "Polling log" section of my build results and for the offending job (which builds my 0.0.1 branch), I see this:

          ----------------------
          Started on Apr 6, 2018 5:02:00 PM Workspace is offline.
          Scheduling a new build to get a workspace. (nonexisting_workspace)
          Done. Took 0 ms
          Changes found

          ----------------------

          So... it looks like the job is triggered each time because the jenkins server can't find an existing workspace where a previous build has been performed.  Am I understanding that correctly?

          Is there no way to detect changes on a branch without having an existing workspace with the branch checked out?   In my case, our jenkins server assigns build jobs to agents running in a docker swarm, where the agent is effectively wiped clean after every build, hence a subsequent build will be done on a completely clean randomly-assigned agent.

          Phil Adams added a comment - - edited I just thought to check the "Polling log" section of my build results and for the offending job (which builds my 0.0.1 branch), I see this: ---------------------- Started on Apr 6, 2018 5:02:00 PM Workspace is offline. Scheduling a new build to get a workspace. (nonexisting_workspace) Done. Took 0 ms Changes found ---------------------- So... it looks like the job is triggered each time because the jenkins server can't find an existing workspace where a previous build has been performed.  Am I understanding that correctly? Is there no way to detect changes on a branch without having an existing workspace with the branch checked out?   In my case, our jenkins server assigns build jobs to agents running in a docker swarm, where the agent is effectively wiped clean after every build, hence a subsequent build will be done on a completely clean randomly-assigned agent.

          Jacob Keller added a comment -

          padamstx, it could maybe use the remote polling method, which doesn't require the workspace, but that gets turned off for a variety of reasons.

           

          It sounds like a different issue than the one I am describing here. I do have a potential fix posted at https://github.com/jenkinsci/git-plugin/pull/579 and you'd be welcome to try it out and see if it fixes your problem, (though I doubt it would...)

          Jacob Keller added a comment - padamstx , it could maybe use the remote polling method, which doesn't require the workspace, but that gets turned off for a variety of reasons.   It sounds like a different issue than the one I am describing here. I do have a potential fix posted at https://github.com/jenkinsci/git-plugin/pull/579  and you'd be welcome to try it out and see if it fixes your problem, (though I doubt it would...)

          Phil Adams added a comment -

          Thanks Jacob, do you know how I could tell the git plugin to use the remote polling method instead of requiring a workspace?

          Phil Adams added a comment - Thanks Jacob, do you know how I could tell the git plugin to use the remote polling method instead of requiring a workspace?

          Jacob Keller added a comment -

          > Thanks Jacob, do you know how I could tell the git plugin to use the remote polling method instead of requiring a workspace?

          Make sure that you don't use any extensions which enable worktree polling. These include "author in change log", "change log to branch", "require workspace when polling", "Message exclusion", "User exclusion", or "Path restriction"

          I do not understand offhand why "author in change log" and "change log to branch" require this at the moment, but message exclusion, user exclusion, and path restriction require it because these features aren't implemented within the remote polling code. (I suspect it's possible to do so, but no one has done it).

          Additionally, remote polling is not setup to handle more than a single branch, so you need to make sure you only use one branch.

          It's in theory possible to fix these limitations, but I do not offhand know the work involved to do so.

          Jacob Keller added a comment - > Thanks Jacob, do you know how I could tell the git plugin to use the remote polling method instead of requiring a workspace? Make sure that you don't use any extensions which enable worktree polling. These include "author in change log", "change log to branch", "require workspace when polling", "Message exclusion", "User exclusion", or "Path restriction" I do not understand offhand why "author in change log" and "change log to branch" require this at the moment, but message exclusion, user exclusion, and path restriction require it because these features aren't implemented within the remote polling code. (I suspect it's possible to do so, but no one has done it). Additionally, remote polling is not setup to handle more than a single branch, so you need to make sure you only use one branch. It's in theory possible to fix these limitations, but I do not offhand know the work involved to do so.

          Phil Adams added a comment -

          Jacob, thanks for the info... it turns out that I did in fact have the "author in change log" option selected, and that was why workspace polling was being used.   By disabling that option, the plugin started doing remote polling.

          Phil Adams added a comment - Jacob, thanks for the info... it turns out that I did in fact have the "author in change log" option selected, and that was why workspace polling was being used.   By disabling that option, the plugin started doing remote polling.

          Jacob Keller added a comment -

          I submitted a pull request https://github.com/jenkinsci/git-plugin/pull/583 which should remove that restriction (as far as I can tell, it was inadvertently added and there's no reason that the extension "author in change log" depends on the workspace polling.

          Jacob Keller added a comment - I submitted a pull request https://github.com/jenkinsci/git-plugin/pull/583 which should remove that restriction (as far as I can tell, it was inadvertently added and there's no reason that the extension "author in change log" depends on the workspace polling.

          Jacob Keller added a comment -

          I believe I fixed this with the following pull request: https://github.com/jenkinsci/git-plugin/pull/579

          Jacob Keller added a comment - I believe I fixed this with the following pull request: https://github.com/jenkinsci/git-plugin/pull/579

          Mark Waite added a comment -

          I've been evaluating this pull request today. Evaluation is not complete, but it is looking good. I'd like to include this in the 4.0.0-beta1 release. However, I first need to merge several other changes (the require-jdk-8 PR and a few others that I've already evaluated).

          I need some help rarabaolaza to understand the test failures on the pull request. I'll connect with him early next week for further discussion.

          Mark Waite added a comment - I've been evaluating this pull request today. Evaluation is not complete, but it is looking good. I'd like to include this in the 4.0.0-beta1 release. However, I first need to merge several other changes (the require-jdk-8 PR and a few others that I've already evaluated). I need some help rarabaolaza to understand the test failures on the pull request. I'll connect with him early next week for further discussion.

          Jacob Keller added a comment -

          I've had this pull request running as well in my test instance that I setup to test the thousands of branches case with the build details pull request as well, so far I haven't found any issues either.

          I'm curious what the ATH thing is as well. Hopefully we can get to the bottom of that one.

          Jacob Keller added a comment - I've had this pull request running as well in my test instance that I setup to test the thousands of branches case with the build details pull request as well, so far I haven't found any issues either. I'm curious what the ATH thing is as well. Hopefully we can get to the bottom of that one.

            Unassigned Unassigned
            jekeller Jacob Keller
            Votes:
            2 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: