Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46279

Git plugin marks the same commit to be built multiple times

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • git-plugin
    • None
    • Git plugin 3.5.1
      Jenkins 2.74

      We have a pipeline job (live branch) that's set up to build a certain branch - every 10 minutes it polls for changes. The rest of our builds are set up to do a multibranch pipeline for the other branches in our repository (which we do to work around the fact we can't set a job priority for our live branch to build it before other branches).

      I currently have 9 builds for our live branch job in the queue because it polls for changes and sees a new commit in there which hasn't been built. Because there's no free executor at the moment, none of them have started so the commit hasn't been marked as "built" yet. The multibranch pipeline job doesn't do this.

          [JENKINS-46279] Git plugin marks the same commit to be built multiple times

          Mark Waite added a comment -

          If those 9 pending builds are each for different SHA1 hashes, then I believe that is worksing as expected. If polling detects a new SHA1 on a branch it is monitoring, then it schedules a build for that branch. I believe that Pipeline intentionally allows multiple builds to be queued by default for a single job, while freestyle and matrix jobs by default only queue one build after the running build.

          Are the pending builds for distinct SHA1 hashes?

          Mark Waite added a comment - If those 9 pending builds are each for different SHA1 hashes, then I believe that is worksing as expected. If polling detects a new SHA1 on a branch it is monitoring, then it schedules a build for that branch. I believe that Pipeline intentionally allows multiple builds to be queued by default for a single job, while freestyle and matrix jobs by default only queue one build after the running build. Are the pending builds for distinct SHA1 hashes?

          Patrick Rose added a comment - - edited

          They're for the same SHA1 hash:

          Started on 17-Aug-2017 18:11:00
          Using strategy: Default
          [poll] Last Built Revision: Revision 71a6660dafad9df161f4b4b17a76012685f9b750 (refs/remotes/origin/nursingnet/live)
          using GIT_SSH to set credentials 
           > git ls-remote -h jenkins_git@git.zedcore.com:/srv/git/zedcore # timeout=10
          Found 2293 remote heads on jenkins_git@git.zedcore.com:/srv/git/zedcore
          [poll] Latest remote head revision on refs/heads/nursingnet/live is: 4d2f7644049b6956f7060840120611414226648d
          Using strategy: Default
          [poll] Last Built Revision: Revision 4648f23128248c245ea5c703c810c1c73b19c41c (refs/remotes/origin/cjsegw/live)
          using GIT_SSH to set credentials 
           > git ls-remote -h git+ssh://jenkins_git@git.zedcore.com/srv/git/cjsegw # timeout=10
          Found 8 remote heads on git+ssh://jenkins_git@git.zedcore.com/srv/git/cjsegw
          [poll] Latest remote head revision on refs/heads/cjsegw/live is: 4648f23128248c245ea5c703c810c1c73b19c41c - already built by 39
          Done. Took 0.4 sec
          Changes found
          Started on 17-Aug-2017 16:31:00
          Using strategy: Default
          [poll] Last Built Revision: Revision 71a6660dafad9df161f4b4b17a76012685f9b750 (refs/remotes/origin/nursingnet/live)
          using GIT_SSH to set credentials 
           > git ls-remote -h jenkins_git@git.zedcore.com:/srv/git/zedcore # timeout=10
          Found 2293 remote heads on jenkins_git@git.zedcore.com:/srv/git/zedcore
          [poll] Latest remote head revision on refs/heads/nursingnet/live is: 4d2f7644049b6956f7060840120611414226648d
          Using strategy: Default
          [poll] Last Built Revision: Revision 4648f23128248c245ea5c703c810c1c73b19c41c (refs/remotes/origin/cjsegw/live)
          using GIT_SSH to set credentials 
           > git ls-remote -h git+ssh://jenkins_git@git.zedcore.com/srv/git/cjsegw # timeout=10
          Found 8 remote heads on git+ssh://jenkins_git@git.zedcore.com/srv/git/cjsegw
          [poll] Latest remote head revision on refs/heads/cjsegw/live is: 4648f23128248c245ea5c703c810c1c73b19c41c - already built by 39
          Done. Took 0.29 sec
          Changes found

          Patrick Rose added a comment - - edited They're for the same SHA1 hash: Started on 17-Aug-2017 18:11:00 Using strategy: Default [poll] Last Built Revision: Revision 71a6660dafad9df161f4b4b17a76012685f9b750 (refs/remotes/origin/nursingnet/live) using GIT_SSH to set credentials > git ls-remote -h jenkins_git@git.zedcore.com:/srv/git/zedcore # timeout=10 Found 2293 remote heads on jenkins_git@git.zedcore.com:/srv/git/zedcore [poll] Latest remote head revision on refs/heads/nursingnet/live is: 4d2f7644049b6956f7060840120611414226648d Using strategy: Default [poll] Last Built Revision: Revision 4648f23128248c245ea5c703c810c1c73b19c41c (refs/remotes/origin/cjsegw/live) using GIT_SSH to set credentials > git ls-remote -h git+ssh: //jenkins_git@git.zedcore.com/srv/git/cjsegw # timeout=10 Found 8 remote heads on git+ssh: //jenkins_git@git.zedcore.com/srv/git/cjsegw [poll] Latest remote head revision on refs/heads/cjsegw/live is: 4648f23128248c245ea5c703c810c1c73b19c41c - already built by 39 Done. Took 0.4 sec Changes found Started on 17-Aug-2017 16:31:00 Using strategy: Default [poll] Last Built Revision: Revision 71a6660dafad9df161f4b4b17a76012685f9b750 (refs/remotes/origin/nursingnet/live) using GIT_SSH to set credentials > git ls-remote -h jenkins_git@git.zedcore.com:/srv/git/zedcore # timeout=10 Found 2293 remote heads on jenkins_git@git.zedcore.com:/srv/git/zedcore [poll] Latest remote head revision on refs/heads/nursingnet/live is: 4d2f7644049b6956f7060840120611414226648d Using strategy: Default [poll] Last Built Revision: Revision 4648f23128248c245ea5c703c810c1c73b19c41c (refs/remotes/origin/cjsegw/live) using GIT_SSH to set credentials > git ls-remote -h git+ssh: //jenkins_git@git.zedcore.com/srv/git/cjsegw # timeout=10 Found 8 remote heads on git+ssh: //jenkins_git@git.zedcore.com/srv/git/cjsegw [poll] Latest remote head revision on refs/heads/cjsegw/live is: 4648f23128248c245ea5c703c810c1c73b19c41c - already built by 39 Done. Took 0.29 sec Changes found

          Mark Waite added a comment -

          That seems like a bug, though I'm not entirely sure of the rules used by pipeline polling. When polling happens, if the detected SHA1 has not been built, and is not already in the queue to be built, then it should be queued to be built.

          Could you perform time specific tests to see if each build appears very near the time of the polling?

          Mark Waite added a comment - That seems like a bug, though I'm not entirely sure of the rules used by pipeline polling. When polling happens, if the detected SHA1 has not been built, and is not already in the queue to be built, then it should be queued to be built. Could you perform time specific tests to see if each build appears very near the time of the polling?

          Patrick Rose added a comment -

          Looks like it. The polling log has

          Started on 23-Aug-2017 16:41:00
          Using strategy: Default
          [poll] Last Built Revision: Revision e9406f06aa465edcd12b8b18c0d1f1ea6de29679 (refs/remotes/origin/nursingnet/live)
          using GIT_SSH to set credentials 
           > git ls-remote -h jenkins_git@git.zedcore.com:/srv/git/zedcore # timeout=10
          Found 2306 remote heads on jenkins_git@git.zedcore.com:/srv/git/zedcore
          [poll] Latest remote head revision on refs/heads/nursingnet/live is: 4c86db715fe7c7789225613d866d5b849f01f15c
          Using strategy: Default
          [poll] Last Built Revision: Revision 4648f23128248c245ea5c703c810c1c73b19c41c (refs/remotes/origin/cjsegw/live)
          using GIT_SSH to set credentials 
           > git ls-remote -h git+ssh://jenkins_git@git.zedcore.com/srv/git/cjsegw # timeout=10
          Found 8 remote heads on git+ssh://jenkins_git@git.zedcore.com/srv/git/cjsegw
          [poll] Latest remote head revision on refs/heads/cjsegw/live is: 4648f23128248c245ea5c703c810c1c73b19c41c - already built by 42
          Done. Took 0.31 sec
          Changes found

          Then the build began at 16:42(ish). Looking at the build.xml file, the QuietPeriodCompleted element has 1503502867256 (which is 23-Aug-2017 16:41:07.256 in my timezone)

          Patrick Rose added a comment - Looks like it. The polling log has Started on 23-Aug-2017 16:41:00 Using strategy: Default [poll] Last Built Revision: Revision e9406f06aa465edcd12b8b18c0d1f1ea6de29679 (refs/remotes/origin/nursingnet/live) using GIT_SSH to set credentials > git ls-remote -h jenkins_git@git.zedcore.com:/srv/git/zedcore # timeout=10 Found 2306 remote heads on jenkins_git@git.zedcore.com:/srv/git/zedcore [poll] Latest remote head revision on refs/heads/nursingnet/live is: 4c86db715fe7c7789225613d866d5b849f01f15c Using strategy: Default [poll] Last Built Revision: Revision 4648f23128248c245ea5c703c810c1c73b19c41c (refs/remotes/origin/cjsegw/live) using GIT_SSH to set credentials > git ls-remote -h git+ssh: //jenkins_git@git.zedcore.com/srv/git/cjsegw # timeout=10 Found 8 remote heads on git+ssh: //jenkins_git@git.zedcore.com/srv/git/cjsegw [poll] Latest remote head revision on refs/heads/cjsegw/live is: 4648f23128248c245ea5c703c810c1c73b19c41c - already built by 42 Done. Took 0.31 sec Changes found Then the build began at 16:42(ish). Looking at the build.xml file, the QuietPeriodCompleted element has 1503502867256 (which is 23-Aug-2017 16:41:07.256 in my timezone)

          Patrick Rose added a comment -

          I think I see what the problem is by looking at the build.xml files - the ones that are currently waiting to be built haven't got a <hudson.plugins.git.util.BuildData plugin="git@3.5.1"> element in there. Earlier builds do, and they've marked other commits as built by a certain branch.

          I guess it's because it's not sure which commit it's going to build until the build actually starts? If I push:

          1. Push commit A to the branch
          2. Wait for it to add to the queue (behind some other build)
          3. Push commit B
          4. Stop the long running build so it builds this branch

          Which commit should it build? I think it'd be commit B but I haven't tested that yet.

          Patrick Rose added a comment - I think I see what the problem is by looking at the build.xml files - the ones that are currently waiting to be built haven't got a <hudson.plugins.git.util.BuildData plugin="git@3.5.1"> element in there. Earlier builds do, and they've marked other commits as built by a certain branch. I guess it's because it's not sure which commit it's going to build until the build actually starts? If I push: Push commit A to the branch Wait for it to add to the queue (behind some other build) Push commit B Stop the long running build so it builds this branch Which commit should it build? I think it'd be commit B but I haven't tested that yet.

          Patrick Rose added a comment -

          Is there any workaround you can suggest? I just had 40+ builds in the queue that took up our executors (so we're already behind this morning) because these builds are marked as high priority

          Patrick Rose added a comment - Is there any workaround you can suggest? I just had 40+ builds in the queue that took up our executors (so we're already behind this morning) because these builds are marked as high priority

          Mark Waite added a comment -

          One work around might be to switch from polling every 10 minutes to repository change notification (/git/notifyCommit) from the git repository. Refer to Kohsuke's "polling must die" blog post for more details.

          If you can't find a way to avoid periodic polling, then you could consider creating a single job which polls for changes, and have it perform the notifyCommit only when it detects changes.

          Neither of those techniques is foolproof, but they may resolve your issue.

          Mark Waite added a comment - One work around might be to switch from polling every 10 minutes to repository change notification (/git/notifyCommit) from the git repository. Refer to Kohsuke's " polling must die " blog post for more details. If you can't find a way to avoid periodic polling, then you could consider creating a single job which polls for changes, and have it perform the notifyCommit only when it detects changes. Neither of those techniques is foolproof, but they may resolve your issue.

          Patrick Rose added a comment -

          I don't think we can easily use the first solution due to the fact we have an old non-pipeline build where we're working around the fact that the git plugin didn't poll for changes while a build was on going which meant that we'd end up with executors not being used. I also don't think it'd fix the issue either, because as far as the polling is concerned the build at the HEAD of the branch hasn't been built (which is the issue we're hitting)

          I'll see if I can do a separate workaround via some other Jenkins job

          Patrick Rose added a comment - I don't think we can easily use the first solution due to the fact we have an old non-pipeline build where we're working around the fact that the git plugin didn't poll for changes while a build was on going which meant that we'd end up with executors not being used. I also don't think it'd fix the issue either, because as far as the polling is concerned the build at the HEAD of the branch hasn't been built (which is the issue we're hitting) I'll see if I can do a separate workaround via some other Jenkins job

          Gil Shinar added a comment -

          Any expectations for this but to be resolved?

           

          Thanks

          Gil Shinar added a comment - Any expectations for this but to be resolved?   Thanks

          Mark Waite added a comment -

          gils not unless someone investigates the details and submits a pull request to fix it. It is not on my priority list currently.

          Mark Waite added a comment - gils not unless someone investigates the details and submits a pull request to fix it. It is not on my priority list currently.

            Unassigned Unassigned
            drugcrazed Patrick Rose
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: