Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-30350

Jobs get triggered twice from polling git scm

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • git-plugin
    • None

      We are seeing a weird problem occasionally on our jenkins setup ver 1.627
      git client 1.19
      git plugin 2.4 on ubuntu machine.
      I have a job configured to poll scm every 5 minutes
      sometimes, when there is a change the job gets triggered twice
      once with the relevant changes and the second time says "no changes"

          [JENKINS-30350] Jobs get triggered twice from polling git scm

          Mark Waite added a comment -

          Can you duplicate the problem with another job? with another repository? with another Jenkins server?

          Can you provide steps to configure the job which shows how to duplicate the problem?

          Mark Waite added a comment - Can you duplicate the problem with another job? with another repository? with another Jenkins server? Can you provide steps to configure the job which shows how to duplicate the problem?

          John Chittum added a comment -

          I can confirm this on multiple projects on a completely separate system. Info:

          Jenkins v. 1.596.2
          Git Plugin 2.3.5
          Git Client 1.17.0

          SCM- Stash 3.11.0

          Builds are automatically kicked off using Stash Webhook to Jenkins. Stash logs show a single commit being created, and a single call going out to Jenkins. Builds are then queued at the same time from the single call from Stash Webhook to Jenkins. Both builds are done with the exact same revision ID

          This has happened with three separate projects, spaced about a week apart. It happens infrequently, as we have builds running regularly. Today was the first time I saw it happen twice on the same project.

          John Chittum added a comment - I can confirm this on multiple projects on a completely separate system. Info: Jenkins v. 1.596.2 Git Plugin 2.3.5 Git Client 1.17.0 SCM- Stash 3.11.0 Builds are automatically kicked off using Stash Webhook to Jenkins. Stash logs show a single commit being created, and a single call going out to Jenkins. Builds are then queued at the same time from the single call from Stash Webhook to Jenkins. Both builds are done with the exact same revision ID This has happened with three separate projects, spaced about a week apart. It happens infrequently, as we have builds running regularly. Today was the first time I saw it happen twice on the same project.

          tsondergaard added a comment -

          Seeing same behaviour as reported in previous comment, except we are using bitbucket webhooks to "git-notify" jenkins. We are running

          jenkins 1.609.2
          git client plugin 1.19.0
          git plugin 2.4.0

          tsondergaard added a comment - Seeing same behaviour as reported in previous comment, except we are using bitbucket webhooks to "git-notify" jenkins. We are running jenkins 1.609.2 git client plugin 1.19.0 git plugin 2.4.0

          I can confirm this problem still exists in Jenkins 2.0. I have scm polling turned on every minute and the second builds starts exactly one minute after the first one. Both builds are triggered by the same git revision. This makes me think that there is something fundamentally wrong with the scm polling implementation. It should probably check if build is already in progress when the next interval is invoked and skip the interval. At the moment it seems that polling interval does not care about the ongoing builds but just starts a new one since the checked revision was not built yet. This also explains why similar issues are reported when build only duplicates twice. That is because the first build finishes fast enough not to trigger the third build.

          Hopefully this gets fixed soon because it's a major bug. Scm poller is basically wasting resources until the first build finishes.

          Klemen Ferjančič added a comment - I can confirm this problem still exists in Jenkins 2.0. I have scm polling turned on every minute and the second builds starts exactly one minute after the first one. Both builds are triggered by the same git revision. This makes me think that there is something fundamentally wrong with the scm polling implementation. It should probably check if build is already in progress when the next interval is invoked and skip the interval. At the moment it seems that polling interval does not care about the ongoing builds but just starts a new one since the checked revision was not built yet. This also explains why similar issues are reported when build only duplicates twice. That is because the first build finishes fast enough not to trigger the third build. Hopefully this gets fixed soon because it's a major bug. Scm poller is basically wasting resources until the first build finishes.

          Shoo Yoo Yoon added a comment -

          Confirm with Jenkins 2.42 and most of plugin is the latest version at this moment.
          I agree with Klemen Ferjančič . My scenario is following:

          • Jenkins queue is 5 items.
          • each build takes 2 hours.
          • GIT polling for 15 minutes.

          When there is a new git commit, GIT polling will queue a build because all executors is busy.
          After 15 minutes, this build still in queue because all executors is still busy. At this moment, GIT polling will check again then it queue a new build because "Last Built Revision" remains.
          Result: there are two the same build which are queued, in the the Jenkins queue.

          My suggestion:
          GIT polling should check whether a similar build is queued.

          Shoo Yoo Yoon added a comment - Confirm with Jenkins 2.42 and most of plugin is the latest version at this moment. I agree with Klemen Ferjančič . My scenario is following: Jenkins queue is 5 items. each build takes 2 hours. GIT polling for 15 minutes. When there is a new git commit, GIT polling will queue a build because all executors is busy. After 15 minutes, this build still in queue because all executors is still busy. At this moment, GIT polling will check again then it queue a new build because "Last Built Revision" remains. Result: there are two the same build which are queued, in the the Jenkins queue. My suggestion: GIT polling should check whether a similar build is queued.

          Sam Deane added a comment - - edited

          I can also confirm that this is still happening for us, and has been for a long time.

          We’re currently on 2.46.1, but I’ve seen this bug since long before 2.0.

          We are also using git polling, with the interval set to a couple of minutes.

          In one case we have a job which is restricted to one slave - it ends up running twice, sequentially. In another case we have a job that can be picked up by multiple slaves - sometimes more than one of them ends up running it.

          I don’t entirely buy the explanations given above, since our interval is short, but our jobs take 20-30 minutes; and yet we only seem to have each job duplicated once (eg it runs twice), and not multiple times. 

          Sam Deane added a comment - - edited I can also confirm that this is still happening for us, and has been for a long time. We’re currently on 2.46.1, but I’ve seen this bug since long before 2.0. We are also using git polling, with the interval set to a couple of minutes. In one case we have a job which is restricted to one slave - it ends up running twice, sequentially. In another case we have a job that can be picked up by multiple slaves - sometimes more than one of them ends up running it. I don’t entirely buy the explanations given above, since our interval is short, but our jobs take 20-30 minutes; and yet we only seem to have each job duplicated once (eg it runs twice), and not multiple times. 

          Just tested 2.46.2 in Docker container, could not reproduce the bug but I won't claim it is actually fixed.

          1. Add a hello world git repo
          2. Set polling to * * * * *
          3. Build command: /bin/sleep 130

          This makes the build step overextend the polling schedule. Un?-fortunately I did not get a second build on scm change.
          Latest remote head revision on refs/heads/master is: 57b85c7d801eae284d8b1ab9f6fcd1fb3390a15f - already built by 3

          This is in the logs while build 3 is still going.

          I also did some light digging through the code and it seems that when a build is started it is immediately saved with the sha1 revision so I was unable to determine any real flaws in the logic. I was unable to determine how and when build is saved to persistent storage though and I don't know if Jenkins uses some kind of cache. But the code does check if revision has been built already and opening the ongoing build does list the correct sha1 revision.

          The only thing that bothered me is that code and comments refer to builds as if they are "in the past" and already finished, I didn't find any differentiation between finished builds and ongoing builds.

           

          Perhaps if we manage to come up with a simple reproducible scenario someone from Jenkins team can take a look at this, I suspect it is an easy fix once reproduced and narrowed down.

          Klemen Ferjančič added a comment - Just tested 2.46.2 in Docker container, could not reproduce the bug but I won't claim it is actually fixed. Add a hello world git repo Set polling to * * * * * Build command: /bin/sleep 130 This makes the build step overextend the polling schedule. Un?-fortunately I did not get a second build on scm change. Latest remote head revision on refs/heads/master is: 57b85c7d801eae284d8b1ab9f6fcd1fb3390a15f - already built by 3 This is in the logs while build 3 is still going. I also did some light digging through the code and it seems that when a build is started it is immediately saved with the sha1 revision so I was unable to determine any real flaws in the logic. I was unable to determine how and when build is saved to persistent storage though and I don't know if Jenkins uses some kind of cache. But the code does check if revision has been built already and opening the ongoing build does list the correct sha1 revision. The only thing that bothered me is that code and comments refer to builds as if they are "in the past" and already finished, I didn't find any differentiation between finished builds and ongoing builds.   Perhaps if we manage to come up with a simple reproducible scenario someone from Jenkins team can take a look at this, I suspect it is an easy fix once reproduced and narrowed down.

          Jon B added a comment -

          I have also encountered this problem in my jenkins installation. This is particularly scary in my case due to what happens after a pipeline starts. I hope someone can save the day.

          Jon B added a comment - I have also encountered this problem in my jenkins installation. This is particularly scary in my case due to what happens after a pipeline starts. I hope someone can save the day.

          This issue exists not only with polling git, but also with polling svn.

           

          Manasa Godavarthi added a comment - This issue exists not only with polling git, but also with polling svn.  

          Christian added a comment -

          I've noticed the issue on multibranch pipelines that use polling. Say a build for a branch takes 10 minutes to run and polling occurs every 2 minutes. If branch A gets committed then branch B gets committed, Jenkins will keep duplicating jobs for branch B every two minutes. This is because Jenkins checks branch B to see if the latest has been built and if not schedules a build, regardless of if one has already been scheduled.

          cen is correct that Jenkins saves the sha1 when the build is started, but if the build for that branch cannot be started yet because another build is running then multiple builds will keep queuing up. Could the sha1 be saved once the build is queued? Could there be a way to block Jenkins from queuing more than 1 build from the same branch at a time? Is there a better way to prevent duplicate builds from being queued?

          Christian added a comment - I've noticed the issue on multibranch pipelines that use polling. Say a build for a branch takes 10 minutes to run and polling occurs every 2 minutes. If branch A gets committed then branch B gets committed, Jenkins will keep duplicating jobs for branch B every two minutes. This is because Jenkins checks branch B to see if the latest has been built and if not schedules a build, regardless of if one has already been scheduled. cen is correct that Jenkins saves the sha1 when the build is started, but if the build for that branch cannot be started yet because another build is running then multiple builds will keep queuing up. Could the sha1 be saved once the build is queued? Could there be a way to block Jenkins from queuing more than 1 build from the same branch at a time? Is there a better way to prevent duplicate builds from being queued?

          Marty S added a comment -

          I'm having the exact same scenario as gonz0_12.

          Does anyone have a workaround yet?

          Marty S added a comment - I'm having the exact same scenario as gonz0_12 . Does anyone have a workaround yet?

          Christian added a comment - - edited

          Unfortunately the only thing that seems to work is either switching to hooks (which we can't do for reasons) or increase the time between polling. Of course, by increasing the time between polling, your builds will trigger less frequently. We have ours at about 1 hour, which means there could be a delay of up to 1 hour between the time someone commits and the time Jenkins picks it up. I really wish someone would fix this bug, but seeing as it has been around for over 3 years I don't think it will happen.

          Christian added a comment - - edited Unfortunately the only thing that seems to work is either switching to hooks (which we can't do for reasons) or increase the time between polling. Of course, by increasing the time between polling, your builds will trigger less frequently. We have ours at about 1 hour, which means there could be a delay of up to 1 hour between the time someone commits and the time Jenkins picks it up. I really wish someone would fix this bug, but seeing as it has been around for over 3 years I don't think it will happen.

          We were seeing the same problem happen consistently for our project, every time a change would be commited to git, two builds were being triggered with our job setup to poll for new changes every 2 minutes.

          But checking out our repository takes more than 4 minutes. So i assume the commit id is being stored right after the checkout has finished, which is why it was only 1 build that was always being queued up.

          I would recommend setting the poll schedule to a value that will definitely allow for a full repository checkout before polling again. In our case, 6 minutes seems to be doing the trick.

          Charles Clement added a comment - We were seeing the same problem happen consistently for our project, every time a change would be commited to git, two builds were being triggered with our job setup to poll for new changes every 2 minutes. But checking out our repository takes more than 4 minutes. So i assume the commit id is being stored right after the checkout has finished, which is why it was only 1 build that was always being queued up. I would recommend setting the poll schedule to a value that will definitely allow for a full repository checkout before polling again. In our case, 6 minutes seems to be doing the trick.

          cen We are also facing the same issue. Upon investigating this can reproducible, When you close the Source branch after the Merge request.

          Jenkins triggers multiple builds for the each push

          Maheshkumar Jeyaraj added a comment - cen We are also facing the same issue. Upon investigating this can reproducible, When you close the Source branch after the Merge request. Jenkins triggers multiple builds for the each push

          nikhil nanal added a comment -

          I am facing this issue as well with Multibranch pipelines on Jenkins 2.164 LTS and exactly as described by Christian above, my polling interval is 2 minutes and builds take longer. when there is pending build it gets retriggerred multiple ( arbitrary number) of times 

          nikhil nanal added a comment - I am facing this issue as well with Multibranch pipelines on Jenkins 2.164 LTS and exactly as described by Christian above, my polling interval is 2 minutes and builds take longer. when there is pending build it gets retriggerred multiple ( arbitrary number) of times 

          Mohamed Afsal added a comment -

          Hi Team, 

           

          Any update on this, I'm also facing the issue with Multibranch pipelines on Jenkins 2.22.1LTS.
          Dublicate Job shown as 

          Branch API version : 2.5.5
          Git plugin version : 4.2.2

          Mohamed Afsal added a comment - Hi Team,    Any update on this, I'm also facing the issue with Multibranch pipelines on Jenkins 2.22.1LTS. Dublicate Job shown as  Started by an SCM change  (22 times) Branch API version : 2.5.5 Git plugin version :  4.2.2

          Mark Waite added a comment -

          No update to offer afsal. Have you checked the alternatives offered in earlier comments in the issue?

          Mark Waite added a comment - No update to offer afsal . Have you checked the alternatives offered in earlier comments in the issue?

          Mohamed Afsal added a comment -

          markewaite Yes, tried but still issue persist

          Mohamed Afsal added a comment - markewaite Yes, tried but still issue persist

          Marwan added a comment -

          Hi, I encountered this problem and I found a workaround. we are using Github hooks and Jenkins.

          I wanted to trigger a Jenkins job for branches [let's say master, develop, and test]that I want to specify to trigger a job on push event. 

           

          for every push on one of those branches in the list, the job was triggered a couple of times.

          my workaround was to create a Jenkins job that triggers for every one of those branches alone

          in my Jenkins git "Branch Specifier (blank for 'any')"  I put only origin/master

          another job origin/develop and so on.

           

          what I noticed that sometimes it also trigger the same branches multiple times( after adding a filter in "polling ignores commits from certain users"

          the solution was to enable "Quiet period" in the advanced option under general and put the value of 10 seconds.

           

          TL'DR 

          1) create a job for each branch to trigger

          2) enable Quiet period in advanced option under general and put to a higher value than the default(5 is the default in my company)

           

           

          I did not check but maybe doing number 2 only can solve the problem.

           

           

          Marwan added a comment - Hi, I encountered this problem and I found a workaround. we are using Github hooks and Jenkins. I wanted to trigger a Jenkins job for branches [let's say master, develop, and test] that I want to specify to trigger a job on push event.    for every push on one of those branches in the list, the job was triggered a couple of times. my workaround was to create a Jenkins job that triggers for every one of those branches alone in my Jenkins git "Branch Specifier (blank for 'any')"  I put only origin/master another job origin/develop and so on.   what I noticed that sometimes it also trigger the same branches multiple times( after adding a filter in "polling ignores commits from certain users" the solution was to enable "Quiet period" in the advanced option under general and put the value of 10 seconds.   TL'DR  1) create a job for each branch to trigger 2) enable Quiet period in advanced option under general and put to a higher value than the default(5 is the default in my company)     I did not check but maybe doing number 2 only can solve the problem.    

          Vladislav Ponomarev added a comment - - edited

          It looks like the reason mentioned here in comment-364906 is exactly correct.

          Our (free style) build job has the build trigger configured as "GitHub hook trigger for GITScm polling".

          The moment we enabled "Automatically delete head branches" under our GitHub repo' settings, we started seeing the master branch builds triggered twice.

          I checked under GitHub webhook page - there are three virtually simultaneous events when the pull request is merged and the branch is automatically deleted:

          • for the master branch - "merge" event
          • for a pull request - "closed" event
          • for a head branch - "delete" event

          I presume the latter is the new one which causes an extra master build to be triggered - omitting the reason why it triggers a build for master and not for a  [deleted] head branch.

          For now we worked around it by stopping the build if the value of GIT_COMMIT and  GIT_PREVIOUS_COMMIT environment variables are different.

          Looking forward to see a proper fix though.

           

          Vladislav Ponomarev added a comment - - edited It looks like the reason mentioned here in  comment-364906  is exactly correct. Our (free style) build job has the build trigger configured as "GitHub hook trigger for GITScm polling". The moment we enabled "Automatically delete head branches" under our GitHub repo' settings, we started seeing the master branch builds triggered twice. I checked under GitHub webhook page - there are three virtually simultaneous events when the pull request is merged and the branch is automatically deleted: for the master branch - "merge" event for a pull request - "closed" event for a head branch - "delete" event I presume the latter is the new one which causes an extra master build to be triggered - omitting the reason why it triggers a build for master and not for a  [deleted] head branch. For now we worked around it by stopping the build if the value of GIT_COMMIT and  GIT_PREVIOUS_COMMIT environment variables are different. Looking forward to see a proper fix though.  

          Still seeing this issue in 2022. I can't believe it!

          Menasheh Peromsik added a comment - Still seeing this issue in 2022. I can't believe it!

            Unassigned Unassigned
            ohadbasan Ohad Basan
            Votes:
            29 Vote for this issue
            Watchers:
            37 Start watching this issue

              Created:
              Updated: