Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-30350

Jobs get triggered twice from polling git scm

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • git-plugin
    • None

      We are seeing a weird problem occasionally on our jenkins setup ver 1.627
      git client 1.19
      git plugin 2.4 on ubuntu machine.
      I have a job configured to poll scm every 5 minutes
      sometimes, when there is a change the job gets triggered twice
      once with the relevant changes and the second time says "no changes"

          [JENKINS-30350] Jobs get triggered twice from polling git scm

          Ohad Basan created issue -

          Mark Waite added a comment -

          Can you duplicate the problem with another job? with another repository? with another Jenkins server?

          Can you provide steps to configure the job which shows how to duplicate the problem?

          Mark Waite added a comment - Can you duplicate the problem with another job? with another repository? with another Jenkins server? Can you provide steps to configure the job which shows how to duplicate the problem?

          John Chittum added a comment -

          I can confirm this on multiple projects on a completely separate system. Info:

          Jenkins v. 1.596.2
          Git Plugin 2.3.5
          Git Client 1.17.0

          SCM- Stash 3.11.0

          Builds are automatically kicked off using Stash Webhook to Jenkins. Stash logs show a single commit being created, and a single call going out to Jenkins. Builds are then queued at the same time from the single call from Stash Webhook to Jenkins. Both builds are done with the exact same revision ID

          This has happened with three separate projects, spaced about a week apart. It happens infrequently, as we have builds running regularly. Today was the first time I saw it happen twice on the same project.

          John Chittum added a comment - I can confirm this on multiple projects on a completely separate system. Info: Jenkins v. 1.596.2 Git Plugin 2.3.5 Git Client 1.17.0 SCM- Stash 3.11.0 Builds are automatically kicked off using Stash Webhook to Jenkins. Stash logs show a single commit being created, and a single call going out to Jenkins. Builds are then queued at the same time from the single call from Stash Webhook to Jenkins. Both builds are done with the exact same revision ID This has happened with three separate projects, spaced about a week apart. It happens infrequently, as we have builds running regularly. Today was the first time I saw it happen twice on the same project.

          tsondergaard added a comment -

          Seeing same behaviour as reported in previous comment, except we are using bitbucket webhooks to "git-notify" jenkins. We are running

          jenkins 1.609.2
          git client plugin 1.19.0
          git plugin 2.4.0

          tsondergaard added a comment - Seeing same behaviour as reported in previous comment, except we are using bitbucket webhooks to "git-notify" jenkins. We are running jenkins 1.609.2 git client plugin 1.19.0 git plugin 2.4.0

          I can confirm this problem still exists in Jenkins 2.0. I have scm polling turned on every minute and the second builds starts exactly one minute after the first one. Both builds are triggered by the same git revision. This makes me think that there is something fundamentally wrong with the scm polling implementation. It should probably check if build is already in progress when the next interval is invoked and skip the interval. At the moment it seems that polling interval does not care about the ongoing builds but just starts a new one since the checked revision was not built yet. This also explains why similar issues are reported when build only duplicates twice. That is because the first build finishes fast enough not to trigger the third build.

          Hopefully this gets fixed soon because it's a major bug. Scm poller is basically wasting resources until the first build finishes.

          Klemen Ferjančič added a comment - I can confirm this problem still exists in Jenkins 2.0. I have scm polling turned on every minute and the second builds starts exactly one minute after the first one. Both builds are triggered by the same git revision. This makes me think that there is something fundamentally wrong with the scm polling implementation. It should probably check if build is already in progress when the next interval is invoked and skip the interval. At the moment it seems that polling interval does not care about the ongoing builds but just starts a new one since the checked revision was not built yet. This also explains why similar issues are reported when build only duplicates twice. That is because the first build finishes fast enough not to trigger the third build. Hopefully this gets fixed soon because it's a major bug. Scm poller is basically wasting resources until the first build finishes.
          R. Tyler Croy made changes -
          Workflow Original: JNJira [ 165452 ] New: JNJira + In-Review [ 181978 ]
          Michael Sawyer made changes -
          Link New: This issue relates to JENKINS-37381 [ JENKINS-37381 ]

          Shoo Yoo Yoon added a comment -

          Confirm with Jenkins 2.42 and most of plugin is the latest version at this moment.
          I agree with Klemen Ferjančič . My scenario is following:

          • Jenkins queue is 5 items.
          • each build takes 2 hours.
          • GIT polling for 15 minutes.

          When there is a new git commit, GIT polling will queue a build because all executors is busy.
          After 15 minutes, this build still in queue because all executors is still busy. At this moment, GIT polling will check again then it queue a new build because "Last Built Revision" remains.
          Result: there are two the same build which are queued, in the the Jenkins queue.

          My suggestion:
          GIT polling should check whether a similar build is queued.

          Shoo Yoo Yoon added a comment - Confirm with Jenkins 2.42 and most of plugin is the latest version at this moment. I agree with Klemen Ferjančič . My scenario is following: Jenkins queue is 5 items. each build takes 2 hours. GIT polling for 15 minutes. When there is a new git commit, GIT polling will queue a build because all executors is busy. After 15 minutes, this build still in queue because all executors is still busy. At this moment, GIT polling will check again then it queue a new build because "Last Built Revision" remains. Result: there are two the same build which are queued, in the the Jenkins queue. My suggestion: GIT polling should check whether a similar build is queued.

          Sam Deane added a comment - - edited

          I can also confirm that this is still happening for us, and has been for a long time.

          We’re currently on 2.46.1, but I’ve seen this bug since long before 2.0.

          We are also using git polling, with the interval set to a couple of minutes.

          In one case we have a job which is restricted to one slave - it ends up running twice, sequentially. In another case we have a job that can be picked up by multiple slaves - sometimes more than one of them ends up running it.

          I don’t entirely buy the explanations given above, since our interval is short, but our jobs take 20-30 minutes; and yet we only seem to have each job duplicated once (eg it runs twice), and not multiple times. 

          Sam Deane added a comment - - edited I can also confirm that this is still happening for us, and has been for a long time. We’re currently on 2.46.1, but I’ve seen this bug since long before 2.0. We are also using git polling, with the interval set to a couple of minutes. In one case we have a job which is restricted to one slave - it ends up running twice, sequentially. In another case we have a job that can be picked up by multiple slaves - sometimes more than one of them ends up running it. I don’t entirely buy the explanations given above, since our interval is short, but our jobs take 20-30 minutes; and yet we only seem to have each job duplicated once (eg it runs twice), and not multiple times. 

          Just tested 2.46.2 in Docker container, could not reproduce the bug but I won't claim it is actually fixed.

          1. Add a hello world git repo
          2. Set polling to * * * * *
          3. Build command: /bin/sleep 130

          This makes the build step overextend the polling schedule. Un?-fortunately I did not get a second build on scm change.
          Latest remote head revision on refs/heads/master is: 57b85c7d801eae284d8b1ab9f6fcd1fb3390a15f - already built by 3

          This is in the logs while build 3 is still going.

          I also did some light digging through the code and it seems that when a build is started it is immediately saved with the sha1 revision so I was unable to determine any real flaws in the logic. I was unable to determine how and when build is saved to persistent storage though and I don't know if Jenkins uses some kind of cache. But the code does check if revision has been built already and opening the ongoing build does list the correct sha1 revision.

          The only thing that bothered me is that code and comments refer to builds as if they are "in the past" and already finished, I didn't find any differentiation between finished builds and ongoing builds.

           

          Perhaps if we manage to come up with a simple reproducible scenario someone from Jenkins team can take a look at this, I suspect it is an easy fix once reproduced and narrowed down.

          Klemen Ferjančič added a comment - Just tested 2.46.2 in Docker container, could not reproduce the bug but I won't claim it is actually fixed. Add a hello world git repo Set polling to * * * * * Build command: /bin/sleep 130 This makes the build step overextend the polling schedule. Un?-fortunately I did not get a second build on scm change. Latest remote head revision on refs/heads/master is: 57b85c7d801eae284d8b1ab9f6fcd1fb3390a15f - already built by 3 This is in the logs while build 3 is still going. I also did some light digging through the code and it seems that when a build is started it is immediately saved with the sha1 revision so I was unable to determine any real flaws in the logic. I was unable to determine how and when build is saved to persistent storage though and I don't know if Jenkins uses some kind of cache. But the code does check if revision has been built already and opening the ongoing build does list the correct sha1 revision. The only thing that bothered me is that code and comments refer to builds as if they are "in the past" and already finished, I didn't find any differentiation between finished builds and ongoing builds.   Perhaps if we manage to come up with a simple reproducible scenario someone from Jenkins team can take a look at this, I suspect it is an easy fix once reproduced and narrowed down.

            Unassigned Unassigned
            ohadbasan Ohad Basan
            Votes:
            29 Vote for this issue
            Watchers:
            37 Start watching this issue

              Created:
              Updated: