Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38443

Pollscm trigger runs the job twice for the same commit

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Reopened (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: core, pipeline
    • Labels:
      None
    • Environment:
      Jenkins 2.7.4
    • Similar Issues:

      Description

      We've got the job with pollscm trigger enabled for two repositories.
      The job is triggered twice with one minute interval, though both commit hashes are the same.

      Pollscm logs look like this:

      Started on Sep 22, 2016 9:47:00 AM
      Using strategy: Default
      [poll] Last Built Revision: Revision <revA> (go/master)
      ...
      [poll] Latest remote head revision on refs/heads/master is: <revA> - already built by 1
      Using strategy: Default
      ...
      [poll] Latest remote head revision on refs/heads/master is: <revB>
      Done. Took 1 sec
      Changes found
      
      Started on Sep 22, 2016 9:48:00 AM
      Using strategy: Default
      [poll] Last Built Revision: Revision <revA> (go/master)
      ...
      [poll] Latest remote head revision on refs/heads/master is: <revA> - already built by 2
      Using strategy: Default
      ...
      [poll] Latest remote head revision on refs/heads/master is: <revB>
      Done. Took 0.65 sec
      Changes found
      

      <revA, revB> pair is the same in two job runs.

        Attachments

          Issue Links

            Activity

            Hide
            sonneveldsmartward Nick Sonneveld added a comment - - edited

            When I had a pipeline job's PollSCM property configured with "@daily", I witnessed 7 builds triggered for the same changeset. All were "Started by an SCM change". Is it possible that PollSCM isn't checking for a running build with that changeset? Although why would it attempt to build more than once a day?

            Show
            sonneveldsmartward Nick Sonneveld added a comment - - edited When I had a pipeline job's PollSCM property configured with "@daily", I witnessed 7 builds triggered for the same changeset. All were "Started by an SCM change". Is it possible that PollSCM isn't checking for a running build with that changeset? Although why would it attempt to build more than once a day?
            Hide
            ifn Shat Shabaev added a comment -

            It seems that setting concurrent to false prevents from rerunning the job for the same commit.
            I'm not sure that it is connected with Pipeline plugin.

            Show
            ifn Shat Shabaev added a comment - It seems that setting concurrent to false prevents from rerunning the job for the same commit. I'm not sure that it is connected with Pipeline plugin.
            Hide
            danielbeck Daniel Beck added a comment -

            You're probably polling too often. Jenkins looks at past builds to determine what has already been built, and if there's not enough time to start a build and checkout from SCM to record what the current state is, Jenkins will just queue another build, as what has been found hasn't been built yet.

            Show
            danielbeck Daniel Beck added a comment - You're probably polling too often. Jenkins looks at past builds to determine what has already been built, and if there's not enough time to start a build and checkout from SCM to record what the current state is, Jenkins will just queue another build, as what has been found hasn't been built yet.
            Hide
            sonneveldsmartward Nick Sonneveld added a comment - - edited

            As far as I know, we're only polling "daily" per branch, configured via pollSCM. We do have a mercurial hook that GETs "/mercurial/notifyCommit" for each commit but I feel that shouldn't affect things as that is just notifying jenkins that "something" changed. The parent multibranch job does not have any polling configured.

            Show
            sonneveldsmartward Nick Sonneveld added a comment - - edited As far as I know, we're only polling "daily" per branch, configured via pollSCM. We do have a mercurial hook that GETs "/mercurial/notifyCommit" for each commit but I feel that shouldn't affect things as that is just notifying jenkins that "something" changed. The parent multibranch job does not have any polling configured.
            Hide
            ifn Shat Shabaev added a comment -

            Jenkins will just queue another build, as what has been found hasn't been built yet

            As far as I understand, queuing the job repeatedly for the same commit is ok. Strange though. Thanks.

            Show
            ifn Shat Shabaev added a comment - Jenkins will just queue another build, as what has been found hasn't been built yet As far as I understand, queuing the job repeatedly for the same commit is ok. Strange though. Thanks.
            Hide
            sonneveldsmartward Nick Sonneveld added a comment -

            Linking to related JENKINS-39092 . The difference is that no polling interval is set, only change notifications

            Show
            sonneveldsmartward Nick Sonneveld added a comment - Linking to related JENKINS-39092 . The difference is that no polling interval is set, only change notifications
            Hide
            timblaktu Tim Black added a comment - - edited

            We encountered this problem in a dramatic fashion last night in a branch of a multibranch pipeline job whose pipelines did NOT set disableConcurrentBuilds() option, in a jenkins cluster containing:

            • 1 Controller (Zero Executors)
            • 2 Build Agents (2 Executors Each)

            A single branch job using `pollSCM('H/5 * * * *')` trigger entered a feedback look where every 5 minutes it was triggered (for the same revision) even though there were no SCM changes.

            Reading Daniel Beck's comment, I see that the pollSCM mechanism didn't see that this specific revision was built because these builds were waiting for an executor for long periods of time, so the SCM polling triggers would keep happening even if there were no changes.

            The executor stall can be considered the "root cause", and we can attempt to fix this with infrastrucutre, however, it would be nice to have a way to prevent pollSCM triggers from "running amok" in this way.

            Yes, we could disableConcurrentBuilds() in all branch job pipelines, but pushing such a change to all branches is very cumbersome. It would be nice to be able to specify this in JobDSL. But as I discuss Here multibranch pipelines do not appear to support disabling Concurrent builds via jobDSL. As I describe, rate limiting is not appropriate here, since it requires the jobDSL to know something about build time. We want a simple "build serialization" that is provided by disableConcurrentBuilds in pipeline, without requiring every Jenkinsfile to specify it.

            Show
            timblaktu Tim Black added a comment - - edited We encountered this problem in a dramatic fashion last night in a branch of a multibranch pipeline job whose pipelines did NOT set disableConcurrentBuilds() option, in a jenkins cluster containing: 1 Controller (Zero Executors) 2 Build Agents (2 Executors Each) A single branch job using `pollSCM('H/5 * * * *')` trigger entered a feedback look where every 5 minutes it was triggered (for the same revision) even though there were no SCM changes. Reading Daniel Beck 's comment, I see that the pollSCM mechanism didn't see that this specific revision was built because these builds were waiting for an executor for long periods of time, so the SCM polling triggers would keep happening even if there were no changes. The executor stall can be considered the "root cause", and we can attempt to fix this with infrastrucutre, however, it would be nice to have a way to prevent pollSCM triggers from "running amok" in this way. Yes, we could disableConcurrentBuilds() in all branch job pipelines, but pushing such a change to all branches is very cumbersome. It would be nice to be able to specify this in JobDSL. But as I discuss  Here  multibranch pipelines do not appear to support disabling Concurrent builds via jobDSL. As I describe, rate limiting is not appropriate here, since it requires the jobDSL to know something about build time. We want a simple "build serialization" that is provided by disableConcurrentBuilds in pipeline, without requiring every Jenkinsfile to specify it.
            Hide
            michaelmess Michael Meß added a comment -

            As I wrote here
            probably it might help to poll the SCM only when there is a free executor ready to immediately take over the job to be started. Then we do not end up queueing up jobs indefinitely, however it might not avoid in all cases that a job is started twice for a change, but it should avoid that the pollSCM will run amok triggering more and more builds.

            Show
            michaelmess Michael Meß added a comment - As I wrote here probably it might help to poll the SCM only when there is a free executor ready to immediately take over the job to be started. Then we do not end up queueing up jobs indefinitely, however it might not avoid in all cases that a job is started twice for a change, but it should avoid that the pollSCM will run amok triggering more and more builds.
            Hide
            michaelmess Michael Meß added a comment -

            When there is a change pollSCM could also remember the state of the last polling, when it was "No changes" then a job may be even triggered when no executor is available as then it is sure that this is a fresh change, not yet with a build in progress.
            Then no new jobs are triggered until the previously started job handled the change and brought the state back to "No changes".
            This would also allow parallel builds as it is assured that the last jobs has checked out already before a new one gets triggered.

            Show
            michaelmess Michael Meß added a comment - When there is a change pollSCM could also remember the state of the last polling, when it was "No changes" then a job may be even triggered when no executor is available as then it is sure that this is a fresh change, not yet with a build in progress. Then no new jobs are triggered until the previously started job handled the change and brought the state back to "No changes". This would also allow parallel builds as it is assured that the last jobs has checked out already before a new one gets triggered.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              ifn Shat Shabaev
              Votes:
              5 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated: