Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38443

Pollscm trigger runs the job twice for the same commit

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Reopened (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: core, pipeline
    • Labels:
      None
    • Environment:
      Jenkins 2.7.4
    • Similar Issues:

      Description

      We've got the job with pollscm trigger enabled for two repositories.
      The job is triggered twice with one minute interval, though both commit hashes are the same.

      Pollscm logs look like this:

      Started on Sep 22, 2016 9:47:00 AM
      Using strategy: Default
      [poll] Last Built Revision: Revision <revA> (go/master)
      ...
      [poll] Latest remote head revision on refs/heads/master is: <revA> - already built by 1
      Using strategy: Default
      ...
      [poll] Latest remote head revision on refs/heads/master is: <revB>
      Done. Took 1 sec
      Changes found
      
      Started on Sep 22, 2016 9:48:00 AM
      Using strategy: Default
      [poll] Last Built Revision: Revision <revA> (go/master)
      ...
      [poll] Latest remote head revision on refs/heads/master is: <revA> - already built by 2
      Using strategy: Default
      ...
      [poll] Latest remote head revision on refs/heads/master is: <revB>
      Done. Took 0.65 sec
      Changes found
      

      <revA, revB> pair is the same in two job runs.

        Attachments

          Issue Links

            Activity

            Hide
            ifn Shat Shabaev added a comment -

            Jenkins will just queue another build, as what has been found hasn't been built yet

            As far as I understand, queuing the job repeatedly for the same commit is ok. Strange though. Thanks.

            Show
            ifn Shat Shabaev added a comment - Jenkins will just queue another build, as what has been found hasn't been built yet As far as I understand, queuing the job repeatedly for the same commit is ok. Strange though. Thanks.
            Hide
            sonneveldsmartward Nick Sonneveld added a comment -

            Linking to related JENKINS-39092 . The difference is that no polling interval is set, only change notifications

            Show
            sonneveldsmartward Nick Sonneveld added a comment - Linking to related JENKINS-39092 . The difference is that no polling interval is set, only change notifications
            Hide
            timblaktu Tim Black added a comment - - edited

            We encountered this problem in a dramatic fashion last night in a branch of a multibranch pipeline job whose pipelines did NOT set disableConcurrentBuilds() option, in a jenkins cluster containing:

            • 1 Controller (Zero Executors)
            • 2 Build Agents (2 Executors Each)

            A single branch job using `pollSCM('H/5 * * * *')` trigger entered a feedback look where every 5 minutes it was triggered (for the same revision) even though there were no SCM changes.

            Reading Daniel Beck's comment, I see that the pollSCM mechanism didn't see that this specific revision was built because these builds were waiting for an executor for long periods of time, so the SCM polling triggers would keep happening even if there were no changes.

            The executor stall can be considered the "root cause", and we can attempt to fix this with infrastrucutre, however, it would be nice to have a way to prevent pollSCM triggers from "running amok" in this way.

            Yes, we could disableConcurrentBuilds() in all branch job pipelines, but pushing such a change to all branches is very cumbersome. It would be nice to be able to specify this in JobDSL. But as I discuss Here multibranch pipelines do not appear to support disabling Concurrent builds via jobDSL. As I describe, rate limiting is not appropriate here, since it requires the jobDSL to know something about build time. We want a simple "build serialization" that is provided by disableConcurrentBuilds in pipeline, without requiring every Jenkinsfile to specify it.

            Show
            timblaktu Tim Black added a comment - - edited We encountered this problem in a dramatic fashion last night in a branch of a multibranch pipeline job whose pipelines did NOT set disableConcurrentBuilds() option, in a jenkins cluster containing: 1 Controller (Zero Executors) 2 Build Agents (2 Executors Each) A single branch job using `pollSCM('H/5 * * * *')` trigger entered a feedback look where every 5 minutes it was triggered (for the same revision) even though there were no SCM changes. Reading Daniel Beck 's comment, I see that the pollSCM mechanism didn't see that this specific revision was built because these builds were waiting for an executor for long periods of time, so the SCM polling triggers would keep happening even if there were no changes. The executor stall can be considered the "root cause", and we can attempt to fix this with infrastrucutre, however, it would be nice to have a way to prevent pollSCM triggers from "running amok" in this way. Yes, we could disableConcurrentBuilds() in all branch job pipelines, but pushing such a change to all branches is very cumbersome. It would be nice to be able to specify this in JobDSL. But as I discuss  Here  multibranch pipelines do not appear to support disabling Concurrent builds via jobDSL. As I describe, rate limiting is not appropriate here, since it requires the jobDSL to know something about build time. We want a simple "build serialization" that is provided by disableConcurrentBuilds in pipeline, without requiring every Jenkinsfile to specify it.
            Hide
            michaelmess Michael Meß added a comment -

            As I wrote here
            probably it might help to poll the SCM only when there is a free executor ready to immediately take over the job to be started. Then we do not end up queueing up jobs indefinitely, however it might not avoid in all cases that a job is started twice for a change, but it should avoid that the pollSCM will run amok triggering more and more builds.

            Show
            michaelmess Michael Meß added a comment - As I wrote here probably it might help to poll the SCM only when there is a free executor ready to immediately take over the job to be started. Then we do not end up queueing up jobs indefinitely, however it might not avoid in all cases that a job is started twice for a change, but it should avoid that the pollSCM will run amok triggering more and more builds.
            Hide
            michaelmess Michael Meß added a comment -

            When there is a change pollSCM could also remember the state of the last polling, when it was "No changes" then a job may be even triggered when no executor is available as then it is sure that this is a fresh change, not yet with a build in progress.
            Then no new jobs are triggered until the previously started job handled the change and brought the state back to "No changes".
            This would also allow parallel builds as it is assured that the last jobs has checked out already before a new one gets triggered.

            Show
            michaelmess Michael Meß added a comment - When there is a change pollSCM could also remember the state of the last polling, when it was "No changes" then a job may be even triggered when no executor is available as then it is sure that this is a fresh change, not yet with a build in progress. Then no new jobs are triggered until the previously started job handled the change and brought the state back to "No changes". This would also allow parallel builds as it is assured that the last jobs has checked out already before a new one gets triggered.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              ifn Shat Shabaev
              Votes:
              5 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated: