Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40722

Poll SCM results in a Pipeline run every interval (unnecessarily)

      My Jenkinsfile includes an SCMTrigger pipelineTrigger for polling SCM for changes:

      properties([
          [$class: 'jenkins.model.BuildDiscarderProperty', strategy: [$class: 'LogRotator', numToKeepStr: '96']],
          pipelineTriggers([[$class:"SCMTrigger", scmpoll_spec:"H/10 * * * *"]]),
      ])
      

      This repository is used via a GitHub Organization Folder on ci.jenkins.io, and seems to believe that changes are being made every ten minutes (per the scmpoll_spec above, despite no such changes being made.

      See screenshots for more context

      This results in an aggressive waste of resources in our Jenkins environment

        1. scmpoll_136.png
          scmpoll_136.png
          43 kB
        2. scmpoll_137.png
          scmpoll_137.png
          43 kB
        3. scmpoll_history.png
          scmpoll_history.png
          26 kB

          [JENKINS-40722] Poll SCM results in a Pipeline run every interval (unnecessarily)

          Daniel Beck added a comment -

          Polling intervals shorter than the build duration could the the problem here.

          Daniel Beck added a comment - Polling intervals shorter than the build duration could the the problem here.

          You should not use polling triggers when a job is part of an Organization Folder. The Organization Folder will trigger builds for you as it controls the SCM revision that is being checked out when you run a build (in other words, even when the polling triggers a build, that build will be for the last revision detected by the MultiBranch project not the revision that polling has detected)

          Stephen Connolly added a comment - You should not use polling triggers when a job is part of an Organization Folder. The Organization Folder will trigger builds for you as it controls the SCM revision that is being checked out when you run a build (in other words, even when the polling triggers a build, that build will be for the last revision detected by the MultiBranch project not the revision that polling has detected)

          Daniel Beck added a comment -

          Besides making no sense, is this documented anywhere?

          Daniel Beck added a comment - Besides making no sense, is this documented anywhere?

          R. Tyler Croy added a comment - - edited

          stephenconnolly, I generally use organization folders as a "multibranch on steroids", in that I don't want to set up Pipeline projects in Jenkins for each repository and the organization folder let's me automate that aspect of it nicely.

          If that's not something hrmpw wants to encourage, while I disagree with it, I think this behavior should be disabled, warned, or errored upon. IMHO it's a poor user experience to have org folders do something unexpected without ever setting that expectation that the behavior is "wrong".

          R. Tyler Croy added a comment - - edited stephenconnolly , I generally use organization folders as a "multibranch on steroids", in that I don't want to set up Pipeline projects in Jenkins for each repository and the organization folder let's me automate that aspect of it nicely. If that's not something hrmpw wants to encourage, while I disagree with it, I think this behavior should be disabled, warned, or errored upon. IMHO it's a poor user experience to have org folders do something unexpected without ever setting that expectation that the behavior is "wrong".

          Patrick Wolf added a comment -

          I don't understand stephenconnolly's point here.

          If I'm not using a webhook to trigger builds then polling is probably the next best way to get updates. That has always been the case in Jenkins.

          My understanding is that regardless of a webhook or polling trigger the multibranch job will by default ("checkout scm") checkout the head of that branch and run the job based on the latest and greatest in the repo. Are you saying that isn't true, Stephen?

          Patrick Wolf added a comment - I don't understand stephenconnolly 's point here. If I'm not using a webhook to trigger builds then polling is probably the next best way to get updates. That has always been the case in Jenkins. My understanding is that regardless of a webhook or polling trigger the multibranch job will by default ("checkout scm") checkout the head of that branch and run the job based on the latest and greatest in the repo. Are you saying that isn't true, Stephen?

          The way Multibranch works is that every time it scans the repository, for each branch it gets the latest revision.

          Then it re-configures each branches job to build that revision only and triggers a build if the previous configuration was for a different revision.

          If you want SCM polling, you should have the scan run at the polling interval as the scan will do one efficient scan for all branches rather than having each branch scan.

          If you want some branches to not build so often, you should have a rate limit branch property on those branches, etc

          Now for an organization folder, we should probably provide a mechanism for defining have repository specific properties (analogous to branch specific properties) so that the foo repository can poll every 10 minutes while all other repositories poll daily or something like that.

          The other thing I want to investigate is polling the events for GitHub, e.g. https://developer.github.com/v3/activity/events/#list-repository-events and https://developer.github.com/v3/activity/events/#list-events-for-an-organization as those should be a much more efficient polling algorithm for the more delta aware modern SCM API event subsystem. Probably this would work as a global setting for the GitHub server where you define the server as either being web-hook push events or being api poll-events (here is the frequency) and then we can have an admin level log of the polling of events which would then get dispatched to the event listeners. (The issue I see with poll-events is that there are a number of events that GitHub says in the documentation are only delivered by web-hook... if one of those is a critical event then we are SooL)

          OK, so back to this issue.

          • It is necessary for Multibranch to configure each branch job's SCM to be pinned to the revision to build
          • It is confusing to users that clicking build on a branch job will not build the latest code, but rather rebuild the revision

          The user confusion can probably be cleared up by tweaking the branch project's actions.

          Hack Solution: If we replace(or rename) the "Build Now" button with a "Rebuild Now" button that would clarify for the user that we will be rebuilding the same revision not building the latest revision. Additionally we could inject a "Build Now" button that uses SCM API to query the head revision of the branch and reconfigure the SCM (while also setting the marker revision to suppress any delayed events from auto-triggering the job)

          Real Solution: Change core so that SCMDecisionHandler gets consulted after polling has detected changes but before the job gets triggered. Then branch-api would be able to update the Multibranch tracking state and the SCM so that the polling triggered job:

          1. Builds the latest revision
          2. Correctly records that the latest revision has been built in order to ensure that a scan does not retrigger the build

          It should be relatively easy to inject such a hook into https://github.com/jenkinsci/jenkins/blob/d111e2ac1658c8fa5fb768e7d1233613b4b9992d/core/src/main/java/hudson/triggers/SCMTrigger.java#L621-L639 and then have Branch API implement the hook to perform its requirements.

          There is one final gotcha... namely concurrent builds... if there are concurrent builds of the same branch, we would be re-writing the job configuration as each job starts... that may mean that by the time the first build reaches checkout scm it actually is checking out the next build's revision... I wonder if we should actually mandate some more of implementers of the Branch API... (ok I'm rambling now)

          Stephen Connolly added a comment - The way Multibranch works is that every time it scans the repository, for each branch it gets the latest revision. Then it re-configures each branches job to build that revision only and triggers a build if the previous configuration was for a different revision. If you want SCM polling, you should have the scan run at the polling interval as the scan will do one efficient scan for all branches rather than having each branch scan. If you want some branches to not build so often, you should have a rate limit branch property on those branches, etc Now for an organization folder, we should probably provide a mechanism for defining have repository specific properties (analogous to branch specific properties) so that the foo repository can poll every 10 minutes while all other repositories poll daily or something like that. The other thing I want to investigate is polling the events for GitHub, e.g. https://developer.github.com/v3/activity/events/#list-repository-events and https://developer.github.com/v3/activity/events/#list-events-for-an-organization as those should be a much more efficient polling algorithm for the more delta aware modern SCM API event subsystem. Probably this would work as a global setting for the GitHub server where you define the server as either being web-hook push events or being api poll-events (here is the frequency) and then we can have an admin level log of the polling of events which would then get dispatched to the event listeners. (The issue I see with poll-events is that there are a number of events that GitHub says in the documentation are only delivered by web-hook... if one of those is a critical event then we are SooL) OK, so back to this issue. It is necessary for Multibranch to configure each branch job's SCM to be pinned to the revision to build It is confusing to users that clicking build on a branch job will not build the latest code, but rather rebuild the revision The user confusion can probably be cleared up by tweaking the branch project's actions. Hack Solution: If we replace(or rename) the "Build Now" button with a "Rebuild Now" button that would clarify for the user that we will be rebuilding the same revision not building the latest revision. Additionally we could inject a "Build Now" button that uses SCM API to query the head revision of the branch and reconfigure the SCM (while also setting the marker revision to suppress any delayed events from auto-triggering the job) Real Solution: Change core so that SCMDecisionHandler gets consulted after polling has detected changes but before the job gets triggered. Then branch-api would be able to update the Multibranch tracking state and the SCM so that the polling triggered job: 1. Builds the latest revision 2. Correctly records that the latest revision has been built in order to ensure that a scan does not retrigger the build It should be relatively easy to inject such a hook into https://github.com/jenkinsci/jenkins/blob/d111e2ac1658c8fa5fb768e7d1233613b4b9992d/core/src/main/java/hudson/triggers/SCMTrigger.java#L621-L639 and then have Branch API implement the hook to perform its requirements. There is one final gotcha... namely concurrent builds... if there are concurrent builds of the same branch, we would be re-writing the job configuration as each job starts... that may mean that by the time the first build reaches checkout scm it actually is checking out the next build's revision... I wonder if we should actually mandate some more of implementers of the Branch API... (ok I'm rambling now)

          Code changed in jenkins
          User: Robby Pocase
          Path:
          src/main/java/org/jenkinsci/plugins/p4/tagging/TagAction.java
          http://jenkins-ci.org/commit/p4-plugin/c0812f03e85c93d817fac65b21e9df230add5669
          Log:
          Retry getLastAction until build is complete

          A builds change actions will not be available until
          after SCM checkout. In cases of executor starvation
          this can result in infinite queuing. This
          works around JENKINS-40722 to provide concurrent
          builds with P4 polling/Perforce Triggered Builds

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Robby Pocase Path: src/main/java/org/jenkinsci/plugins/p4/tagging/TagAction.java http://jenkins-ci.org/commit/p4-plugin/c0812f03e85c93d817fac65b21e9df230add5669 Log: Retry getLastAction until build is complete A builds change actions will not be available until after SCM checkout. In cases of executor starvation this can result in infinite queuing. This works around JENKINS-40722 to provide concurrent builds with P4 polling/Perforce Triggered Builds

            kohsuke Kohsuke Kawaguchi
            rtyler R. Tyler Croy
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: