Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40722

Poll SCM results in a Pipeline run every interval (unnecessarily)

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      My Jenkinsfile includes an SCMTrigger pipelineTrigger for polling SCM for changes:

      properties([
          [$class: 'jenkins.model.BuildDiscarderProperty', strategy: [$class: 'LogRotator', numToKeepStr: '96']],
          pipelineTriggers([[$class:"SCMTrigger", scmpoll_spec:"H/10 * * * *"]]),
      ])
      

      This repository is used via a GitHub Organization Folder on ci.jenkins.io, and seems to believe that changes are being made every ten minutes (per the scmpoll_spec above, despite no such changes being made.

      See screenshots for more context

      This results in an aggressive waste of resources in our Jenkins environment

        Attachments

        1. scmpoll_136.png
          scmpoll_136.png
          43 kB
        2. scmpoll_137.png
          scmpoll_137.png
          43 kB
        3. scmpoll_history.png
          scmpoll_history.png
          26 kB

          Issue Links

            Activity

            Hide
            danielbeck Daniel Beck added a comment -

            Polling intervals shorter than the build duration could the the problem here.

            Show
            danielbeck Daniel Beck added a comment - Polling intervals shorter than the build duration could the the problem here.
            Hide
            stephenconnolly Stephen Connolly added a comment -

            You should not use polling triggers when a job is part of an Organization Folder. The Organization Folder will trigger builds for you as it controls the SCM revision that is being checked out when you run a build (in other words, even when the polling triggers a build, that build will be for the last revision detected by the MultiBranch project not the revision that polling has detected)

            Show
            stephenconnolly Stephen Connolly added a comment - You should not use polling triggers when a job is part of an Organization Folder. The Organization Folder will trigger builds for you as it controls the SCM revision that is being checked out when you run a build (in other words, even when the polling triggers a build, that build will be for the last revision detected by the MultiBranch project not the revision that polling has detected)
            Hide
            danielbeck Daniel Beck added a comment -

            Besides making no sense, is this documented anywhere?

            Show
            danielbeck Daniel Beck added a comment - Besides making no sense, is this documented anywhere?
            Hide
            rtyler R. Tyler Croy added a comment - - edited

            Stephen Connolly, I generally use organization folders as a "multibranch on steroids", in that I don't want to set up Pipeline projects in Jenkins for each repository and the organization folder let's me automate that aspect of it nicely.

            If that's not something Patrick Wolf wants to encourage, while I disagree with it, I think this behavior should be disabled, warned, or errored upon. IMHO it's a poor user experience to have org folders do something unexpected without ever setting that expectation that the behavior is "wrong".

            Show
            rtyler R. Tyler Croy added a comment - - edited Stephen Connolly , I generally use organization folders as a "multibranch on steroids", in that I don't want to set up Pipeline projects in Jenkins for each repository and the organization folder let's me automate that aspect of it nicely. If that's not something Patrick Wolf wants to encourage, while I disagree with it, I think this behavior should be disabled, warned, or errored upon. IMHO it's a poor user experience to have org folders do something unexpected without ever setting that expectation that the behavior is "wrong".
            Hide
            hrmpw Patrick Wolf added a comment -

            I don't understand Stephen Connolly's point here.

            If I'm not using a webhook to trigger builds then polling is probably the next best way to get updates. That has always been the case in Jenkins.

            My understanding is that regardless of a webhook or polling trigger the multibranch job will by default ("checkout scm") checkout the head of that branch and run the job based on the latest and greatest in the repo. Are you saying that isn't true, Stephen?

            Show
            hrmpw Patrick Wolf added a comment - I don't understand Stephen Connolly 's point here. If I'm not using a webhook to trigger builds then polling is probably the next best way to get updates. That has always been the case in Jenkins. My understanding is that regardless of a webhook or polling trigger the multibranch job will by default ("checkout scm") checkout the head of that branch and run the job based on the latest and greatest in the repo. Are you saying that isn't true, Stephen?
            Hide
            stephenconnolly Stephen Connolly added a comment -

            The way Multibranch works is that every time it scans the repository, for each branch it gets the latest revision.

            Then it re-configures each branches job to build that revision only and triggers a build if the previous configuration was for a different revision.

            If you want SCM polling, you should have the scan run at the polling interval as the scan will do one efficient scan for all branches rather than having each branch scan.

            If you want some branches to not build so often, you should have a rate limit branch property on those branches, etc

            Now for an organization folder, we should probably provide a mechanism for defining have repository specific properties (analogous to branch specific properties) so that the foo repository can poll every 10 minutes while all other repositories poll daily or something like that.

            The other thing I want to investigate is polling the events for GitHub, e.g. https://developer.github.com/v3/activity/events/#list-repository-events and https://developer.github.com/v3/activity/events/#list-events-for-an-organization as those should be a much more efficient polling algorithm for the more delta aware modern SCM API event subsystem. Probably this would work as a global setting for the GitHub server where you define the server as either being web-hook push events or being api poll-events (here is the frequency) and then we can have an admin level log of the polling of events which would then get dispatched to the event listeners. (The issue I see with poll-events is that there are a number of events that GitHub says in the documentation are only delivered by web-hook... if one of those is a critical event then we are SooL)

            OK, so back to this issue.

            • It is necessary for Multibranch to configure each branch job's SCM to be pinned to the revision to build
            • It is confusing to users that clicking build on a branch job will not build the latest code, but rather rebuild the revision

            The user confusion can probably be cleared up by tweaking the branch project's actions.

            Hack Solution: If we replace(or rename) the "Build Now" button with a "Rebuild Now" button that would clarify for the user that we will be rebuilding the same revision not building the latest revision. Additionally we could inject a "Build Now" button that uses SCM API to query the head revision of the branch and reconfigure the SCM (while also setting the marker revision to suppress any delayed events from auto-triggering the job)

            Real Solution: Change core so that SCMDecisionHandler gets consulted after polling has detected changes but before the job gets triggered. Then branch-api would be able to update the Multibranch tracking state and the SCM so that the polling triggered job:

            1. Builds the latest revision
            2. Correctly records that the latest revision has been built in order to ensure that a scan does not retrigger the build

            It should be relatively easy to inject such a hook into https://github.com/jenkinsci/jenkins/blob/d111e2ac1658c8fa5fb768e7d1233613b4b9992d/core/src/main/java/hudson/triggers/SCMTrigger.java#L621-L639 and then have Branch API implement the hook to perform its requirements.

            There is one final gotcha... namely concurrent builds... if there are concurrent builds of the same branch, we would be re-writing the job configuration as each job starts... that may mean that by the time the first build reaches checkout scm it actually is checking out the next build's revision... I wonder if we should actually mandate some more of implementers of the Branch API... (ok I'm rambling now)

            Show
            stephenconnolly Stephen Connolly added a comment - The way Multibranch works is that every time it scans the repository, for each branch it gets the latest revision. Then it re-configures each branches job to build that revision only and triggers a build if the previous configuration was for a different revision. If you want SCM polling, you should have the scan run at the polling interval as the scan will do one efficient scan for all branches rather than having each branch scan. If you want some branches to not build so often, you should have a rate limit branch property on those branches, etc Now for an organization folder, we should probably provide a mechanism for defining have repository specific properties (analogous to branch specific properties) so that the foo repository can poll every 10 minutes while all other repositories poll daily or something like that. The other thing I want to investigate is polling the events for GitHub, e.g. https://developer.github.com/v3/activity/events/#list-repository-events and https://developer.github.com/v3/activity/events/#list-events-for-an-organization as those should be a much more efficient polling algorithm for the more delta aware modern SCM API event subsystem. Probably this would work as a global setting for the GitHub server where you define the server as either being web-hook push events or being api poll-events (here is the frequency) and then we can have an admin level log of the polling of events which would then get dispatched to the event listeners. (The issue I see with poll-events is that there are a number of events that GitHub says in the documentation are only delivered by web-hook... if one of those is a critical event then we are SooL) OK, so back to this issue. It is necessary for Multibranch to configure each branch job's SCM to be pinned to the revision to build It is confusing to users that clicking build on a branch job will not build the latest code, but rather rebuild the revision The user confusion can probably be cleared up by tweaking the branch project's actions. Hack Solution: If we replace(or rename) the "Build Now" button with a "Rebuild Now" button that would clarify for the user that we will be rebuilding the same revision not building the latest revision. Additionally we could inject a "Build Now" button that uses SCM API to query the head revision of the branch and reconfigure the SCM (while also setting the marker revision to suppress any delayed events from auto-triggering the job) Real Solution: Change core so that SCMDecisionHandler gets consulted after polling has detected changes but before the job gets triggered. Then branch-api would be able to update the Multibranch tracking state and the SCM so that the polling triggered job: 1. Builds the latest revision 2. Correctly records that the latest revision has been built in order to ensure that a scan does not retrigger the build It should be relatively easy to inject such a hook into https://github.com/jenkinsci/jenkins/blob/d111e2ac1658c8fa5fb768e7d1233613b4b9992d/core/src/main/java/hudson/triggers/SCMTrigger.java#L621-L639 and then have Branch API implement the hook to perform its requirements. There is one final gotcha... namely concurrent builds... if there are concurrent builds of the same branch, we would be re-writing the job configuration as each job starts... that may mean that by the time the first build reaches checkout scm it actually is checking out the next build's revision... I wonder if we should actually mandate some more of implementers of the Branch API... (ok I'm rambling now)
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Robby Pocase
            Path:
            src/main/java/org/jenkinsci/plugins/p4/tagging/TagAction.java
            http://jenkins-ci.org/commit/p4-plugin/c0812f03e85c93d817fac65b21e9df230add5669
            Log:
            Retry getLastAction until build is complete

            A builds change actions will not be available until
            after SCM checkout. In cases of executor starvation
            this can result in infinite queuing. This
            works around JENKINS-40722 to provide concurrent
            builds with P4 polling/Perforce Triggered Builds

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Robby Pocase Path: src/main/java/org/jenkinsci/plugins/p4/tagging/TagAction.java http://jenkins-ci.org/commit/p4-plugin/c0812f03e85c93d817fac65b21e9df230add5669 Log: Retry getLastAction until build is complete A builds change actions will not be available until after SCM checkout. In cases of executor starvation this can result in infinite queuing. This works around JENKINS-40722 to provide concurrent builds with P4 polling/Perforce Triggered Builds

              People

              Assignee:
              kohsuke Kohsuke Kawaguchi
              Reporter:
              rtyler R. Tyler Croy
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: