Status: Resolved (View Workflow)
Resolution: Won't Do
My Jenkinsfile includes an SCMTrigger pipelineTrigger for polling SCM for changes:
properties([ [$class: 'jenkins.model.BuildDiscarderProperty', strategy: [$class: 'LogRotator', numToKeepStr: '96']], pipelineTriggers([[$class:"SCMTrigger", scmpoll_spec:"H/10 * * * *"]]), ])
This repository is used via a GitHub Organization Folder on ci.jenkins.io, and seems to believe that changes are being made every ten minutes (per the scmpoll_spec above, despite no such changes being made.
See screenshots for more context
This results in an aggressive waste of resources in our Jenkins environment
- is related to
JENKINS-41074 UX Issue with Polling in Multibranch Pipeline
- relates to
JENKINS-41072 Poll the GitHub Events API as an alternative to webhook
JENKINS-41073 MultiBranch projects should veto SCM Polling
You should not use polling triggers when a job is part of an Organization Folder. The Organization Folder will trigger builds for you as it controls the SCM revision that is being checked out when you run a build (in other words, even when the polling triggers a build, that build will be for the last revision detected by the MultiBranch project not the revision that polling has detected)
stephenconnolly, I generally use organization folders as a "multibranch on steroids", in that I don't want to set up Pipeline projects in Jenkins for each repository and the organization folder let's me automate that aspect of it nicely.
If that's not something hrmpw wants to encourage, while I disagree with it, I think this behavior should be disabled, warned, or errored upon. IMHO it's a poor user experience to have org folders do something unexpected without ever setting that expectation that the behavior is "wrong".
I don't understand stephenconnolly's point here.
If I'm not using a webhook to trigger builds then polling is probably the next best way to get updates. That has always been the case in Jenkins.
My understanding is that regardless of a webhook or polling trigger the multibranch job will by default ("checkout scm") checkout the head of that branch and run the job based on the latest and greatest in the repo. Are you saying that isn't true, Stephen?
The way Multibranch works is that every time it scans the repository, for each branch it gets the latest revision.
Then it re-configures each branches job to build that revision only and triggers a build if the previous configuration was for a different revision.
If you want SCM polling, you should have the scan run at the polling interval as the scan will do one efficient scan for all branches rather than having each branch scan.
If you want some branches to not build so often, you should have a rate limit branch property on those branches, etc
Now for an organization folder, we should probably provide a mechanism for defining have repository specific properties (analogous to branch specific properties) so that the foo repository can poll every 10 minutes while all other repositories poll daily or something like that.
The other thing I want to investigate is polling the events for GitHub, e.g. https://developer.github.com/v3/activity/events/#list-repository-events and https://developer.github.com/v3/activity/events/#list-events-for-an-organization as those should be a much more efficient polling algorithm for the more delta aware modern SCM API event subsystem. Probably this would work as a global setting for the GitHub server where you define the server as either being web-hook push events or being api poll-events (here is the frequency) and then we can have an admin level log of the polling of events which would then get dispatched to the event listeners. (The issue I see with poll-events is that there are a number of events that GitHub says in the documentation are only delivered by web-hook... if one of those is a critical event then we are SooL)
OK, so back to this issue.
- It is necessary for Multibranch to configure each branch job's SCM to be pinned to the revision to build
- It is confusing to users that clicking build on a branch job will not build the latest code, but rather rebuild the revision
The user confusion can probably be cleared up by tweaking the branch project's actions.
Hack Solution: If we replace(or rename) the "Build Now" button with a "Rebuild Now" button that would clarify for the user that we will be rebuilding the same revision not building the latest revision. Additionally we could inject a "Build Now" button that uses SCM API to query the head revision of the branch and reconfigure the SCM (while also setting the marker revision to suppress any delayed events from auto-triggering the job)
Real Solution: Change core so that SCMDecisionHandler gets consulted after polling has detected changes but before the job gets triggered. Then branch-api would be able to update the Multibranch tracking state and the SCM so that the polling triggered job:
1. Builds the latest revision
2. Correctly records that the latest revision has been built in order to ensure that a scan does not retrigger the build
It should be relatively easy to inject such a hook into https://github.com/jenkinsci/jenkins/blob/d111e2ac1658c8fa5fb768e7d1233613b4b9992d/core/src/main/java/hudson/triggers/SCMTrigger.java#L621-L639 and then have Branch API implement the hook to perform its requirements.
There is one final gotcha... namely concurrent builds... if there are concurrent builds of the same branch, we would be re-writing the job configuration as each job starts... that may mean that by the time the first build reaches checkout scm it actually is checking out the next build's revision... I wonder if we should actually mandate some more of implementers of the Branch API... (ok I'm rambling now)
Code changed in jenkins
User: Robby Pocase
Retry getLastAction until build is complete
A builds change actions will not be available until
after SCM checkout. In cases of executor starvation
this can result in infinite queuing. This
JENKINS-40722 to provide concurrent
builds with P4 polling/Perforce Triggered Builds
Polling intervals shorter than the build duration could the the problem here.