Had a long discussion with michaelneale about this. Briefly, if you are not rebuilding when the base branch changes, then Jenkins is not really verifying that the merge product is valid. So you need to pick some tradeoff between fidelity and performance: something in between the true merge build and the head build.

One option would be to merge against the latest base commit which is not newer (according to recorded commit timestamp—not necessarily related to push time) than the PR head commit. This would probably be the closest to the prior behavior of this plugin, which used the poorly documented and unreliable merge ref from GitHub, and would match the behavior of your workaround if I understand it. Another option would be to merge against new base commits, but only at limited intervals, like a day or a week—so for example if you had an old PR sitting around dormant, it would periodically get rechecked against new base versions, but not on every base branch push.

Yes, once explained it makes sense. Technically current PR builders (not this one) do the wrong thing and give a false sense of the state of a validation and merge ready-ness.

As Jesse pointed out - this is a trade of of: How often can you tolerate a broken master: 1) sometimes 3) never.

jglick this seems a subtle point, having behavior like exisitng PR builders would be probably interesting, although all of this is hard to communicate without arm waving, so not sure how to present it as an option (but this is probably not the first time it has come up as a point of confusion).

Thanks for the explanations Jesse & Michael,

While I completely understand the intent, I think we really need to have an option to inhibit the re-trigger yet keep the ability to build merge instead of the PR's HEAD.

In our case:

• We have a team of 12 constantly raising PR's & merging them into the baseline
• There is an average of about 10 PR's which are in an open state
• Each build takes about 25 mins to perform
• A minimum of 10 nodes are required to execute a sub-set of tests in parallel for each build

When we enabled the 'merge with base branch' functionality the following very quickly occurred:

• Jenkins rapidly started queuing jobs up to something like 400
• Our autoscaling group backing the ECS docker nodes for each parallel job went from the usually quiet 3-5 went straight to it's cap of 20
• Each PR ended up having multiple builds executing all on different baselines
• None of the developers were able to determine whether their PR's were valid because they were constantly rebuilding
• Github started rejecting API requests because we were over the rate limit - I imagine because branch indexing was churning over & over for every github hook execution
• With Github check protection turned on it was not possible to merge a PR because the commit status notification was nearly always 'building'
• The team basically chose to ignore the CI system ... if the CI isn't the source of truth then anarchy!

Perhaps if our situation was somewhat different then it'd be perfectly acceptable to let it do rebuilds when the baseline changes, however in our case (& probably many others) as you can see it actually just crushes development team productivity not to mention all of the infrastructure which is attached.

Pretty please, can we not have to make ugly workaround hacks in the pipeline itself because we can't filter the triggering but want the git plugin(s) to do the merge for us?

nullify005 I had a similar situation - many open PRs, and it would exhaust things. In that case I turned of the merge build (as it was worth it in my case) but that isn't true for everyone. I think that is the only option for now - but if you really want the validated merge, then what is needed is a new option: do not rebuild when master changes, which I think jesse mentioned as a possibility

I may not follow correctly, but I believe one more option may be to select the 'unmerged head' for all PRs - origin and fork - and then use the GitHub protected branch feature to force branches (PRs) to be up to date with the base branch before being allowed to merge:

With GitHub Branch Source 2.0.0-beta-1 (available from the experimental update center now or 2.0.0 (available in early January 2017)

When you push a change to the base branch, it should trigger only the base branch. The PR-merge branches will get rebuilt when the indexing kicks in as indexing will detect that the merge commit is now different, but that should only happen once a day/week (depending on how often you configure indexing) so should be much less of an issue

stephenconnolly that actually sounds like a reasonably useful solution to that problem. Build storms are common with that turned on... so mostly I have turned that off as it just wasn't practical when you have a bunch of pull requests in progress...

Stephen Connolly, yes that sounds like a good solution. We are considering changing our PR build support recommendation to be:
If you want Jenkins PR build support, choose the option to build the PR at head (not merge). Any changes to your task branch and PR will result in the PR being built at head. To insure that your merge doesn’t destabilize master, choose the Git “branch protection” option on Master.

Sorry I meant to say that I liked Kurt Madel's solution (branch protection). Stephens solution could still result in build storms.

It's also hard to describe all these options to people - the reality is "do you want lots of builds when you merge to master" - but most people would say "no".

I think, the most of people(include me) want...

Build Trigger:
When PR's head(unmerged) updated.

Build Target(what to checkout):
merged commit with base branch.(build with merged Jenkinsfile)

Is this addressed by the Github Branch Source updates?

rainwaj: no.

mapi you more or less described what I already wrote:

merge against the latest base commit which is not newer […] than the PR head commit

under the assumption that the build starts quickly and the checkout in fact comes before any other base branch commit lands. If the checkout is delayed (for example because of a busy build queue), the behavior I described is preferable to simply merging against the tip of the base branch in that it is more deterministic what the merge commit will be.

By the way it seems that currently while a full reindex of the repository will trigger builds of all merge PR jobs which have a new base branch commit, an event (such as webhook) updating the base branch does not immediately trigger those builds.

Is there a way to rate limit those builds triggered by the full reindexes? We have a large (2 gig) repo with 60-100 PRs open at any one time, and lots of PR churn. When that indexing kicks in, it queues all the PRs for a rebuild, and often things start breaking as the master tries to check out all those PRs at once. The problem is I can't seem to configure the master's checkout behavior at all (referring to the master's flyweight executors checking things out before we run the Jenkinsfile).

hrmpw FYI another use-case ^^^

+1 to spencermalone's comment. I was hoping there might be an additional option in recent plugin updates to only rebuild PRs when folks commit to their PR and not when the base branch is updated. 

I'm a fan of the current behavior, it helps ensure safety on PRs that are left open for weeks. If we're going to stay that route, I just need some way to just say "Don't let the master try to checkout more than x repos at once for the purposes of changelog + Jenkinsfile loading"

Related question: is there a way to differentiate whether a rebuild was triggered by update to base branch or head? I've setup our pipeline job so that I differentiate whether a job is manually or autotriggered by checking currentBuild.rawBuild.getCause(hudson.model.Cause$UserIdCause).properties.shortDescription for "Started by".

It would be super helpful if the "Branch Indexing" message were more granular since I may want to take different action with my build depending on the event that triggered the hook, etc. Thanks

I wanted to add here a solution I found that feels like a nice workaround at least for my needs:

1. In GitHub Org options check to build both merged and unmerged PRs so you have a bunch of PR jobs ending in -merge and -head.
2. In the auto branch triggering field add PR-\d+-head so you auto build all head jobs but not all merge jobs. This will avoid the build storm that causes so many problems.
3. From inside the Jenkinsfile add something like this:

   // have head job trigger merge job...
   if (autoTriggered() && env.BRANCH_NAME.endsWith('head')) {
       def headJobName = "${env.BRANCH_NAME}"
       def mergeJobName = (headJobName.substring(0, headJobName.length() - 4)).concat('merge') // replace head with merge
       build job: mergeJobName, wait: false
       return
   }

    

autoTriggered() is the function I added that returns false if cause description contains "Started by...". All this does is check if it's a head job and if so it triggers the corresponding merge job and exits. 

It feels a little too simple and too good to be true, but at first glance it seems like this gives best of both worlds of auto triggering merge jobs only on changes to head and avoiding the buildStorm. 

With GitHub Branch Source 2.0.0-beta-1 (available from the experimental update center now or 2.0.0 (available in early January 2017)

When you push a change to the base branch, it should trigger only the base branch. The PR-merge branches will get rebuilt when the indexing kicks in as indexing will detect that the merge commit is now different, but that should only happen once a day/week (depending on how often you configure indexing) so should be much less of an issue

Is it just me, or does that change cause a build storm at 9:00 every morning? (Assuming there's been a change in master branch.)

Is there any way of controlling the time of the daily indexing?

I assumed this was based on the last time indexing happened + 24 hours. We normally get the storm in the AM as well, but I ran an index late last night so that I can see if it indexes in the evening today. If not, it's definitely supposed to be "Build if this hasn't built in the past day", so you could arguably setup another job or cron job to force an indexing at whatever time you want. Hopefully it's just doing "last indexing + 24 hours" as the behavior though.

To clarify, when I say indexing, I mean running a "Scan Repository Now" from the repo's folder page.

The Multibranch pipeline's "Suppress automatic SCM triggering" for the `master` branch seems to stop this happening every time `master` changes. It still seems to happen whenever the repository is reindexed, however. Given there doesn't seem to be a current workaround I think we're going to have to go back to the GitHub Pull Request Builder plugin (which has its own issues). Note that the "Suppress automatic SCM triggering" doesn't seem to be available for GitHub Organisation jobs for no apparent reason.

As for our use-case: I'm a Homebrew (macOS package manager) maintainer and our builds are run on limited hardware and take anywhere from 5 minutes to 5 hours to build. Because of how our CI/testing functions and the number of packages we have 99.999% of the time `master` changes have no effect on a given PR (and when they do: we're on the ball enough to notice that). The way things are at the moment with ~70 open PRs our CI infrastructure simply can't come close to keeping up. Every time that all PRs are requeued it would take several days to clear the queue (at which point we've been unable to merge any other PRs for several days). I'm happy to talk more about our use-case if that will help.

My suggestions would be to ensure "Suppress automatic SCM triggering" also affects indexing and is available on GitHub Organisation jobs in short-term and in the medium term provide configuration to disable the master rebuild behave altogether. Opinionated design is great when your users don't have diverse workflows but that doesn't really apply in this case.

Regardless, thanks to everyone working on this for your work on Jenkins.

Opened a PR here: https://github.com/jenkinsci/branch-api-plugin/pull/109/files

So there's a property that is supposed to disable repository scan / branch indexing triggered builds, but it currently also disables all builds caused by events (such as webhooks, etc.) so it's not super useful for much right now. That PR should fix it, so that we can disable org scans triggering builds by going...

properties([overrideIndexTriggers(false)])

in a Jenkinsfile.

In my mind, this would satisfy the ticket, does anyone have other thoughts?

But those that want to disable all options could just use both 

properties([overrideIndexTriggers(false), overrideEventTriggers(false)])

My problem is that currently, those of us that want organization scans to not trigger builds, but still want to run them for the other things they do (such as old branch job cleanup, or finding "lost" PRs) are kind of left without a way to do what we need. This patch lets people choose their default behavior on a per-branch basis.

I agree that the original property seems like a mistake when combined with everything else.

How far out is the interesting head/revision idea? I understand the desire to push forward on a big revision like that, but this has been a pretty big blocker for us for a while. The patch is relatively simple, and it seems like the interesting head/revision stuff is way off on the horizon. Currently this is a key issue that we're feeling every day. 

EDIT: Sorry, after rereading, I see more of the emphasis on the creating extension points. I'll look into that, but this is my first contribution to jenkins plugins, so it'll take me some time to get used to how extensions work.

This is something that should be solved in github-branch-source, not branch-api, though a common extension in scm-api might be useful so the same UI can be reused for BitBucket etc.

I'm also now seeing build storms from this issue.

One can turn off "Periodically if not otherwise run" to prevent the storms, but then you're left with orphaned jobs being left behind when a branch is deleted. Therefore, if one tries to use the branch source plugin, they either have to deal with build storms, or with orphaned jobs.

I don't think I saw this issue back when the core functionality existed in the GitHub Organization Folders plugin. Perhaps this is a regression?

For the record, if you want to avoid triggering rebuilds when just the target branch of the PR has changed then you want https://github.com/jenkinsci/basic-branch-build-strategies-plugin/blob/master/docs/user.adoc#change-requests 

IMHO this problem is closed by the feature "Ignore rebuilding merge branches when only the target branch changed" in the Basic Branch Build Strategy plugin's "Change Requests" strategy

Resolution:

• Install Basic Branch Build Strategies plugin and add the "Change Requests" with "Ignore rebuilding merge branches when only the target branch changed" selected to your build strategies

For safety, you may want to then configure GitHub to refuse to let PRs be merged unless they are up to date with the base branch—IOW to only permit fast-forward merges from the GUI.

I can confirm that setting up the Basic Branch Build Strategies plugin and selecting Regular Branches and Change Requests (with Ignore rebuilding merge branches checked) indeed gives us exactly what we want. Thanks for the tip!

stephenconnolly hey, i've installed the plugins, but i can't find the options on my multiple pipeline job .. Could you help me please ?

aldochristiaan By multiple pipeline job do you mean GitHub Organisation jobs?  Because I agree, the proposed solution here doesn't seem to apply, AFAICT, to the GitHub Source Branch Plugin multibranch-pipeline job.

So those users still need a solution to these build storms that just take down your Jenkins machine every time a base branch (i.e. master) is updated.  I guess we need another ticket.