Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37491

Build Origin PRs (merge with base branch) conducts rebuilds when baseline changes

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • None
    • github-branch-source 1.8.1
      jenkins 2.18

      When 'Build Origin PRs (merge with base branch)' is selected any PR's which are filed against a base branch are re-triggered when the base branch changes.

      While this is all fine if you have a small # of PR's & the builds are quick, if you have a large team & backlog of PR's then any merge to the base branch will automatically trigger every PR which is using the same baseline.

      When you then end up with is a Jenkins that has a massive queue to process & github often complaining that you've exceeded the API rate limit.

      The concept of building the merge branch is great, however at the moment I really can't use it because of the previously stated behaviour & causes massive issues for our builds.

      Previous versions of the plugin didn't exhibit this behaviour at all, nor do other systems like travis which build PR's using the merge branch, nor did the github pull request builder plugin.

      As a workaround I'm forced to use the 'Build Origin PRs (unmerged HEAD)' and then conduct the merge or checkout of the GH merge branch itself within the pipeline which is less than ideal.

      Can we have an option to inhibit a re-trigger when the baseline changes for PRs with merge?

      btw I like where this plugin is going

          [JENKINS-37491] Build Origin PRs (merge with base branch) conducts rebuilds when baseline changes

          Lee Webb created issue -

          Jesse Glick added a comment -

          Had a long discussion with michaelneale about this. Briefly, if you are not rebuilding when the base branch changes, then Jenkins is not really verifying that the merge product is valid. So you need to pick some tradeoff between fidelity and performance: something in between the true merge build and the head build.

          One option would be to merge against the latest base commit which is not newer (according to recorded commit timestamp—not necessarily related to push time) than the PR head commit. This would probably be the closest to the prior behavior of this plugin, which used the poorly documented and unreliable merge ref from GitHub, and would match the behavior of your workaround if I understand it. Another option would be to merge against new base commits, but only at limited intervals, like a day or a week—so for example if you had an old PR sitting around dormant, it would periodically get rechecked against new base versions, but not on every base branch push.

          Jesse Glick added a comment - Had a long discussion with michaelneale about this. Briefly, if you are not rebuilding when the base branch changes, then Jenkins is not really verifying that the merge product is valid. So you need to pick some tradeoff between fidelity and performance: something in between the true merge build and the head build. One option would be to merge against the latest base commit which is not newer (according to recorded commit timestamp—not necessarily related to push time) than the PR head commit. This would probably be the closest to the prior behavior of this plugin, which used the poorly documented and unreliable merge ref from GitHub, and would match the behavior of your workaround if I understand it. Another option would be to merge against new base commits, but only at limited intervals, like a day or a week—so for example if you had an old PR sitting around dormant, it would periodically get rechecked against new base versions, but not on every base branch push.

          Michael Neale added a comment -

          Yes, once explained it makes sense. Technically current PR builders (not this one) do the wrong thing and give a false sense of the state of a validation and merge ready-ness.

          As Jesse pointed out - this is a trade of of: How often can you tolerate a broken master: 1) sometimes 3) never.

          jglick this seems a subtle point, having behavior like exisitng PR builders would be probably interesting, although all of this is hard to communicate without arm waving, so not sure how to present it as an option (but this is probably not the first time it has come up as a point of confusion).

          Michael Neale added a comment - Yes, once explained it makes sense. Technically current PR builders (not this one) do the wrong thing and give a false sense of the state of a validation and merge ready-ness. As Jesse pointed out - this is a trade of of: How often can you tolerate a broken master: 1) sometimes 3) never. jglick this seems a subtle point, having behavior like exisitng PR builders would be probably interesting, although all of this is hard to communicate without arm waving, so not sure how to present it as an option (but this is probably not the first time it has come up as a point of confusion).

          Lee Webb added a comment -

          Thanks for the explanations Jesse & Michael,

          While I completely understand the intent, I think we really need to have an option to inhibit the re-trigger yet keep the ability to build merge instead of the PR's HEAD.

          In our case:

          • We have a team of 12 constantly raising PR's & merging them into the baseline
          • There is an average of about 10 PR's which are in an open state
          • Each build takes about 25 mins to perform
          • A minimum of 10 nodes are required to execute a sub-set of tests in parallel for each build

          When we enabled the 'merge with base branch' functionality the following very quickly occurred:

          • Jenkins rapidly started queuing jobs up to something like 400
          • Our autoscaling group backing the ECS docker nodes for each parallel job went from the usually quiet 3-5 went straight to it's cap of 20
          • Each PR ended up having multiple builds executing all on different baselines
          • None of the developers were able to determine whether their PR's were valid because they were constantly rebuilding
          • Github started rejecting API requests because we were over the rate limit - I imagine because branch indexing was churning over & over for every github hook execution
          • With Github check protection turned on it was not possible to merge a PR because the commit status notification was nearly always 'building'
          • The team basically chose to ignore the CI system ... if the CI isn't the source of truth then anarchy!

          Perhaps if our situation was somewhat different then it'd be perfectly acceptable to let it do rebuilds when the baseline changes, however in our case (& probably many others) as you can see it actually just crushes development team productivity not to mention all of the infrastructure which is attached.

          Pretty please, can we not have to make ugly workaround hacks in the pipeline itself because we can't filter the triggering but want the git plugin(s) to do the merge for us?

          Lee Webb added a comment - Thanks for the explanations Jesse & Michael, While I completely understand the intent, I think we really need to have an option to inhibit the re-trigger yet keep the ability to build merge instead of the PR's HEAD. In our case: We have a team of 12 constantly raising PR's & merging them into the baseline There is an average of about 10 PR's which are in an open state Each build takes about 25 mins to perform A minimum of 10 nodes are required to execute a sub-set of tests in parallel for each build When we enabled the 'merge with base branch' functionality the following very quickly occurred: Jenkins rapidly started queuing jobs up to something like 400 Our autoscaling group backing the ECS docker nodes for each parallel job went from the usually quiet 3-5 went straight to it's cap of 20 Each PR ended up having multiple builds executing all on different baselines None of the developers were able to determine whether their PR's were valid because they were constantly rebuilding Github started rejecting API requests because we were over the rate limit - I imagine because branch indexing was churning over & over for every github hook execution With Github check protection turned on it was not possible to merge a PR because the commit status notification was nearly always 'building' The team basically chose to ignore the CI system ... if the CI isn't the source of truth then anarchy! Perhaps if our situation was somewhat different then it'd be perfectly acceptable to let it do rebuilds when the baseline changes, however in our case (& probably many others) as you can see it actually just crushes development team productivity not to mention all of the infrastructure which is attached. Pretty please, can we not have to make ugly workaround hacks in the pipeline itself because we can't filter the triggering but want the git plugin(s) to do the merge for us?

          Michael Neale added a comment -

          nullify005 I had a similar situation - many open PRs, and it would exhaust things. In that case I turned of the merge build (as it was worth it in my case) but that isn't true for everyone. I think that is the only option for now - but if you really want the validated merge, then what is needed is a new option: do not rebuild when master changes, which I think jesse mentioned as a possibility

          Michael Neale added a comment - nullify005 I had a similar situation - many open PRs, and it would exhaust things. In that case I turned of the merge build (as it was worth it in my case) but that isn't true for everyone. I think that is the only option for now - but if you really want the validated merge, then what is needed is a new option: do not rebuild when master changes, which I think jesse mentioned as a possibility
          Kurt Madel made changes -
          Attachment New: screenshot-1.png [ 33788 ]

          Kurt Madel added a comment -

          I may not follow correctly, but I believe one more option may be to select the 'unmerged head' for all PRs - origin and fork - and then use the GitHub protected branch feature to force branches (PRs) to be up to date with the base branch before being allowed to merge:

          Kurt Madel added a comment - I may not follow correctly, but I believe one more option may be to select the 'unmerged head' for all PRs - origin and fork - and then use the GitHub protected branch feature to force branches (PRs) to be up to date with the base branch before being allowed to merge:

          With GitHub Branch Source 2.0.0-beta-1 (available from the experimental update center now or 2.0.0 (available in early January 2017)

          When you push a change to the base branch, it should trigger only the base branch. The PR-merge branches will get rebuilt when the indexing kicks in as indexing will detect that the merge commit is now different, but that should only happen once a day/week (depending on how often you configure indexing) so should be much less of an issue

          Stephen Connolly added a comment - With GitHub Branch Source 2.0.0-beta-1 (available from the experimental update center now or 2.0.0 (available in early January 2017) When you push a change to the base branch, it should trigger only the base branch. The PR-merge branches will get rebuilt when the indexing kicks in as indexing will detect that the merge commit is now different, but that should only happen once a day/week (depending on how often you configure indexing) so should be much less of an issue

          Michael Neale added a comment -

          stephenconnolly that actually sounds like a reasonably useful solution to that problem. Build storms are common with that turned on... so mostly I have turned that off as it just wasn't practical when you have a bunch of pull requests in progress...

          Michael Neale added a comment - stephenconnolly that actually sounds like a reasonably useful solution to that problem. Build storms are common with that turned on... so mostly I have turned that off as it just wasn't practical when you have a bunch of pull requests in progress...

          Bill Hamilton added a comment -

          Stephen Connolly, yes that sounds like a good solution. We are considering changing our PR build support recommendation to be:
          If you want Jenkins PR build support, choose the option to build the PR at head (not merge). Any changes to your task branch and PR will result in the PR being built at head. To insure that your merge doesn’t destabilize master, choose the Git “branch protection” option on Master.

          Bill Hamilton added a comment - Stephen Connolly, yes that sounds like a good solution. We are considering changing our PR build support recommendation to be: If you want Jenkins PR build support, choose the option to build the PR at head (not merge). Any changes to your task branch and PR will result in the PR being built at head. To insure that your merge doesn’t destabilize master, choose the Git “branch protection” option on Master.

            Unassigned Unassigned
            nullify005 Lee Webb
            Votes:
            6 Vote for this issue
            Watchers:
            20 Start watching this issue

              Created:
              Updated:
              Resolved: