Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54864

github multibranch builds fail to build with latest branch api update

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • branch-api-plugin
    • None
    • Jenkins 2.138.3 on Ubuntu
      git plugin 3.9.1 and branch api plugin 2.1.1 is broken.
      git plugin 3.9.0 and branch api plugin 2.0.21 is ok.
    • branch-api 2.5.4

      The latest updates broke github multibranch builds.

      The branches are being discovered, but all automaticdiscovery of all new changes on all branches results in the following typical log entries:

      Checking pull-requests...
       
      Checking pull request #3
      ‘Jenkinsfile’ found
      Met criteria
      Changes detected: PR-3 (null → 24d2f4cb5debd9b3f6f8c86383eb077be0dee0c4+517f061b1a7db844659ca98d1b61a7dcd0b6fb82)
      No automatic builds for PR-3

      These builds should have triggered.  Backing out to the last stable set of plugins fixes the problem.

          [JENKINS-54864] github multibranch builds fail to build with latest branch api update

          Jason Crow added a comment - - edited

          Having the same problem PR's are detected but never built.  I've reverted and PR's are building again. However there are error messages showing:

           
          Dependency errors:

          Some plugins could not be loaded due to unsatisfied dependencies. Fix these issues and restart Jenkins to restore the functionality provided by these plugins.
          Basic Branch Build Strategies Plugin version 1.1.1Branch API Plugin v2.0.21 is older than required. To fix, install v2.1.0 or later.
          Go to plugin managerConfigure which of these warnings are shown
           
           Warnings have been published for the following currently installed components.Git plugin 3.9.0 Server-side request forgery

          Jason Crow added a comment - - edited Having the same problem PR's are detected but never built.  I've reverted and PR's are building again. However there are error messages showing:   Dependency errors: Some plugins could not be loaded due to unsatisfied dependencies. Fix these issues and restart Jenkins to restore the functionality provided by these plugins. Basic Branch Build Strategies Plugin version 1.1.1Branch API Plugin v2.0.21 is older than required. To fix, install v2.1.0 or later. Go to plugin managerConfigure which of these warnings are shown    Warnings have been published for the following currently installed components. Git plugin 3.9.0 Server-side request forgery

          Mark Waite added a comment -

          It should be enough to revert branch API plugin 2.1.1.

          The changes between git plugin 3.9.0 and 3.9.1 added a requirement that POST must be used when checking the repository browser (SECURITY-810) and some minor automated testing changes.

          Mark Waite added a comment - It should be enough to revert branch API plugin 2.1.1. The changes between git plugin 3.9.0 and 3.9.1 added a requirement that POST must be used when checking the repository browser (SECURITY-810) and some minor automated testing changes.

          Jason Crow added a comment -

          Cool, works with just the API plugin reverted.

          Jason Crow added a comment - Cool, works with just the API plugin reverted.

          Jason Crow added a comment -

          Does anyone know what's going on with branch api plugin 2.1.1? We have continuous deployment pipeline that's dependent on PR builds to pass github checks and the PR's never build with this plugin version.

          Jason Crow added a comment - Does anyone know what's going on with branch api plugin 2.1.1? We have continuous deployment pipeline that's dependent on PR builds to pass github checks and the PR's never build with this plugin version.

          Mark Waite added a comment -

          rsandell and jglick were involved in the most recent release of the branch API plugin, though the changes from branch api plugin 2.1.0 to 2.1.1 seem unlikely to have caused this change of behavior.

          Mark Waite added a comment - rsandell and jglick were involved in the most recent release of the branch API plugin, though the changes from branch api plugin 2.1.0 to 2.1.1 seem unlikely to have caused this change of behavior.

          Jesse Glick added a comment -

          Perhaps caused by JENKINS-47859 by stephenconnolly. Just guessing from the plugin changelog.

          Jesse Glick added a comment - Perhaps caused by JENKINS-47859 by stephenconnolly . Just guessing from the plugin changelog.

          Jason Crow added a comment -

          markewaite I don't think it was between 2.1.0 and 2.1.1, I had to roll back to 2.0.21 to return to a "semi-functioning" pipeline - still seeing unexplained issues with PR builds with discovery and status reporting back to github.

          Jason Crow added a comment - markewaite I don't think it was between 2.1.0 and 2.1.1, I had to roll back to 2.0.21 to return to a "semi-functioning" pipeline - still seeing unexplained issues with PR builds with discovery and status reporting back to github.

          I suspect you had "Automatic branch project triggering" configured to something like master, PR-*

          This gets mapped to matching branches that have the set of names as automatic build, but doesn't include Pull requests in automatic builds. As a workaround, just add Pull Requests to the build strategies and pull requests should start building.

          If somebody is interested in fixing this you would need to:

          • Establish that all SCMSource implementations only use PR- as the prefix for pull requests... if there is an SCMSource implementation that uses something like CR- then we cannot fix this issue as we would not know if PR-* is intended to target branches that start with PR- or pull requests.
          • If we know that PR- is guaranteed to only apply to pull requests, then modify the migration so that it parses the "Automatic branch project triggering" value, pulls out the PR- ones and adds named branch build strategies for all others. If there is any PR- entries then add the change request build strategy

          Stephen Connolly added a comment - I suspect you had "Automatic branch project triggering" configured to something like master, PR-* This gets mapped to matching branches that have the set of names as automatic build, but doesn't include Pull requests in automatic builds. As a workaround, just add Pull Requests to the build strategies and pull requests should start building. If somebody is interested in fixing this you would need to: Establish that all SCMSource implementations only use PR- as the prefix for pull requests... if there is an SCMSource implementation that uses something like CR- then we cannot fix this issue as we would not know if PR-* is intended to target branches that start with PR- or pull requests. If we know that PR- is guaranteed to only apply to pull requests, then modify the migration so that it parses the "Automatic branch project triggering" value, pulls out the PR- ones and adds named branch build strategies for all others. If there is any PR- entries then add the change request build strategy

          Jesse Glick added a comment - - edited

          AFAICT you need to install a new Basic Branch Build Strategies plugin to recover that functionality, and there should have been an administrative monitor displayed to alert you to this fact. (pay attention to what stephenconnolly wrote instead)

          Jesse Glick added a comment - - edited AFAICT you need to install a new Basic Branch Build Strategies plugin to recover that functionality, and there should have been an administrative monitor displayed to alert you to this fact. (pay attention to what stephenconnolly wrote instead)

          Jason Crow added a comment - - edited

          stephenconnolly My Automatic branch project triggering is set to .*

          The Basic Branch Build Strategies plugin is installed, when I went through this upgrade the only way I got PR's to build automatically again was to rollback the branch-api-plugin to 2.0.21.  Hence the following message:

          Dependency errors:

          Some plugins could not be loaded due to unsatisfied dependencies. Fix these issues and restart Jenkins to restore the functionality provided by these plugins.

          Basic Branch Build Strategies Plugin version 1.1.1

          Branch API Plugin v2.0.21 is older than required. To fix, install v2.1.0 or later.

          Jason Crow added a comment - - edited stephenconnolly My Automatic branch project triggering is set to .* The Basic Branch Build Strategies plugin is installed, when I went through this upgrade the only way I got PR's to build automatically again was to rollback the branch-api-plugin to 2.0.21.  Hence the following message: Dependency errors: Some plugins could not be loaded due to unsatisfied dependencies. Fix these issues and restart Jenkins to restore the functionality provided by these plugins. Basic Branch Build Strategies Plugin version 1.1.1 Branch API Plugin v2.0.21 is older than required. To fix, install v2.1.0 or later.

          Steve Mason added a comment -

          Just to add - we had this problem and adding the "Change requests" build strategy seems to have made our PRs build automatically again on the latest version of the plugin

          Steve Mason added a comment - Just to add - we had this problem and adding the "Change requests" build strategy seems to have made our PRs build automatically again on the latest version of the plugin

          Jason Crow added a comment -

          Thanks spmason this fixes the issue. I think this was more a difference in plugin functionality than anything else. This issue could probably be closed.

          Jason Crow added a comment - Thanks spmason this fixes the issue. I think this was more a difference in plugin functionality than anything else. This issue could probably be closed.

          Steve Mason added a comment -

          The only bug I can see is that this config should have been copied over from the previous version correctly

          Steve Mason added a comment - The only bug I can see is that this config should have been copied over from the previous version correctly

          Stephen Connolly added a comment - - edited

          So if somebody can come up with an expression parser for the old configuration string that can reliably distinguish between:

          • PR-* means build all branches whos names begin with PR- such as PR-foobar and do not build pull requests because our SCMSource doesn't even have the concept.
          • PR-* means build all pull-requests but we don't want to accidentally build branches with names that begin with PR- because we just want to build the master branch.
          • CR-* means build all pull-requests because our SCMSource calls them Change Requests and uses CR- as the prefix

          I would be happy to merge.

          Until then, the least risk (given that building a branch can result in deploying things into production or expose secrets to a build that isn't trusted) is to err on the side of "if we don't know, do nothing".

          In any case, we could argue that the old `Automatic branch project triggering` has a bug of requiring you to know to add PR-* in order to build pull requests... just that people were relying on the bug!

          Stephen Connolly added a comment - - edited So if somebody can come up with an expression parser for the old configuration string that can reliably distinguish between: PR-* means build all branches whos names begin with PR- such as PR-foobar and do not build pull requests because our SCMSource doesn't even have the concept. PR-* means build all pull-requests but we don't want to accidentally build branches with names that begin with PR- because we just want to build the master branch. CR-* means build all pull-requests because our SCMSource calls them Change Requests and uses CR- as the prefix I would be happy to merge. Until then, the least risk (given that building a branch can result in deploying things into production or expose secrets to a build that isn't trusted) is to err on the side of "if we don't know, do nothing". In any case, we could argue that the old `Automatic branch project triggering` has a bug of requiring you to know to add PR-* in order to build pull requests... just that people were relying on the bug!

          And just in case somebody says "who'd call your branches PR-*?" suppose you are using JIRA and your JIRA project ID is PR... then it might make extreme sense to name all your branches by the JIRA ID that they are developing... so that JIRA ID PR-5612 is in the branch PR-5612... if you are using an SCMSource that doesn't have Change Requests... perhaps you rolled a custom one similar to https://github.com/stephenc/asf-gitpubsub-jenkins-plugin for your own in-house GIT hosting...

          You might not want to build any of the OPS ticket branches on your Jenkins, so you just set the automatic building to master,PR-*

          So if we are going to embed the logic to figure these things out into code... we should handle the above!

          Stephen Connolly added a comment - And just in case somebody says "who'd call your branches PR-* ?" suppose you are using JIRA and your JIRA project ID is PR ... then it might make extreme sense to name all your branches by the JIRA ID that they are developing... so that JIRA ID PR-5612 is in the branch PR-5612 ... if you are using an SCMSource that doesn't have Change Requests... perhaps you rolled a custom one similar to https://github.com/stephenc/asf-gitpubsub-jenkins-plugin for your own in-house GIT hosting... You might not want to build any of the OPS ticket branches on your Jenkins, so you just set the automatic building to master,PR-* So if we are going to embed the logic to figure these things out into code... we should handle the above!

          Jon Tancer added a comment -

          I can confirm spmason's solution fixed our issue.

          For posterity:

          If you use GitHub Organizations with Jenkins and Jenkins is no longer building commits from PR branches, perform the following steps to resolve the issue

          1. Open the GitHub Organization folder
          2. On the left hand side, select Configure
          3. Scroll down to Build Strategies which is nested underneath the Projects section
          4. Press Add
          5. Select Change requests
          6. Scroll down to the bottom of the page and press Save

          {{}}{{}}You will now see the GitHub Organization being re-scanned and all your missing PR builds will begin to build.

           

          Jon Tancer added a comment - I can confirm spmason 's solution fixed our issue. For posterity: If you use GitHub Organizations with Jenkins and Jenkins is no longer building commits from PR branches, perform the following steps to resolve the issue Open the GitHub Organization folder On the left hand side, select Configure Scroll down to Build Strategies  which is nested underneath the Projects section Press Add Select Change requests Scroll down to the bottom of the page and press  Save {{}}{{}}You will now see the GitHub Organization being re-scanned and all your missing PR builds will begin to build.  

          John Mellor added a comment -

          Can someone please detail the alleged workaround details?  I do not see the config sections noted by Steve Mason and Jon Tancer in either of my sites.  I see the note above explaining what I should alter in the github section config when I use the deprecated and replaced organization plugin, but not how to do this in an up-to-date system.  Is it possible to perform a workaround without using the do-not-use plugins?

           

          John Mellor added a comment - Can someone please detail the alleged workaround details?  I do not see the config sections noted by Steve Mason and Jon Tancer in either of my sites.  I see the note above explaining what I should alter in the github section config when I use the deprecated and replaced organization plugin, but not how to do this in an up-to-date system.  Is it possible to perform a workaround without using the do-not-use plugins?  

          Mark Waite added a comment -

          alt_jmellor did you review the comment from stephenconnolly? He noted:

          Add Pull Requests to the build strategies and pull requests should start building.

          Mark Waite added a comment - alt_jmellor did you review the comment from stephenconnolly ? He noted: Add Pull Requests to the build strategies and pull requests should start building.

          John Mellor added a comment -

          Sorry for being obtuse, but pull requests ARE in the build stategies already.  I do not see what I should have to add to all the jobs.  Typical job screenshot attached.  What needs to be added?

          John Mellor added a comment - Sorry for being obtuse, but pull requests ARE in the build stategies already.  I do not see what I should have to add to all the jobs.  Typical job screenshot attached.  What needs to be added?

          Steve Mason added a comment - - edited

          alt_jmellor That section is "GitHub Organisation", you want to add the "Change requests" strategy under the "Build strategies" section. See this screenshot:

           

          Steve Mason added a comment - - edited alt_jmellor That section is "GitHub Organisation", you want to add the "Change requests" strategy under the "Build strategies" section. See this screenshot:  

          John Mellor added a comment -

          Thanks Steve, but I still do not have enough context.  I do not have that section in the job configs like you do.  What plugin provides that configuration section?  Screenshot of what I see attached.

          John Mellor added a comment - Thanks Steve, but I still do not have enough context.  I do not have that section in the job configs like you do.  What plugin provides that configuration section?  Screenshot of what I see attached.

          Andy Airey added a comment -

          I'm also facing this issue and also didn't see this option (using Bitbucket Branch Source Plugin).
          It only appears when you migrate to the "Named Branch Plugin", following this message from the Recommendations (the number with red background in the top bar of the UI in Jenkins):

          The "Automatic branch project triggering" option has been replaced with a "Named branch" build strategy

          Andy Airey added a comment - I'm also facing this issue and also didn't see this option (using Bitbucket Branch Source Plugin). It only appears when you migrate to the "Named Branch Plugin", following this message from the Recommendations (the number with red background in the top bar of the UI in Jenkins): The "Automatic branch project triggering" option has been replaced with a "Named branch" build strategy

          Mark Waite added a comment -

          aairey for the benefit of others that may refer to this bug report, would you be willing to attach a screenshot of the Bitbucket Branch Source Plugin configuration setting?

          I realize that this bug report is about the GitHub multibranch, but many readers may find it and assume that Bitbucket and GitHub plugins have the same options and same configurations (which they do not).

          Mark Waite added a comment - aairey  for the benefit of others that may refer to this bug report, would you be willing to attach a screenshot of the Bitbucket Branch Source Plugin configuration setting? I realize that this bug report is about the GitHub multibranch, but many readers may find it and assume that Bitbucket and GitHub plugins have the same options and same configurations (which they do not).

          Hi,

          Same problem on my side.

          Any update on this issue ?

          Thanks

          Sebastien Gagnon added a comment - Hi, Same problem on my side. Any update on this issue ? Thanks

          Mark Waite added a comment -

          sebgagn did you apply the instructions in the summary of the configuration change that is usually the best way to resolve the issue?

          Mark Waite added a comment - sebgagn did you apply the instructions in the summary of the configuration change that is usually the best way to resolve the issue?

          John Mellor added a comment -

          Instead of suggesting that people apply the workaround, having to fix hundreds of thousands of jobs worldwide, how about just reverting the breaking change?  I alone have 3570 jobs to inspect and make a code change to because of this error.  How do I get back to the point of NOT having to make this change, and simply have the github functionality work as expected?

           

          John Mellor added a comment - Instead of suggesting that people apply the workaround, having to fix hundreds of thousands of jobs worldwide, how about just reverting the breaking change?  I alone have 3570 jobs to inspect and make a code change to because of this error.  How do I get back to the point of NOT having to make this change, and simply have the github functionality work as expected?  

          alt_jmellor we understand your frustration, but please be more careful with your phrasing, and be respectful of people time here. This is not a customer support channel.

          Baptiste Mathus added a comment - alt_jmellor we understand your frustration, but please be more careful with your phrasing, and be respectful of people time here. This is not a customer support channel.

          Devin Nusbaum added a comment -

          Version 2.5.4 of Branch API plugin was just released. This version undeprecates the original property (and two others that were deprecated at the same time) and disables the automatic migration going forward (but does not try to reverse it automatically to avoid making things worse).

          Devin Nusbaum added a comment - Version 2.5.4 of Branch API plugin was just released. This version undeprecates the original property (and two others that were deprecated at the same time) and disables the automatic migration going forward (but does not try to reverse it automatically to avoid making things worse).

          Eugene G added a comment - - edited

          I upgraded Branch API plugin from 2.1.2 to 2.5.4 and still see the same message in the logs:

            Checking branch dummy
                ‘Jenkinsfile’ found
              Met criteria
          Changes detected: dummy (null → cd9913c17e529deb34a84ae6f3d21ab647fedd52)
          No automatic build triggered for dummy
          

          Build strategies for GitHub Organization contain "Change requests" strategy and "Ignore rebuilding merge branches when only the target branch changed" flag is enabled.
          "Discover branches" strategy has "Exclude branches that are also filed as PRs" value but there's no PR for the "dummy" branch that I created. Pull Requests trigger builds as expected, but new branches are ignored.

          Update: I just added another rule into the "Build strategies" section of the GitHub Organization properties. The "Named branches" rule with "*" wildcard triggered all the builds even though I still see that "null → something" in the logs.

             Checking branch dummy
                ‘Jenkinsfile’ found
              Met criteria
          Changes detected: dummy (null → cd9913c17e529deb34a84ae6f3d21ab647fedd52)
          Scheduled build for branch: dummy
          

          Eugene G added a comment - - edited I upgraded Branch API plugin from 2.1.2 to 2.5.4 and still see the same message in the logs: Checking branch dummy ‘Jenkinsfile’ found Met criteria Changes detected: dummy ( null → cd9913c17e529deb34a84ae6f3d21ab647fedd52) No automatic build triggered for dummy Build strategies for GitHub Organization contain "Change requests" strategy and "Ignore rebuilding merge branches when only the target branch changed" flag is enabled. "Discover branches" strategy has "Exclude branches that are also filed as PRs" value but there's no PR for the "dummy" branch that I created. Pull Requests trigger builds as expected, but new branches are ignored. Update: I just added another rule into the "Build strategies" section of the GitHub Organization properties. The "Named branches" rule with "*" wildcard triggered all the builds even though I still see that "null → something" in the logs. Checking branch dummy ‘Jenkinsfile’ found Met criteria Changes detected: dummy ( null → cd9913c17e529deb34a84ae6f3d21ab647fedd52) Scheduled build for branch: dummy

          Pedro added a comment - - edited

          As far as I understand, "Automatic Branch Project Triggering" should not be appearing on the settings anymore. But I'm still seeing that option on my instance, as shown below:


          And as some here have commented, I'm not able to see the "Build Strategies" section under Projects. Currently on Jenkins 2.289.1, Branch API v2.6.4.

          Excuse me if this is not the proper way to comment on an old thread, but what has been shown here solves a constraint I'm having and I'm not able to apply it because I'm not able to see it.

          UPDATE: Checked the documentation here and found that I was missing the Basic Branch Build Strategies plugin.

          Pedro added a comment - - edited As far as I understand, "Automatic Branch Project Triggering" should not be appearing on the settings anymore. But I'm still seeing that option on my instance, as shown below: And as some here have commented, I'm not able to see the "Build Strategies" section under Projects. Currently on Jenkins 2.289.1, Branch API v2.6.4. Excuse me if this is not the proper way to comment on an old thread, but what has been shown here solves a constraint I'm having and I'm not able to apply it because I'm not able to see it. UPDATE: Checked the documentation here and found that I was missing the Basic Branch Build Strategies plugin.

            dnusbaum Devin Nusbaum
            alt_jmellor John Mellor
            Votes:
            8 Vote for this issue
            Watchers:
            25 Start watching this issue

              Created:
              Updated:
              Resolved: