Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41121

GitHub Branch Source upgrade can cause a lot of rebuilds

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • scm-api-plugin
    • None

      The gist of this is that due to some encoding change made in the SCM API 2.0 release, conventional URIs

      This was originally noticed by the Blue Ocean team in this pull request discussing integrating SCM API 2.0 changes. See also JENKINS-40299.

      The blog post discusses some of the data migrations which are supposed to take place, however the URI changing issue results in two bad behaviors:

      * URIs change for Multibranch-based Pipelines which use "special characters" in their names, such as improvement/JENKINS-41047 would previously be located at: https://ci.blueocean.io/job/blueocean/job/improvement%2FJENKINS-41047/. After these changes it would be found at https://ci.blueocean.io/job/blueocean/job/improvement-JENKINS-41047.d61e72/
      * Post-upgrade, rebooting the Jenkins installation results in *every* Multibranch Pipeline being retriggered. Such as this Pipeline despite no new changes being made.


      Because the upgrade to SCM API 2.0 is a one-direction-only change, I propose we blacklist the updated versions of the plugins referenced in this post before too many people upgrade a non-reversible upgrade.

      Incorrect analysis with lots of FUD.

      The renaming of jobs has nothing to do here.

      The issue here is that GitHub Branch Source 1.x passes a null id to the constructor of it's SCMSource: https://github.com/jenkinsci/github-branch-source-plugin/blob/1.10-stable/src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMNavigator.java#L312

      This means that every time you perform a scan of the organization folder, all the SCMSource instances get a new ID.

      When the branch changes SCMSource instance (i.e. the ID is different) then the Branch API is supposed to trigger a rebuild (because the branch is from a different source, never mind that it may be the same revision, we have no way of knowing)

      In 2.x we have a consistent ID: https://github.com/jenkinsci/github-branch-source-plugin/blob/master/src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMNavigator.java#L446

      The issue here is that GitHub Branch Source did not correctly ensure that the SCMSource from a Organization Folder had a consistent ID.

      Thus, every time you do a scan of an organization folder, with the old logic it should have been re-triggering a build... It wasn't because of other bugs... this leads to things like "I cannot re-trigger PRs", etc

      In fixing the problems in GitHub Branch Source, we get to a point where everything is behaving correctly, but the data for the previous branch origins is not current and thus the PRs have not been built from their current source.

          [JENKINS-41121] GitHub Branch Source upgrade can cause a lot of rebuilds

          R. Tyler Croy created issue -
          R. Tyler Croy made changes -
          Link New: This issue is related to JENKINS-40299 [ JENKINS-40299 ]
          R. Tyler Croy made changes -
          Summary Original: New scm-api versions changes branch name encoding breaking URIs and causing data loss New: New scm-api versions changes branch name encoding breaking URIs and DoSing
          R. Tyler Croy made changes -
          Summary Original: New scm-api versions changes branch name encoding breaking URIs and DoSing New: New scm-api versions changes branch name encoding breaking URIs and can overload the environment
          R. Tyler Croy made changes -
          Assignee Original: R. Tyler Croy [ rtyler ]
          Stephen Connolly made changes -
          Summary Original: New scm-api versions changes branch name encoding breaking URIs and can overload the environment New: GitHub Branch Source upgrade can cause a lot of rebuilds
          Stephen Connolly made changes -
          Description Original: The gist of this is that due to some encoding change made in the SCM API 2.0 release, conventional URIs

          This was originally noticed by the Blue Ocean team in [this pull request|https://github.com/jenkinsci/blueocean-plugin/pull/650#issuecomment-272958642] discussing integrating SCM API 2.0 changes. See also JENKINS-40299.

          [The blog post|https://jenkins.io/blog/2017/01/17/scm-api-2/] discusses some of the data migrations which are supposed to take place, however the URI changing issue results in two bad behaviors:

          * URIs change for Multibranch-based Pipelines which use "special characters" in their names, such as {{improvement/JENKINS-41047}} would previously be located at: {{https://ci.blueocean.io/job/blueocean/job/improvement%2FJENKINS-41047/}}. After these changes it would be found at {{https://ci.blueocean.io/job/blueocean/job/improvement-JENKINS-41047.d61e72/}}
          * Post-upgrade, rebooting the Jenkins installation results in **every** Multibranch Pipeline being retriggered. Such as [this Pipeline|https://ci.blueocean.io/job/blueocean/job/JENKINS-39625-part1/] despite no new changes being made.


          Because the upgrade to SCM API 2.0 is a one-direction-only change, I propose we blacklist the updated versions of the plugins referenced in [this post|https://jenkins.io/blog/2017/01/17/scm-api-2/] before too many people upgrade a non-reversible upgrade.
          New: {color:red}The gist of this is that due to some encoding change made in the SCM API 2.0 release, conventional URIs

          This was originally noticed by the Blue Ocean team in [this pull request|https://github.com/jenkinsci/blueocean-plugin/pull/650#issuecomment-272958642] discussing integrating SCM API 2.0 changes. See also JENKINS-40299.

          [The blog post|https://jenkins.io/blog/2017/01/17/scm-api-2/] discusses some of the data migrations which are supposed to take place, however the URI changing issue results in two bad behaviors:

          * URIs change for Multibranch-based Pipelines which use "special characters" in their names, such as {{improvement/JENKINS-41047}} would previously be located at: {{https://ci.blueocean.io/job/blueocean/job/improvement%2FJENKINS-41047/}}. After these changes it would be found at {{https://ci.blueocean.io/job/blueocean/job/improvement-JENKINS-41047.d61e72/}}
          * Post-upgrade, rebooting the Jenkins installation results in **every** Multibranch Pipeline being retriggered. Such as [this Pipeline|https://ci.blueocean.io/job/blueocean/job/JENKINS-39625-part1/] despite no new changes being made.


          Because the upgrade to SCM API 2.0 is a one-direction-only change, I propose we blacklist the updated versions of the plugins referenced in [this post|https://jenkins.io/blog/2017/01/17/scm-api-2/] before too many people upgrade a non-reversible upgrade.{color}

          Incorrect analysis with lots of FUD.

          The issue here is that GitHub Branch Source did not correctly ensure that the SCMSource from a Organization Folder had a consistent ID.

          Thus, every time you do a scan of an organization folder, with the old logic it should have been re-triggering a build... It wasn't because of other bugs... this leads to things like "I cannot re-trigger PRs", etc

          In fixing the problems in GitHub Branch Source, we get to a point where everything is behaving correctly, but the data for the previous branch origins is not current and thus the PRs have not been built from their current source.
          Stephen Connolly made changes -
          Description Original: {color:red}The gist of this is that due to some encoding change made in the SCM API 2.0 release, conventional URIs

          This was originally noticed by the Blue Ocean team in [this pull request|https://github.com/jenkinsci/blueocean-plugin/pull/650#issuecomment-272958642] discussing integrating SCM API 2.0 changes. See also JENKINS-40299.

          [The blog post|https://jenkins.io/blog/2017/01/17/scm-api-2/] discusses some of the data migrations which are supposed to take place, however the URI changing issue results in two bad behaviors:

          * URIs change for Multibranch-based Pipelines which use "special characters" in their names, such as {{improvement/JENKINS-41047}} would previously be located at: {{https://ci.blueocean.io/job/blueocean/job/improvement%2FJENKINS-41047/}}. After these changes it would be found at {{https://ci.blueocean.io/job/blueocean/job/improvement-JENKINS-41047.d61e72/}}
          * Post-upgrade, rebooting the Jenkins installation results in **every** Multibranch Pipeline being retriggered. Such as [this Pipeline|https://ci.blueocean.io/job/blueocean/job/JENKINS-39625-part1/] despite no new changes being made.


          Because the upgrade to SCM API 2.0 is a one-direction-only change, I propose we blacklist the updated versions of the plugins referenced in [this post|https://jenkins.io/blog/2017/01/17/scm-api-2/] before too many people upgrade a non-reversible upgrade.{color}

          Incorrect analysis with lots of FUD.

          The issue here is that GitHub Branch Source did not correctly ensure that the SCMSource from a Organization Folder had a consistent ID.

          Thus, every time you do a scan of an organization folder, with the old logic it should have been re-triggering a build... It wasn't because of other bugs... this leads to things like "I cannot re-trigger PRs", etc

          In fixing the problems in GitHub Branch Source, we get to a point where everything is behaving correctly, but the data for the previous branch origins is not current and thus the PRs have not been built from their current source.
          New: {color:red}The gist of this is that due to some encoding change made in the SCM API 2.0 release, conventional URIs

          This was originally noticed by the Blue Ocean team in [this pull request|https://github.com/jenkinsci/blueocean-plugin/pull/650#issuecomment-272958642] discussing integrating SCM API 2.0 changes. See also JENKINS-40299.

          [The blog post|https://jenkins.io/blog/2017/01/17/scm-api-2/] discusses some of the data migrations which are supposed to take place, however the URI changing issue results in two bad behaviors:

          * URIs change for Multibranch-based Pipelines which use "special characters" in their names, such as {{improvement/JENKINS-41047}} would previously be located at: {{https://ci.blueocean.io/job/blueocean/job/improvement%2FJENKINS-41047/}}. After these changes it would be found at {{https://ci.blueocean.io/job/blueocean/job/improvement-JENKINS-41047.d61e72/}}
          * Post-upgrade, rebooting the Jenkins installation results in **every** Multibranch Pipeline being retriggered. Such as [this Pipeline|https://ci.blueocean.io/job/blueocean/job/JENKINS-39625-part1/] despite no new changes being made.


          Because the upgrade to SCM API 2.0 is a one-direction-only change, I propose we blacklist the updated versions of the plugins referenced in [this post|https://jenkins.io/blog/2017/01/17/scm-api-2/] before too many people upgrade a non-reversible upgrade.{color}

          Incorrect analysis with lots of FUD.

          The renaming of jobs has nothing to do here.

          The issue here is that GitHub Branch Source 1.x passes a {{null}} id to the constructor of it's SCMSource: https://github.com/jenkinsci/github-branch-source-plugin/blob/1.10-stable/src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMNavigator.java#L312

          This means that every time you perform a scan of the organization folder, all the SCMSource instances get a new ID.

          When the branch changes SCMSource instance (i.e. the ID is different) then the Branch API is supposed to trigger a rebuild (because the branch is from a different source, never mind that it *may* be the same revision, we have no way of knowing)

          In 2.x we have a consistent ID: https://github.com/jenkinsci/github-branch-source-plugin/blob/master/src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMNavigator.java#L446

          The issue here is that GitHub Branch Source did not correctly ensure that the SCMSource from a Organization Folder had a consistent ID.

          Thus, every time you do a scan of an organization folder, with the old logic it should have been re-triggering a build... It wasn't because of other bugs... this leads to things like "I cannot re-trigger PRs", etc

          In fixing the problems in GitHub Branch Source, we get to a point where everything is behaving correctly, but the data for the previous branch origins is not current and thus the PRs have not been built from their current source.
          Ryan Campbell made changes -
          Epic Link New: JENKINS-41234 [ 177999 ]
          Stephen Connolly made changes -
          Remote Link New: This issue links to "scm-api-plugin#24 (Web Link)" [ 15295 ]
          Stephen Connolly made changes -
          Remote Link New: This issue links to "branch-api-plugin#75 (Web Link)" [ 15296 ]

            stephenconnolly Stephen Connolly
            rtyler R. Tyler Croy
            Votes:
            4 Vote for this issue
            Watchers:
            31 Start watching this issue

              Created:
              Updated:
              Resolved: