Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41121

GitHub Branch Source upgrade can cause a lot of rebuilds

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • scm-api-plugin
    • None

      The gist of this is that due to some encoding change made in the SCM API 2.0 release, conventional URIs

      This was originally noticed by the Blue Ocean team in this pull request discussing integrating SCM API 2.0 changes. See also JENKINS-40299.

      The blog post discusses some of the data migrations which are supposed to take place, however the URI changing issue results in two bad behaviors:

      * URIs change for Multibranch-based Pipelines which use "special characters" in their names, such as improvement/JENKINS-41047 would previously be located at: https://ci.blueocean.io/job/blueocean/job/improvement%2FJENKINS-41047/. After these changes it would be found at https://ci.blueocean.io/job/blueocean/job/improvement-JENKINS-41047.d61e72/
      * Post-upgrade, rebooting the Jenkins installation results in *every* Multibranch Pipeline being retriggered. Such as this Pipeline despite no new changes being made.


      Because the upgrade to SCM API 2.0 is a one-direction-only change, I propose we blacklist the updated versions of the plugins referenced in this post before too many people upgrade a non-reversible upgrade.

      Incorrect analysis with lots of FUD.

      The renaming of jobs has nothing to do here.

      The issue here is that GitHub Branch Source 1.x passes a null id to the constructor of it's SCMSource: https://github.com/jenkinsci/github-branch-source-plugin/blob/1.10-stable/src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMNavigator.java#L312

      This means that every time you perform a scan of the organization folder, all the SCMSource instances get a new ID.

      When the branch changes SCMSource instance (i.e. the ID is different) then the Branch API is supposed to trigger a rebuild (because the branch is from a different source, never mind that it may be the same revision, we have no way of knowing)

      In 2.x we have a consistent ID: https://github.com/jenkinsci/github-branch-source-plugin/blob/master/src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMNavigator.java#L446

      The issue here is that GitHub Branch Source did not correctly ensure that the SCMSource from a Organization Folder had a consistent ID.

      Thus, every time you do a scan of an organization folder, with the old logic it should have been re-triggering a build... It wasn't because of other bugs... this leads to things like "I cannot re-trigger PRs", etc

      In fixing the problems in GitHub Branch Source, we get to a point where everything is behaving correctly, but the data for the previous branch origins is not current and thus the PRs have not been built from their current source.

          [JENKINS-41121] GitHub Branch Source upgrade can cause a lot of rebuilds

          Morgan Goose added a comment -

          stephenconnolly It doesn't look like the scm-api >= 2.0 has actually been released: http://updates.jenkins-ci.org/download/plugins/scm-api/

          Are you sure that this can be closed?

          Morgan Goose added a comment - stephenconnolly It doesn't look like the scm-api >= 2.0 has actually been released: http://updates.jenkins-ci.org/download/plugins/scm-api/ Are you sure that this can be closed?

          Stephen Connolly added a comment - - edited

          morgan_goose The issue was resolved with releases in the experimental update center. The plugin release has been completed and all tests are successful, but is currently being held back from the UCs by https://github.com/jenkinsci/backend-update-center2/pull/109 until Monday to allow for better responsiveness in the event of further issues.

          If you are eager to get the release, you can download this zip https://www.dropbox.com/s/5g4qm8gms53pxqc/scm-api.zip?dl=0 which has the latest versions of all the complete transitive dependency graph of Blue Ocean and SCM API (as Blue Ocean is required to be updated if you update SCM API)

          Otherwise, just wait until Monday

          Stephen Connolly added a comment - - edited morgan_goose The issue was resolved with releases in the experimental update center. The plugin release has been completed and all tests are successful, but is currently being held back from the UCs by https://github.com/jenkinsci/backend-update-center2/pull/109 until Monday to allow for better responsiveness in the event of further issues. If you are eager to get the release, you can download this zip https://www.dropbox.com/s/5g4qm8gms53pxqc/scm-api.zip?dl=0 which has the latest versions of all the complete transitive dependency graph of Blue Ocean and SCM API (as Blue Ocean is required to be updated if you update SCM API) Otherwise, just wait until Monday

          Here is the draft of the blog post to accompany Monday's release into the UC https://github.com/jenkins-infra/jenkins.io/pull/580

          Stephen Connolly added a comment - Here is the draft of the blog post to accompany Monday's release into the UC https://github.com/jenkins-infra/jenkins.io/pull/580

          Morgan Goose added a comment -

          But there are already plugins available for update that explicitly require this version of the scm plugin. Withholding the release is breaking Jenkins stacks. Letting users only upgrade partially, causes Jenkins to suggests resolution of "old data" via Deleting said data.

          If a core plugin is the hub of any number of spoke plugins that require it's use, it's seem pretty disruptive to not release them all at the same time? I've already grabbed the two hpi files, and installed manually:

          • github-organization-folder-1.6.hpi
          • scm-api-2.0.2.hpi

          from https://repo.jenkins-ci.org/public/org/jenkins-ci/plugins/

          Was this release standard? Or did it follow a different track than usual?

          Morgan Goose added a comment - But there are already plugins available for update that explicitly require this version of the scm plugin. Withholding the release is breaking Jenkins stacks. Letting users only upgrade partially, causes Jenkins to suggests resolution of "old data" via Deleting said data. If a core plugin is the hub of any number of spoke plugins that require it's use, it's seem pretty disruptive to not release them all at the same time? I've already grabbed the two hpi files, and installed manually: github-organization-folder-1.6.hpi scm-api-2.0.2.hpi from https://repo.jenkins-ci.org/public/org/jenkins-ci/plugins/ Was this release standard? Or did it follow a different track than usual?

          morgan_goose

          Manually downloading plugins from https://repo.jenkins-ci.org/public/org/jenkins-ci/plugins/ is neither the normal way to install plugins nor the recommended way.

          When you manually download plugins you have to take responsibility to manually download the required dependencies.

          For the SCM API and Blue Ocean related updates that is potentially a list of updates that could grow to 72 plugins depending on what plugins and versions you have installed prior to upgrading... though an otherwise up to date system might only need 24 plugins... and if you do not have Blue Ocean installed that number could drop to somewhere between 7 and 9.

          Given that you have manually installed more than one of the plugins, my recommendation is that you download the ZIP file that I gave a link to in https://issues.jenkins-ci.org/browse/JENKINS-41661 and use the plugins in that ZIP file to update your instance.

          For future reference, the recommended path for installing plugins is via the update center. There are safety checks that the update center applies to help prevent issues. While we can agree that there are more additional safety checks needed, the limited checks it does have are better than no checks which is the net effect of manual downloading.

          Stephen Connolly added a comment - morgan_goose Manually downloading plugins from https://repo.jenkins-ci.org/public/org/jenkins-ci/plugins/ is neither the normal way to install plugins nor the recommended way. When you manually download plugins you have to take responsibility to manually download the required dependencies. For the SCM API and Blue Ocean related updates that is potentially a list of updates that could grow to 72 plugins depending on what plugins and versions you have installed prior to upgrading... though an otherwise up to date system might only need 24 plugins... and if you do not have Blue Ocean installed that number could drop to somewhere between 7 and 9. Given that you have manually installed more than one of the plugins, my recommendation is that you download the ZIP file that I gave a link to in https://issues.jenkins-ci.org/browse/JENKINS-41661 and use the plugins in that ZIP file to update your instance. For future reference, the recommended path for installing plugins is via the update center. There are safety checks that the update center applies to help prevent issues. While we can agree that there are more additional safety checks needed, the limited checks it does have are better than no checks which is the net effect of manual downloading.

          I can tell you that I did not download any plugins by hand, and only used the update center. And still wound up reading this issue.

          Echo everything that Morgan Goose said. I downgraded some plugins because that appeared to be the recommended path, and I only narrowly avoided deleting my settings after reading the same warnings that you saw about old data. Maybe I caught the upgrade moments before you blacklisted the 2.0.1 update? Not sure. The timing is about right though.

          Thanks for reposting the ZIP link. I see that you're planning a Monday release, I will wait for that, thanks for providing a clear(ish) path in the issue documentation. FWIW none of the plugins that complained at me about bad dependency versions, actually appeared to be malfunctioning, but I am not using Blue Ocean or advanced GitHub API plugins yet.

          Kingdon Barrett added a comment - I can tell you that I did not download any plugins by hand, and only used the update center. And still wound up reading this issue. Echo everything that Morgan Goose said. I downgraded some plugins because that appeared to be the recommended path, and I only narrowly avoided deleting my settings after reading the same warnings that you saw about old data. Maybe I caught the upgrade moments before you blacklisted the 2.0.1 update? Not sure. The timing is about right though. Thanks for reposting the ZIP link. I see that you're planning a Monday release, I will wait for that, thanks for providing a clear(ish) path in the issue documentation. FWIW none of the plugins that complained at me about bad dependency versions, actually appeared to be malfunctioning, but I am not using Blue Ocean or advanced GitHub API plugins yet.

          kingdonb you have a different case from Morgan.

          Morgan said:

          I've already grabbed the two hpi files, and installed manually:

          Whereas you downloaded the plugins via the recommended path (and were just bit by the issues described in JENKINS-41661 which is where I am complaining that the UC does not do enough validation. The UC method does at least try to upgrade all the required dependencies... it should grey out plugins that cannot be installed IMHO)

          Kingdon, I recommend reading https://wiki.jenkins-ci.org/display/JENKINS/JENKINS-41661+Post-Mortem for more details on the JENKINS-41661 issue that affected you.

          The plugins will all be released on Monday.

          Sorry for any inconvenience caused.

          Stephen Connolly added a comment - kingdonb you have a different case from Morgan. Morgan said: I've already grabbed the two hpi files, and installed manually: github-organization-folder-1.6.hpi scm-api-2.0.2.hpi from https://repo.jenkins-ci.org/public/org/jenkins-ci/plugins/ Whereas you downloaded the plugins via the recommended path (and were just bit by the issues described in JENKINS-41661 which is where I am complaining that the UC does not do enough validation. The UC method does at least try to upgrade all the required dependencies... it should grey out plugins that cannot be installed IMHO) Kingdon, I recommend reading https://wiki.jenkins-ci.org/display/JENKINS/JENKINS-41661+Post-Mortem for more details on the JENKINS-41661 issue that affected you. The plugins will all be released on Monday. Sorry for any inconvenience caused.

          Yeah, sounds like you're on top of things. Lesson learned: full backups immediately before every upgrade!

          No trouble at all, thanks for your contribution!

          Kingdon Barrett added a comment - Yeah, sounds like you're on top of things. Lesson learned: full backups immediately before every upgrade! No trouble at all, thanks for your contribution!

          kingdonb

          This was a particularly troublesome set of upgrades. Due to the mistakes made in GHBS and BBBS (because of lack of documentation) we had to break data compatibility in a particularly complicated way that prevented the normal evolution mode in Jenkins.

          Hopefully we will not be doing that again any time in the near future

          Stephen Connolly added a comment - kingdonb This was a particularly troublesome set of upgrades. Due to the mistakes made in GHBS and BBBS (because of lack of documentation) we had to break data compatibility in a particularly complicated way that prevented the normal evolution mode in Jenkins. Hopefully we will not be doing that again any time in the near future

          Morgan Goose added a comment -

          stephenconnolly I'd like to clarify, that I only installed those two plugins manually after multiple plugins failed to load saying that these versions were required. All of those complaining plugins were shown as having updates available and were updated in the suggested way (via the plugin update center web UI).

          Morgan Goose added a comment - stephenconnolly I'd like to clarify, that I only installed those two plugins manually after multiple plugins failed to load saying that these versions were required. All of those complaining plugins were shown as having updates available and were updated in the suggested way (via the plugin update center web UI).

            stephenconnolly Stephen Connolly
            rtyler R. Tyler Croy
            Votes:
            4 Vote for this issue
            Watchers:
            31 Start watching this issue

              Created:
              Updated:
              Resolved: