Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48365

Command launcher plugin does not get installed on core update crossing 2.86

      The plugin manager is supposed to install detached plugins if you upgrade from a version which previously had their functionality as part of core to a version which now has it detached. This is not working for some reason and we need to investigate why.

      Steps to reproduce:

      Install 2.84

      Pass setup wizard installing no plugins.

      Update to 2.93 for same home

      Expected:

      Update installs command-launcher plugin (plus deps as needed)

      Actual:

      No plugins installed

          [JENKINS-48365] Command launcher plugin does not get installed on core update crossing 2.86

          Jesse Glick added a comment -

          Possibly related to this apparent bug. I think we have no test coverage for this kind of upgrade scenario because AFAIK JenkinsRule cannot express it.

          Jesse Glick added a comment - Possibly related to this apparent bug . I think we have no test coverage for this kind of upgrade scenario because AFAIK JenkinsRule cannot express it.

          Daniel Beck added a comment -

          Can the ATH do upgrade tests? We could go from 2.85 to SUT and see whether the plugin gets installed (for every future pair of 'attached core version' and 'plugin')

          Daniel Beck added a comment - Can the ATH do upgrade tests? We could go from 2.85 to SUT and see whether the plugin gets installed (for every future pair of 'attached core version' and 'plugin')

          Devin Nusbaum added a comment -

          I was able to reproduce the issue. I didn't see any log entries about upgrading during the upgrade process (even with a custom logger for PluginManager). I used the "Upgrade automatically" button from CoreUpdateMonitor, is it expected for the block in the following if statement to not be entered in that case?

          if (InstallState.UPGRADE.equals(installState)) {
              LOGGER.log(INFO, "Upgrading Jenkins. ...
              ...
          } else {
              ... code Jesse linked to ...
          }
          

          The logic in the else would only upgrade a detached plugin that is already installed which is not the case for command-launcher when upgrading from 2.84 to 2.93. Maybe a recent change has broken something with InstallState?

          Devin Nusbaum added a comment - I was able to reproduce the issue. I didn't see any log entries about upgrading during the upgrade process (even with a custom logger for PluginManager). I used the "Upgrade automatically" button from CoreUpdateMonitor, is it expected for the block in the following if statement to not be entered in that case? if (InstallState.UPGRADE.equals(installState)) { LOGGER.log(INFO, "Upgrading Jenkins. ... ... } else { ... code Jesse linked to ... } The logic in the else would only upgrade a detached plugin that is already installed which is not the case for command-launcher when upgrading from 2.84 to 2.93. Maybe a recent change has broken something with InstallState?

          Daniel Beck added a comment -

          Maybe a recent change has broken something with InstallState?

          That's my guess as well, but so far we have no idea when this last worked. Step 1 is determining when it broke. Note that there are plenty of other detached plugins that are supposed to be installed when you go from 'part of core' to 'in a plugin'. The previous last detached plugin was between 2.7.x and 2.19.x, so this may have been broken for a while (in my limited testing, it appears to have been broken even back then, or has some nontrivial precondition – going from 2.0 to 2.19.4, even having saved the global config and having a job set up, did not install bouncycastle-api).

          If you're going by whether a detached plugin gets installed, you need a variation of the usual bisect search for the culprit version ensuring there's always at least one plugin

          Daniel Beck added a comment - Maybe a recent change has broken something with InstallState? That's my guess as well, but so far we have no idea when this last worked. Step 1 is determining when it broke. Note that there are plenty of other detached plugins that are supposed to be installed when you go from 'part of core' to 'in a plugin'. The previous last detached plugin was between 2.7.x and 2.19.x, so this may have been broken for a while (in my limited testing, it appears to have been broken even back then, or has some nontrivial precondition – going from 2.0 to 2.19.4, even having saved the global config and having a job set up, did not install bouncycastle-api). If you're going by whether a detached plugin gets installed, you need a variation of the usual bisect search for the culprit version ensuring there's always at least one plugin

          Devin Nusbaum added a comment - - edited

          As far as I can tell this has never worked in 2.X. In 1.X all of the plugins bundled in the war get installed, so the detached plugins are added to new installations without problem. I tested various version combos with the first version before 2.16 (including 1.651), and the second after 2.18, but bouncycastle-api was never installed (I tried installing plugins, creating jobs, and changing config.xml before the upgrade but bouncycastle-api was never installed).

          After attaching a debugger and doing an upgrade, I realized PluginManager#loadDetachedPlugins is called before InstallUtil#getDefaultInstallState, so the upgrade branch will never be taken. Looking through Jenkins' constructor, the call to Jenkins#executeReactor (which will execute the 'Loading bundled plugins' task that calls PluginManager#loadDetachedPlugins) does indeed happen a few lines before the call to InstallUtil#proceedToNext (which will update the install state to UPGRADE if an upgrade occurs). As far as I can tell the call to executeReactor does not return until all tasks are complete (though the tasks run in parallel by default), so there isn't even a race condition that would allow this to work occasionally.

          I think the easiest fix here is to check InstallUtil.getLastExecVersion() against Jenkins.VERSION directly in PluginManager#loadDetachedPlugins since Jenkins.installState hasn't been initialized.

          Devin Nusbaum added a comment - - edited As far as I can tell this has never worked in 2.X. In 1.X all of the plugins bundled in the war get installed, so the detached plugins are added to new installations without problem. I tested various version combos with the first version before 2.16 (including 1.651), and the second after 2.18, but bouncycastle-api was never installed (I tried installing plugins, creating jobs, and changing config.xml before the upgrade but bouncycastle-api was never installed). After attaching a debugger and doing an upgrade, I realized PluginManager#loadDetachedPlugins is called before InstallUtil#getDefaultInstallState , so the upgrade branch will never be taken. Looking through Jenkins' constructor, the call to Jenkins#executeReactor (which will execute the 'Loading bundled plugins' task that calls PluginManager#loadDetachedPlugins ) does indeed happen a few lines before the call to  InstallUtil#proceedToNext (which will update the install state to UPGRADE if an upgrade occurs). As far as I can tell the call to executeReactor does not return until all tasks are complete (though the tasks run in parallel by default), so there isn't even a race condition that would allow this to work occasionally. I think the easiest fix here is to check InstallUtil.getLastExecVersion()  against Jenkins.VERSION  directly in PluginManager#loadDetachedPlugins since Jenkins.installState hasn't been initialized.

          Daniel Beck added a comment -

          Fixed towards 2.96

          Daniel Beck added a comment - Fixed towards 2.96

            dnusbaum Devin Nusbaum
            danielbeck Daniel Beck
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: