Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-27694

Download update data immediately when installing a new tool installer plugin

    • Icon: Improvement Improvement
    • Resolution: Fixed
    • Icon: Minor Minor
    • core
    • None

      I've seen it noted in various places recently (mailing list, IRC, personal experience) that the situation after installing a tool installer plugin is suboptimal.

      The JSON file containing the installation information for that tool is only downloaded periodically (daily?) along with all other tool installer data.

      Ideally, users should not have to wait up to 24 hours (or use a workaround: Manage Plugins > Advanced > Check Now) before they can use these plugins.

      So, whenever a DownloadService.Downloadable is loaded into Jenkins, it should fetch the corresponding JSON file from updates.jenkins-ci.org, if necessary.

          [JENKINS-27694] Download update data immediately when installing a new tool installer plugin

          Code changed in jenkins
          User: Daniel Beck
          Path:
          core/src/main/java/hudson/model/DownloadService.java
          http://jenkins-ci.org/commit/jenkins/63186180e0b17c486302cb4ac2c8add47096b637
          Log:
          [FIX JENKINS-27694] Update dynamically loaded plugins' Downloadables

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/java/hudson/model/DownloadService.java http://jenkins-ci.org/commit/jenkins/63186180e0b17c486302cb4ac2c8add47096b637 Log: [FIX JENKINS-27694] Update dynamically loaded plugins' Downloadables

          Code changed in jenkins
          User: Daniel Beck
          Path:
          core/src/main/java/hudson/model/DownloadService.java
          http://jenkins-ci.org/commit/jenkins/038edc1be0108cef96c4965f97d2ee5829ff038d
          Log:
          Merge pull request #1754 from daniel-beck/JENKINS-27694

          [FIX JENKINS-27694] Update dynamically loaded plugins' Downloadables

          Compare: https://github.com/jenkinsci/jenkins/compare/01bf18b83fda...038edc1be010

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/java/hudson/model/DownloadService.java http://jenkins-ci.org/commit/jenkins/038edc1be0108cef96c4965f97d2ee5829ff038d Log: Merge pull request #1754 from daniel-beck/ JENKINS-27694 [FIX JENKINS-27694] Update dynamically loaded plugins' Downloadables Compare: https://github.com/jenkinsci/jenkins/compare/01bf18b83fda...038edc1be010

          dogfood added a comment -

          Integrated in jenkins_main_trunk #4204
          [FIX JENKINS-27694] Update dynamically loaded plugins' Downloadables (Revision 63186180e0b17c486302cb4ac2c8add47096b637)

          Result = SUCCESS
          daniel-beck : 63186180e0b17c486302cb4ac2c8add47096b637
          Files :

          • core/src/main/java/hudson/model/DownloadService.java

          dogfood added a comment - Integrated in jenkins_main_trunk #4204 [FIX JENKINS-27694] Update dynamically loaded plugins' Downloadables (Revision 63186180e0b17c486302cb4ac2c8add47096b637) Result = SUCCESS daniel-beck : 63186180e0b17c486302cb4ac2c8add47096b637 Files : core/src/main/java/hudson/model/DownloadService.java

          Julien HENRY added a comment -

          I'm using Jenkins 1.625.1 and the problem is still not fixed. After installing SonarQube plugin I still have to manually click "Check now" to have SonarQube Runner installation list updated.

          Julien HENRY added a comment - I'm using Jenkins 1.625.1 and the problem is still not fixed. After installing SonarQube plugin I still have to manually click "Check now" to have SonarQube Runner installation list updated.

          Daniel Beck added a comment -

          henryju Anything interesting logged to hudson.model.DownloadService.DownloadableListener during installation (level FINER or higher)?

          How to set up logging: https://wiki.jenkins-ci.org/display/JENKINS/Logging

          Daniel Beck added a comment - henryju Anything interesting logged to hudson.model.DownloadService.DownloadableListener during installation (level FINER or higher)? How to set up logging: https://wiki.jenkins-ci.org/display/JENKINS/Logging

          Julien HENRY added a comment -

          Nothing never logged to hudson.model.DownloadService.DownloadableListener even after restarting the server, and even after manually forcing "Check now". In jenkins.log I can see after hitting "Check now":

          oct. 21, 2015 9:35:22 PM hudson.model.UpdateSite updateData
          INFOS: Obtained the latest update center data file for UpdateSource default
          oct. 21, 2015 9:35:23 PM hudson.model.DownloadService$Downloadable load
          INFOS: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller
          oct. 21, 2015 9:35:24 PM hudson.model.DownloadService$Downloadable load
          INFOS: Obtained the updated data file for hudson.plugins.sonar.SonarRunnerInstaller
          oct. 21, 2015 9:35:24 PM hudson.model.DownloadService$Downloadable load
          INFOS: Obtained the updated data file for hudson.tasks.Ant.AntInstaller
          oct. 21, 2015 9:35:27 PM hudson.model.DownloadService$Downloadable load
          INFOS: Obtained the updated data file for hudson.tools.JDKInstaller
          

          I don't know if that may be a hint but I do the plugin installation using a manually uploaded sonar.hpi (not using update center).

          Julien HENRY added a comment - Nothing never logged to hudson.model.DownloadService.DownloadableListener even after restarting the server, and even after manually forcing "Check now". In jenkins.log I can see after hitting "Check now": oct. 21, 2015 9:35:22 PM hudson.model.UpdateSite updateData INFOS: Obtained the latest update center data file for UpdateSource default oct. 21, 2015 9:35:23 PM hudson.model.DownloadService$Downloadable load INFOS: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller oct. 21, 2015 9:35:24 PM hudson.model.DownloadService$Downloadable load INFOS: Obtained the updated data file for hudson.plugins.sonar.SonarRunnerInstaller oct. 21, 2015 9:35:24 PM hudson.model.DownloadService$Downloadable load INFOS: Obtained the updated data file for hudson.tasks.Ant.AntInstaller oct. 21, 2015 9:35:27 PM hudson.model.DownloadService$Downloadable load INFOS: Obtained the updated data file for hudson.tools.JDKInstaller I don't know if that may be a hint but I do the plugin installation using a manually uploaded sonar.hpi (not using update center).

          I can confirm the same, with 1.625.1. The FINEST added logger does not display any FINE logs, even when Check Now is clicked.
          I observed this when installing golang plugin, regardless of method (via UI or via .hpi file).
          After doing Check Now, only the INFO logs are shown.
          This should be easily reproducible using Jenkins docker image (that I am actually using).

          Radek Antoniuk added a comment - I can confirm the same, with 1.625.1. The FINEST added logger does not display any FINE logs, even when Check Now is clicked. I observed this when installing golang plugin, regardless of method (via UI or via .hpi file). After doing Check Now, only the INFO logs are shown. This should be easily reproducible using Jenkins docker image (that I am actually using).

          FWIW, workaround via Script console:

          import jenkins.model.*
          
          Jenkins j = Jenkins.getInstance()
          j.pluginManager.doCheckUpdatesServer()
          

          Radek Antoniuk added a comment - FWIW, workaround via Script console: import jenkins.model.* Jenkins j = Jenkins.getInstance() j.pluginManager.doCheckUpdatesServer()

          Daniel Beck added a comment -

          Oops. The logger is supposed to be called hudson.model.DownloadService$DownloadableListener ($ sign). Note that this logger only logs when Jenkins would update its internal list of Downloadables, which happens during plugin installation – So "Check Now" not logging anything is to be expected.

          However, I just set up a pristine 1.625.1 instance, set up the logger, and installed the Go Plugin, and…

          Nov 05, 2015 12:46:37 AM FINER hudson.model.DownloadService$DownloadableListener
          Skipping update of metadata for hudson.tasks.Maven.MavenInstaller
          Nov 05, 2015 12:46:37 AM FINER hudson.model.DownloadService$DownloadableListener
          Skipping update of metadata for hudson.tasks.Ant.AntInstaller
          Nov 05, 2015 12:46:37 AM FINER hudson.model.DownloadService$DownloadableListener
          Skipping update of metadata for hudson.tools.JDKInstaller
          Nov 05, 2015 12:46:37 AM FINE hudson.model.DownloadService$DownloadableListener
          Updating metadata for org.jenkinsci.plugins.golang.GolangInstaller
          

          Going back to the global config, I get the versions for Go in a dropdown…

          Daniel Beck added a comment - Oops. The logger is supposed to be called hudson.model.DownloadService$DownloadableListener ( $ sign). Note that this logger only logs when Jenkins would update its internal list of Downloadables , which happens during plugin installation – So "Check Now" not logging anything is to be expected. However, I just set up a pristine 1.625.1 instance, set up the logger, and installed the Go Plugin, and… Nov 05, 2015 12:46:37 AM FINER hudson.model.DownloadService$DownloadableListener Skipping update of metadata for hudson.tasks.Maven.MavenInstaller Nov 05, 2015 12:46:37 AM FINER hudson.model.DownloadService$DownloadableListener Skipping update of metadata for hudson.tasks.Ant.AntInstaller Nov 05, 2015 12:46:37 AM FINER hudson.model.DownloadService$DownloadableListener Skipping update of metadata for hudson.tools.JDKInstaller Nov 05, 2015 12:46:37 AM FINE hudson.model.DownloadService$DownloadableListener Updating metadata for org.jenkinsci.plugins.golang.GolangInstaller Going back to the global config, I get the versions for Go in a dropdown…

          Daniel Beck added a comment -

          Cannot reproduce this not working, so resolving as fixed again.

          Daniel Beck added a comment - Cannot reproduce this not working, so resolving as fixed again.

          Sorry for late answer, steps to reproduce on docker:

          1. mkdir /tmp/jenkins && chmod 777 /tmp/jenkins && docker run --rm -v /tmp/jenkins:/var/jenkins_home -p 10000:8080 jenkins:1.625.1
          2. cd /tmp/jenkins/plugins  && wget http://updates.jenkins-ci.org/latest/golang.hpi
          3. restart Jenkins via /safeRestart
          

          Expected:

          • Jenkins unpacks the .hpi file and runs UpdateCenter to populate GoLang data dropdown

          Actual:

          • Jenkins unpacks the .hpi file but UpdateCenter is not run

          After hitting the "Check Now" on the UI, indeed I can see:

          Nov 24, 2015 7:28:16 PM hudson.model.UpdateSite updateData
          INFO: Obtained the latest update center data file for UpdateSource default
          Nov 24, 2015 7:28:16 PM hudson.model.DownloadService$Downloadable load
          INFO: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller
          Nov 24, 2015 7:28:17 PM hudson.model.DownloadService$Downloadable load
          INFO: Obtained the updated data file for hudson.tasks.Ant.AntInstaller
          Nov 24, 2015 7:28:19 PM hudson.model.DownloadService$Downloadable load
          INFO: Obtained the updated data file for hudson.tools.JDKInstaller
          Nov 24, 2015 7:28:21 PM hudson.model.DownloadService$Downloadable load
          INFO: Obtained the updated data file for org.jenkinsci.plugins.golang.GolangInstaller
          

          However, seems that DownloadService is not invoked automatically during the restart phase - what makes sense if you think of it - but in this case it makes the UI unusable, so maybe the golang plugin itself is not registering something properly?

          Radek Antoniuk added a comment - Sorry for late answer, steps to reproduce on docker: 1. mkdir /tmp/jenkins && chmod 777 /tmp/jenkins && docker run --rm -v /tmp/jenkins:/ var /jenkins_home -p 10000:8080 jenkins:1.625.1 2. cd /tmp/jenkins/plugins && wget http: //updates.jenkins-ci.org/latest/golang.hpi 3. restart Jenkins via /safeRestart Expected: Jenkins unpacks the .hpi file and runs UpdateCenter to populate GoLang data dropdown Actual: Jenkins unpacks the .hpi file but UpdateCenter is not run After hitting the "Check Now" on the UI, indeed I can see: Nov 24, 2015 7:28:16 PM hudson.model.UpdateSite updateData INFO: Obtained the latest update center data file for UpdateSource default Nov 24, 2015 7:28:16 PM hudson.model.DownloadService$Downloadable load INFO: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller Nov 24, 2015 7:28:17 PM hudson.model.DownloadService$Downloadable load INFO: Obtained the updated data file for hudson.tasks.Ant.AntInstaller Nov 24, 2015 7:28:19 PM hudson.model.DownloadService$Downloadable load INFO: Obtained the updated data file for hudson.tools.JDKInstaller Nov 24, 2015 7:28:21 PM hudson.model.DownloadService$Downloadable load INFO: Obtained the updated data file for org.jenkinsci.plugins.golang.GolangInstaller However, seems that DownloadService is not invoked automatically during the restart phase - what makes sense if you think of it - but in this case it makes the UI unusable, so maybe the golang plugin itself is not registering something properly?

          Update:
          This WORKS with jenkins:weekly image that is 1.596 (old for weekly though).
          Seems that it does not work with newer Jenkins 1.625.1...
          or another magic/race condition is happening - but I am consistently reproducing it with the above scenario.

          Radek Antoniuk added a comment - Update: This WORKS with jenkins:weekly image that is 1.596 (old for weekly though). Seems that it does not work with newer Jenkins 1.625.1... or another magic/race condition is happening - but I am consistently reproducing it with the above scenario.

          I can reproduce with that Docker command and with the `golang` plugin (which has that UI bug — JENKINS-27499), but it also happens with others, e.g. the `gradle` plugin.

          Though note that hitting "Check Now" fails, due to the signature error: https://groups.google.com/forum/#!topic/jenkinsci-users/vv3UAIE7-Mg

          So perhaps that's a reason why you see it failing with certain Jenkins versions — the metadata fetch process is failing in the background?

          Christopher Orr added a comment - I can reproduce with that Docker command and with the `golang` plugin (which has that UI bug — JENKINS-27499 ), but it also happens with others, e.g. the `gradle` plugin. Though note that hitting "Check Now" fails, due to the signature error: https://groups.google.com/forum/#!topic/jenkinsci-users/vv3UAIE7-Mg So perhaps that's a reason why you see it failing with certain Jenkins versions — the metadata fetch process is failing in the background?

          I don't think that this is related to the signature bug, because the manual Check Now would not work either and it worked for me (see above logs).

          Radek Antoniuk added a comment - I don't think that this is related to the signature bug, because the manual Check Now would not work either and it worked for me (see above logs).

          The workaround doesn't seem to work with Jenkins 1.625.2.

          Moreover, after clicking "Check now" the page displays
          "Signature verification failed in downloadable 'hudson.tasks.Maven.MavenInstaller' (show details)"
          where "show details" doesn't show any detail.

          Giacomo Boccardo added a comment - The workaround doesn't seem to work with Jenkins 1.625.2. Moreover, after clicking "Check now" the page displays "Signature verification failed in downloadable 'hudson.tasks.Maven.MavenInstaller' (show details)" where "show details" doesn't show any detail.

          The workaround should work fine, but you need to first do a workaround for the other JENKINS-31089.

          Radek Antoniuk added a comment - The workaround should work fine, but you need to first do a workaround for the other JENKINS-31089 .

          warden, that's amazing
          Thanks for you help.

          Giacomo Boccardo added a comment - warden , that's amazing Thanks for you help.

          Daniel Beck added a comment -

          Sorry for late answer, steps to reproduce on docker:

          As you did never really installed the plugin, but just put the file there, Jenkins never got the trigger that it needs to update its metadata. Install using the UI or the CLI and the issue will disappear. Support for scanning manually installed plugins, like you're doing, for missing metadata is a different issue.

          Your attempt with an older release is likely only the result of there not being (recent enough) update center metadata to begin with. If Jenkins downloads everything anew, like in your log snippet, it's a case of exactly that.

          Daniel Beck added a comment - Sorry for late answer, steps to reproduce on docker: As you did never really installed the plugin, but just put the file there, Jenkins never got the trigger that it needs to update its metadata. Install using the UI or the CLI and the issue will disappear. Support for scanning manually installed plugins, like you're doing, for missing metadata is a different issue. Your attempt with an older release is likely only the result of there not being (recent enough) update center metadata to begin with. If Jenkins downloads everything anew, like in your log snippet, it's a case of exactly that.

          A user of the Go plugin emailed me to ask about this problem, and we discovered that they were installing the plugin using the "Download now and install
          after restart" option.

          Indeed, I can reproduce that the tool installer data is downloaded automatically if you install a plugin by clicking "Install without restart", but if you click on "Download now and install after restart", then the tool installer metadata is not downloaded automatically after the download, nor when Jenkins restarts.

          Users should preferably install plugins without restarting, but so long as the other option exists, people are going to click on it. Or they'll install new plugins at the same time as updating older ones. There's also a (probably very rare) chance that a plugin update will add a tool installer. Jenkins should also download the tool installer data in these situations.

          Should this go into a new ticket?

          Christopher Orr added a comment - A user of the Go plugin emailed me to ask about this problem, and we discovered that they were installing the plugin using the "Download now and install after restart" option. Indeed, I can reproduce that the tool installer data is downloaded automatically if you install a plugin by clicking "Install without restart", but if you click on "Download now and install after restart", then the tool installer metadata is not downloaded automatically after the download, nor when Jenkins restarts. Users should preferably install plugins without restarting, but so long as the other option exists, people are going to click on it. Or they'll install new plugins at the same time as updating older ones. There's also a (probably very rare) chance that a plugin update will add a tool installer. Jenkins should also download the tool installer data in these situations. Should this go into a new ticket?

          Daniel Beck added a comment -

          orrc File a new bug, assign to me.

          Daniel Beck added a comment - orrc File a new bug, assign to me.

            danielbeck Daniel Beck
            orrc Christopher Orr
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: