Installing plugins without restart don't work in case if non-refreshable extension is activated. In my case I had https://wiki.jenkins-ci.org/display/JENKINS/Persona+Plugin installed, and received following exception:
hudson.util.IOException2: Failed to dynamically deploy this plugin
at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:1137)
at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:955)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: hudson.util.IOException2: Failed to install build-name-setter-plugin plugin
at hudson.PluginManager.dynamicLoad(PluginManager.java:365)
at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:1133)
... 7 more
Caused by: jenkins.ExtensionRefreshException: hudson.plugins.persona.xml.XmlPersonaFinder@1664d71 doesn't support refresh
at jenkins.model.Jenkins.refreshExtensions(Jenkins.java:1970)
at hudson.PluginManager.dynamicLoad(PluginManager.java:358)
... 8 more
In the code, I see that ExtensionRefreshException thrown in Jenkins.refreshExtensions() is handled to raise IOException2 and fail plugin installation process.
So it seems that such behavior is by design? If so, then I'd suggest to change this behavior, and this bug report should be reclassified as feature request.