-
Bug
-
Resolution: Fixed
-
Critical
-
Powered by SuggestiMate -
Jenkins 2.178
It is a placeholder for https://github.com/jenkinsci/docker/issues/698 and for https://github.com/jenkinsci/jenkins/pull/4000 which addresses it in the core
Using newer cores that have part of it moved to plugins and is now implied dependencies in other plugins is causing to have bad Jenkins installation.
- causes
-
JENKINS-58362 CommandLauncher2Test.requireApproval failure due to two copies of SystemCommandLanguage
-
- Resolved
-
- is related to
-
JENKINS-55582 Convert modules to plugins
-
- Resolved
-
- relates to
-
JENKINS-59552 Detached plugins installed are those with security warnings
-
- Closed
-
- links to
[JENKINS-57528] Jenkins in Docker does not install detached plugins when there is no UC data
This change seems to be breaking our Jenkins when we upgrade it from 2.177 to 2.178.
We tried both the ways:
- Upgraded all plugins to latest before upgrading the Jenkins version from 2.177 to 2.178
- Upgraded Jenkins to 2.178 and we see these dependency errors before we upgrade all the plugins manually with some errors.
I think both the times 2.178 is trying to load lower versions of these implied plugins during the startup and when later actual plugins are being installed and if they require greater version of these base dependencies the installation of that plugin is failing and the chain continues.
Not very sure though, can we have more documentation on this implied vs detached plugins and also required plugins and so on. Thank you.
Is there anyway to disable this functionality of installing these plugins by default and have users an option to explicitly define all these plugins (like what we do today)?
This log it has already installed the script-security 1.56 during the startup however during the actual installation it required 1.58
Here with few lines of exception:
Running from: /root/.ivy2/cache/jenkins/wars/jenkins-2.178.war webroot: EnvVars.masterEnvVars.get("JENKINS_HOME") 2019-05-22 12:43:34.013+0000 [id=1] INFO org.eclipse.jetty.util.log.Log#initialized: Logging initialized @775ms to org.eclipse.jetty.util.log.JavaUtilLog 2019-05-22 12:43:34.234+0000 [id=1] INFO winstone.Logger#logInternal: Beginning extraction from war file 2019-05-22 12:43:34.304+0000 [id=1] WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath 2019-05-22 12:43:34.427+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: jetty-9.4.z-SNAPSHOT; built: 2019-05-02T00:04:53.875Z; git: e1bc35120a6617ee3df052294e433f3a25ce7097; jvm 1.8.0_191-b12 2019-05-22 12:43:34.893+0000 [id=1] INFO o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet 2019-05-22 12:43:35.021+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: DefaultSessionIdManager workerName=node0 2019-05-22 12:43:35.021+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: No SessionScavenger set, using defaults 2019-05-22 12:43:35.029+0000 [id=1] INFO o.e.j.server.session.HouseKeeper#startScavenging: node0 Scavenging every 600000ms Jenkins home directory: /var/jenkins found at: EnvVars.masterEnvVars.get("JENKINS_HOME") 2019-05-22 12:43:35.881+0000 [id=1] INFO o.e.j.s.handler.ContextHandler#doStart: Started w.@565f390{Jenkins v2.178,/,file:///var/jenkins/war/,AVAILABLE}{/var/jenkins/war} 2019-05-22 12:43:35.924+0000 [id=1] INFO o.e.j.server.AbstractConnector#doStart: Started ServerConnector@668bc3d5{HTTP/1.1,[http/1.1]}{0.0.0.0:8080} 2019-05-22 12:43:35.926+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: Started @2690ms 2019-05-22 12:43:35.936+0000 [id=21] INFO winstone.Logger#logInternal: Winstone Servlet Engine v4.0 running: controlPort=disabled 2019-05-22 12:43:37.888+0000 [id=28] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization 2019-05-22 12:43:38.261+0000 [id=31] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin as a dependency: /var/jenkins/plugins/command-launcher.jpi 2019-05-22 12:43:38.276+0000 [id=31] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin as a dependency: /var/jenkins/plugins/jdk-tool.jpi 2019-05-22 12:43:38.306+0000 [id=31] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin as a dependency: /var/jenkins/plugins/script-security.jpi 2019-05-22 12:43:38.358+0000 [id=28] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin as a dependency: /var/jenkins/plugins/bouncycastle-api.jpi 2019-05-22 12:43:38.810+0000 [id=33] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin as a dependency: /var/jenkins/plugins/junit.jpi 2019-05-22 12:43:39.122+0000 [id=33] WARNING hudson.ClassicPluginStrategy#createClassJarFromWebInfClasses: Created /var/jenkins/plugins/gradle-1.31/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness 2019-05-22 12:43:39.280+0000 [id=33] WARNING hudson.ClassicPluginStrategy#createClassJarFromWebInfClasses: Created /var/jenkins/plugins/pegdown-formatter-1.3/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness 2019-05-22 12:43:39.286+0000 [id=33] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin as a dependency: /var/jenkins/plugins/ant.jpi 2019-05-22 12:43:39.669+0000 [id=33] WARNING hudson.ClassicPluginStrategy#createClassJarFromWebInfClasses: Created /var/jenkins/plugins/ant/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness 2019-05-22 12:43:39.673+0000 [id=33] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin as a dependency: /var/jenkins/plugins/external-monitor-job.jpi 2019-05-22 12:43:39.750+0000 [id=33] WARNING hudson.ClassicPluginStrategy#createClassJarFromWebInfClasses: Created /var/jenkins/plugins/external-monitor-job/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness 2019-05-22 12:43:39.756+0000 [id=33] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin as a dependency: /var/jenkins/plugins/matrix-auth.jpi 2019-05-22 12:43:40.098+0000 [id=33] WARNING hudson.ClassicPluginStrategy#createClassJarFromWebInfClasses: Created /var/jenkins/plugins/matrix-auth/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness 2019-05-22 12:43:40.127+0000 [id=33] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin as a dependency: /var/jenkins/plugins/windows-slaves.jpi 2019-05-22 12:43:40.160+0000 [id=33] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin as a dependency: /var/jenkins/plugins/antisamy-markup-formatter.jpi 2019-05-22 12:43:40.177+0000 [id=33] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin as a dependency: /var/jenkins/plugins/matrix-project.jpi 2019-05-22 12:43:40.206+0000 [id=33] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin as a dependency: /var/jenkins/plugins/mailer.jpi 2019-05-22 12:43:40.216+0000 [id=33] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin as a dependency: /var/jenkins/plugins/display-url-api.jpi 2019-05-22 12:43:40.221+0000 [id=33] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin as a dependency: /var/jenkins/plugins/ldap.jpi 2019-05-22 12:43:40.521+0000 [id=33] WARNING hudson.ClassicPluginStrategy#createClassJarFromWebInfClasses: Created /var/jenkins/plugins/ldap/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness 2019-05-22 12:43:40.528+0000 [id=33] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin as a dependency: /var/jenkins/plugins/pam-auth.jpi 2019-05-22 12:43:40.562+0000 [id=33] WARNING hudson.ClassicPluginStrategy#createClassJarFromWebInfClasses: Created /var/jenkins/plugins/pam-auth/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness 2019-05-22 12:43:40.568+0000 [id=33] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin as a dependency: /var/jenkins/plugins/javadoc.jpi 2019-05-22 12:43:40.618+0000 [id=33] WARNING hudson.ClassicPluginStrategy#createClassJarFromWebInfClasses: Created /var/jenkins/plugins/javadoc/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness 2019-05-22 12:43:40.706+0000 [id=27] WARNING hudson.ClassicPluginStrategy#createClassJarFromWebInfClasses: Created /var/jenkins/plugins/toolenv-1.1/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness 2019-05-22 12:43:40.881+0000 [id=30] WARNING hudson.ClassicPluginStrategy#createClassJarFromWebInfClasses: Created /var/jenkins/plugins/email-ext-recipients-column-1.0/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness 2019-05-22 12:43:42.073+0000 [id=30] WARNING hudson.ClassicPluginStrategy#createClassJarFromWebInfClasses: Created /var/jenkins/plugins/saferestart-0.3/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness 2019-05-22 12:43:42.101+0000 [id=26] WARNING hudson.ClassicPluginStrategy#createClassJarFromWebInfClasses: Created /var/jenkins/plugins/jquery-ui-1.0.2/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness 2019-05-22 12:43:43.862+0000 [id=28] WARNING hudson.ClassicPluginStrategy#createClassJarFromWebInfClasses: Created /var/jenkins/plugins/job-dsl-1.74/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness 2019-05-22 12:43:44.219+0000 [id=31] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins 2019-05-22 12:43:44.372+0000 [id=28] INFO j.b.a.SecurityProviderInitializer#addSecurityProvider: Initializing Bouncy Castle security provider. 2019-05-22 12:43:44.711+0000 [id=28] INFO j.b.a.SecurityProviderInitializer#addSecurityProvider: Bouncy Castle security provider initialized. 2019-05-22 12:43:44.794+0000 [id=33] SEVERE jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading plugin Pipeline: Groovy v2.68 (workflow-cps) java.io.IOException: Pipeline: Groovy version 2.68 failed to load. - Script Security Plugin version 1.56 is older than required. To fix, install version 1.58 or later. at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:868) at hudson.PluginManager$2$1$1.run(PluginManager.java:544) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1091) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2019-05-22 12:43:44.796+0000 [id=33] SEVERE jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading plugin Pipeline Graph Analysis Plugin v1.10 (pipeline-graph-analysis) java.io.IOException: Pipeline Graph Analysis Plugin version 1.10 failed to load. - Pipeline: Groovy version 2.68 failed to load. Fix this plugin first. at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:868) at hudson.PluginManager$2$1$1.run(PluginManager.java:544) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1091) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2019-05-22 12:43:44.797+0000 [id=33] SEVERE jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading plugin Pipeline: REST API Plugin v2.11 (pipeline-rest-api) java.io.IOException: Pipeline: REST API Plugin version 2.11 failed to load. - Pipeline Graph Analysis Plugin version 1.10 failed to load. Fix this plugin first. at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:868) at hudson.PluginManager$2$1$1.run(PluginManager.java:544) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1091) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
And also other test use case failing with similar errors. For some instances, we test Jenkins locally before we apply any changes to actual instance, part of that we have script which copies over all the hpi files to plugins directory before we start up the jenkins, this is to make sure these new plugins don't have any security issues and also the dependencies look good.
It is not this issue per se, but it is "Bring the bundled version of the Script Security plugin up to date with recent security advisories, in the unlikely case it is indeed installed from the WAR rather than the update center" which was delivered along with it in https://github.com/jenkinsci/jenkins/pull/4000 . It looks like the code triggered the downgrade somehow.
A separate issue would be appreciated
script which copies over all the hpi files to plugins directory
Note that $JENKINS_HOME/plugins/ is expected to contain *.jpi files, never *.hpi which is the extension used in Maven repositories and update center URLs. The Jenkins plugin manager always performs this rename when installing plugins either from an update center or the install-plugin CLI command, and official packaging scripts such as for Docker do the same, but perhaps your unsupported custom script neglected to do that? If this is the issue, then Jenkins core could be made a bit more robust here and either silently tolerate *.hpi in more cases, print a clear warning about improper file names, or try to automatically clean up. That would be a separate issue that could be linked to this one.
jglick Thanks for the light. We didn't know about this thing, so far we are using .hpi files those are downloaded directly from the repo. All of our jenkins instances are currently running on this setup.
is that just rename or do we need to do anything special after downloading those .hpi files from the repo?
The files in the plugins directory are expected to be named ARTIFACT_ID.jpi, so for example script-security.jpi rather than script-security-1.58.hpi (filename in Artifactory) or script-security.hpi (filename in updates.jenkins.io).
I filed a refinement in jenkins #4039 to improve the behavior and make it more apparent what you were doing wrong.
Thank you very much jglick we changed our script to use .jpi instead of .hpi and also removed the version from the file name. It worked!
Thanks again really appreciate your inputs.
There was an ATH regression which it is hoped form-element-path #8 will help correct.
Is it possible to somehow skip or remove this behavior? We do not use any of those detached plugins and they keep being instantiated (in rather ancient versions) at service startup.
We do not use any of those detached plugins
If you are using the official Docker image as a base, you ought to be able to
RUN touch /usr/share/jenkins/ref/plugins/{whatever-id,another-id}.jpi.disabled
Untested, YMMV.
This feels a little bit untenable if I'm understanding everything correctly, at least for those using the Docker images. Like I'll be playing whack-a-mole either by adding these implied dependencies to my list of installed plugins so they get updated (because of security issues) or by disabling them because they aren't actually used but are pulling in ancient versions with tons of security warnings. It also heavily relies on plugin maintainers to be constantly updating the version of Jenkins their plugin relies on to not be continually pulling in implied deps of old Jenkins core versions.
For example, the simple https://plugins.jenkins.io/purge-build-queue-plugin# is referencing a really old Jenkins core version so it's pulling in a ton of implied deps, most of which are really old and some with security vulnerabilities. Whats the best way to handle this other than having that plugin updated to reference a new Jenkins version so less implied deps are pulled in?
Even when disabled, these are still installed and report security issues. As a workaround I am deleting WEB-INF/detached-plugins from jenkins.war, but I believe it should be at least controllable by some property so that one can consciously opt out of that functionality. Apart from that, what's the point of separating functionality into plugins, when the same functionality is injected (in an outdated version) anyway?
> Apart from that, what's the point of separating functionality into plugins, when the same functionality is injected (in an outdated version) anyway?
My understanding is the functionality isn't included anymore in core Jenkins so when those things got peeled out, in order to keep things from breaking, core will pull in these "implied" dependencies to keep other plugins working that may have used them.
I got that. But plugins may have or may have not used them, as there was no possibility to declare. If core will keep reinstantiating them, we will never be able to get rid of this functionality which was unused. For instance, I may be using Green Balls that only changes icon colors, but it incorporates 14 implied plugins. I am pretty sure it does not use LDAP, JDK installer or Windows agents, yet they keep getting installed, because of the time that the plugin was released, they were part of Jenkins. I don't see any point in re-releasing the plugin or upgrading core requirements as the functionality is quite simple and seems complete, so there is no point in artificial release. I would like to get rid of the functionality I don't need though to reduce the bloat, but the core does not want to let it go. I really believe it should be maintained by people maintaining their Docker images. At least there should be an option I could set to say: 'Yes, I know what I'm doing, thank you.'
disabling them because they aren't actually used but are pulling in ancient versions with tons of security warnings
I am not aware of which ancient versions or security warnings you are referring to here. All detached plugins are expected to be updated in tandem with security advisories. If you see differently, please file a bug report (or a patch). CC danielbeck
I don't see any point in re-releasing the plugin or upgrading core requirements
See discussion in JENKINS-28942. My standing proposal would require the plugin to be re-released, with metadata indicating the most recent core version against which it has been successfully tested, but would not require the minimum core version to be changed.
Fun fact: When greenballs was last released in 2015, 1.638 was the current weekly release, not 1.440 (released 2011). So the implied dependencies to…
- external-monitor-job
- ldap
- pam-auth
- mailer
- matrix-auth
- windows-slaves
- antisamy-markup-formatter
- matrix-project
- junit
i.e. roughly 2/3 of all its dependencies, were basically the maintainer's choice.
jglick I am not seeing the detached plugins being updated. At last when using the Docker container. I'm able to replicate with the https://plugins.jenkins.io/purge-build-queue-plugin# which pulls in LDAP 1.0 for example.
Some logs:
jenkins_1 | INFO: Loading a detached plugin as a dependency: /var/jenkins_home/plugins/ldap.jpi jenkins_1 | WARNING: Created /var/jenkins_home/plugins/ldap/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness jenkins_1 | INFO: Took 0ms for Loading plugin LDAP Plugin v1.11 (ldap) by pool-6-thread-24 jenkins_1 | INFO: Took 0ms for Initializing plugin ldap by pool-6-thread-18
The latest version is 1.20 according to my UI.
Er, I'm sorry. The https://plugins.jenkins.io/pam-auth plugin is the one with the security issue.
jenkins_1 | INFO: Loading a detached plugin as a dependency: /var/jenkins_home/plugins/pam-auth.jpi jenkins_1 | WARNING: Created /var/jenkins_home/plugins/pam-auth/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness jenkins_1 | INFO: Took 0ms for Loading plugin PAM Authentication plugin v1.1 (pam-auth) by pool-6-thread-4 jenkins_1 | INFO: Took 0ms for Initializing plugin pam-auth by pool-6-thread-1
Thanks danielbeck. Did you need me to make a ticket or did you already?
danielbeck any update on this? Was an issue ever made that I can track?
To be landed in 2.178