-
Bug
-
Resolution: Fixed
-
Critical
-
Jenkins 2.103 running on Debian 8 with OpenJDK 1.8.0_131 and Tomcat8
-
Powered by SuggestiMate
I followed my normal procedure in updating our Jenkins builder:
- Update all Jenkins plugins first.
- Update jenkins core (using .war file)
I got the error message about JEP-200 and XStream, so I browsed to the wiki page. It had a note in there about -Dhudson.remoting.ClassFilter=, with a comma-separated list of class names from the log file. I got the list of classes:
# grep -ri rejecting /var/log/tomcat8/catalina.out | awk -F ' ' '{print $2}' | sort | uniq hudson.model.Cause$UserIdCause hudson.model.Hudson$CloudList hudson.model.MyViewsProperty hudson.model.PaneStatusProperties hudson.model.Queue$State hudson.model.UpdateSite hudson.model.View$PropertyList hudson.node_monitors.ArchitectureMonitor hudson.node_monitors.ClockMonitor hudson.node_monitors.DiskSpaceMonitor hudson.node_monitors.ResponseTimeMonitor hudson.node_monitors.SwapSpaceMonitor hudson.node_monitors.TemporarySpaceMonitor hudson.remoting.RemoteInvocationHandler$RPCRequest hudson.scm.SCMRevisionState$None hudson.search.UserSearchProperty hudson.slaves.JNLPLauncher hudson.slaves.RetentionStrategy$2 hudson.tasks.LogRotator hudson.tasks.Shell$DescriptorImpl hudson.triggers.SCMTrigger$BuildAction hudson.triggers.SCMTrigger$DescriptorImpl hudson.triggers.SCMTrigger$SCMTriggerCause hudson.util.CopyOnWriteMap$Hash jenkins.model.BuildDiscarderProperty jenkins.model.ProjectNamingStrategy$DefaultProjectNamingStrategy jenkins.security.ApiTokenProperty jenkins.security.LastGrantedAuthoritiesProperty jenkins.slaves.RemotingWorkDirSettings
So, I added those as a comma-separated list to -Dhudson.remoting.ClassFilter= and restarted Tomcat. Jenkins came back (authentication worked, but no build information is available, and slaves cannot connect), but I am now seeing a message about "You have data stored in an older format and/or unreadable data.". I am a bit afraid I will lose my build history and other metadata if I click on "Discard Unreadable Data". Is that a "safe" operation for my builds metadata?
Also, why do I need to add so many exclusions to the hudson.remoting.ClassFilter, some of which seem to be internal to jenkins/hudson? Shouldn't that "just work"? Did I do something wrong in the upgrade?
- relates to
-
JENKINS-49543 Refusing to marshal org.jenkinsci.main.modules.cli.auth.ssh.UserPropertyImpl on Old Apache TomCat 8.x versions
-
- Resolved
-
- links to
[JENKINS-49147] JEP-200 location-based whitelisting broken in obsolete versions of Tomcat
Shouldn't that "just work"?
Yes of course. Nothing in Jenkins core needs to be whitelisted.
I suspect ClassFilterImpl.isLocationWhitelisted is failing to grasp that your Jenkins core is really Jenkins core, for some reason related to an unusual packaging. Perhaps because you are running on Tomcat—almost everyone uses the built-in Winstone instead, and I do not know of any automated tests which check compatibility with other containers.
You will need to create a FINE logger on jenkins.security.ClassFilterImpl to find out. Normally you will see a single message early on to the effect of
file:/usr/share/jenkins/WEB-INF/lib/jenkins-core-2.103.jar seems to be the location of Jenkins core, OK
Thank you for the help jglick, really appreciate it. I tried creating the FINE logger, but when I went to save, it threw this in the UI:
java.lang.UnsupportedOperationException: Refusing to marshal hudson.logging.LogRecorder for security reasons; see https://jenkins.io/redirect/class-filter/ at hudson.util.XStream2$BlacklistedTypesConverter.marshal(XStream2.java:530) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015) at com.thoughtworks.xstream.XStream.toXML(XStream.java:988) at hudson.XmlFile.write(XmlFile.java:194) Caused: java.io.IOException at hudson.XmlFile.write(XmlFile.java:201) at hudson.logging.LogRecorder.save(LogRecorder.java:328) at hudson.logging.LogRecorder.doConfigSubmit(LogRecorder.java:303) at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343) at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:77) at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:129) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845) at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845) at org.kohsuke.stapler.MetaClass$3.doDispatch(MetaClass.java:209) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:725) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:99) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:618) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:537) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1081) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:658) at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:222) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1570) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1527) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)
It looks like even though the UI threw the error, it did save the log recorder settings. Went back in to it, and this was in there:
Jan 24, 2018 1:26:09 PM FINE jenkins.security.ClassFilterImpl lambda$isLocationWhitelisted$2 jar:file:/var/lib/tomcat8/webapps/ROOT/WEB-INF/lib/jenkins-core-2.103.jar!/hudson/logging/LogRecorder$Target.class is not recognized; rejecting Jan 24, 2018 1:26:09 PM WARNING jenkins.security.ClassFilterImpl lambda$isBlacklisted$1 hudson.logging.LogRecorder$Target in jar:file:/var/lib/tomcat8/webapps/ROOT/WEB-INF/lib/jenkins-core-2.103.jar!/hudson/logging/LogRecorder$Target.class might be dangerous, so rejecting; see https://jenkins.io/redirect/class-filter/
Seems to be a bug in Tomcat. A valid code source would be jar:file:/var/lib/tomcat8/webapps/ROOT/WEB-INF/lib/jenkins-core-2.103.jar!/ or file:/var/lib/tomcat8/webapps/ROOT/WEB-INF/lib/jenkins-core-2.103.jar or even (stretching the limits of logic) jar:file:/var/lib/tomcat8/webapps/ROOT/WEB-INF/lib/jenkins-core-2.103.jar. In no case should it include the resource path within the code source.
At any rate, sounds easy enough to work around, probably using hudson.remoting.Which, assuming I can get a working test environment to reproduce. I recommend you just use the built-in Winstone, though.
Hmm, I cannot reproduce any problem using Tomcat 8.5.27 via the tomcat:latest Docker image. I get for example
permitting hudson.logging.LogRecorder due to its location in file:/usr/local/tomcat/webapps/jenkins/WEB-INF/lib/jenkins-core-2.104-SNAPSHOT.jar
From your log file, seems you are using 8.0.14, so I guess I will try to dig up something older.
Well I put together https://github.com/jglick/tomcat-jenkins but it is working fine for me on any Tomcat version I could find—7, 8.0.x, 8.5.x, 9.
The fix is probably easy enough but I would not like to make a change I cannot test. Is there some way to reproduce this from scratch?
I am able to reproduce this by installing a fresh copy of Tomcat8 on Debian 8 Jessie (which is 8.0.15), and creating a simple config.xml in /home/jenkins/config.xml with the content <hudson/>. It consistently throws the same errors about not recognizing jenkins-core-2.103.jar.
If I remove the config.xml and restart Tomcat8, everything works as expected.
However, no matter what Docker version of Tomcat8 I use, I cannot repro (if I go all the way down to 8.0.15, Jenkins refuses to start at all, and complains about AWT being misconfigured and needing to enable -Djava.awt.headless=true).
At this point, I guess I'll have to switch to running Winstone directly, instead of using Tomcat8, unless you have any other ideas?
Is it safe to point Winstone at my existing /home/jenkins directory (with my config.xml and other metadata)? Will it pick up the previous settings from Tomcat8 (which will be stopped/disabled)?
no matter what Docker version of Tomcat8 I use, I cannot repro
Odd. I will try to reproduce using a non-Docker-based installation, and/or using the Jessie image to install Tomcat from update sources. Again, if I can reproduce, the fix is probably straightforward and might help users of even more exotic containers.
Is it safe to point Winstone at my existing /home/jenkins directory (with my config.xml and other metadata)? Will it pick up the previous settings from Tomcat8
It should. Probably goes without saying, but—create a backup first.
using the Jessie image to install Tomcat from update sources
Finally able to reproduce it this way. Perhaps this was just a bug fixed in the 8.0.x line; 8.0.22, 8.0.38, and 8.0.40 mention potentially related fixes in the changelog. So I wonder why Jessie (even jessie-backports) is shipping a version obsolete since late 2014.
jglick, I migrated our deployment to native Winstone, and everything seems to have come back (slaves connected, jobs show up along with metadata, etc.).
However, when I go to trigger a job, I am getting this now in the job log console:
java.lang.ClassCastException: java.util.HashMap cannot be cast to hudson.scm.SCMRevisionState at org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState.get(MultiSCMRevisionState.java:56) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:110) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:274) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Finished: FAILURE
Is there some way to fix this?
kmott I have not seen that and it looks unrelated. Best to file separately with steps to reproduce, or send to the users’ list.
If I trigger the build again, it goes away, so I'm guessing some transient turd left by Tomcat. Thanks again for all of the help Jesse!
Code changed in jenkins
User: Jesse Glick
Path:
core/src/main/java/jenkins/security/ClassFilterImpl.java
http://jenkins-ci.org/commit/jenkins/655be64a1753a3077a81fc0c34573bca74dcf5a0
Log:
JENKINS-49147 Tolerate unusual CodeSource.location format from old versions of Tomcat.
Code changed in jenkins
User: Oleg Nenashev
Path:
core/src/main/java/jenkins/security/ClassFilterImpl.java
http://jenkins-ci.org/commit/jenkins/cd875f0e5aa8191e414a8b9940b02137d8ffb2f4
Log:
Merge pull request #3264 from jglick/Tomcat-JENKINS-49147
JENKINS-49147 Tolerate unusual CodeSource.location format from old versions of Tomcat
Compare: https://github.com/jenkinsci/jenkins/compare/dbc75125c565...cd875f0e5aa8
This is in 2.104 so realistic LTS baselines will include this (and we're aiming for 2.107).
To clarify a bit further, I did take a snapshot of my Jenkins VM after the upgrade to 2.103 and my hudson.remoting.ClassFilter changes, but before I did "Discard Old Data". I did click on "Discard Old Data", and that doesn't seem to have changed anything. My slave machines are still all disconnected.