-
Bug
-
Resolution: Unresolved
-
Major
-
None
Since a couple of versions we are having sometimes an issue with the startup of Jenkins.
Early in the startup process we see the following error in the catalina.out file:
03-Mar-2024 00:13:25.684 SEVERE [main] org.apache.catalina.session.StandardManager.startInternal Exception loading sessions from persistent storage java.io.EOFException at java.base/java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:3215) at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1643) at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:499) at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:457) at org.apache.catalina.session.StandardSession.doReadObject(StandardSession.java:1571) at org.apache.catalina.session.StandardSession.readObjectData(StandardSession.java:1050) at org.apache.catalina.session.StandardManager.doLoad(StandardManager.java:219) at org.apache.catalina.session.StandardManager.load(StandardManager.java:162) at org.apache.catalina.session.StandardManager.startInternal(StandardManager.java:356) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5247) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:726) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:698) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:696) at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1185) at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1933) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1095) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:477) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1618) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:319) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123) at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423) at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:946) at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1396) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1386) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:919) at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:265) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardService.startInternal(StandardService.java:432) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.startup.Catalina.start(Catalina.java:772) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:476) 03-Mar-2024 00:13:25.719 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal One or more Filters failed to start. Full details will be found in the appropriate container log file 03-Mar-2024 00:13:25.719 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal Context [] startup failed due to previous errors 03-Mar-2024 00:13:25.806 INFO [main] hudson.lifecycle.Lifecycle.onStatusUpdate Stopping Jenkins
Right after that in the log of Jenkins we see the following error:
03-Mar-2024 00:13:25.717 SEVERE [main] org.apache.catalina.core.StandardContext.filterStart Exception starting filter [authentication-filter]
java.lang.IllegalArgumentException: key cannot be empty or null
at org.springframework.util.Assert.hasLength(Assert.java:243)
at org.springframework.security.web.authentication.rememberme.AbstractRememberMeServices.<init>(AbstractRememberMeServices.java:102)
at hudson.security.TokenBasedRememberMeServices2.<init>(TokenBasedRememberMeServices2.java:95)
at hudson.security.SecurityRealm$SecurityComponents.createRememberMeService(SecurityRealm.java:828)
at hudson.security.SecurityRealm$SecurityComponents.<init>(SecurityRealm.java:794)
at hudson.security.SecurityRealm$None.createSecurityComponents(SecurityRealm.java:676)
at hudson.security.SecurityRealm.getSecurityComponents(SecurityRealm.java:556)
at hudson.security.HudsonFilter.reset(HudsonFilter.java:138)
at hudson.security.HudsonFilter.init(HudsonFilter.java:112)
at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:272)
at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:253)
at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:102)
at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4613)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5256)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:726)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:698)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:696)
at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1185)
at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1933)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1095)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:477)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1618)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:319)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:946)
at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1396)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1386)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:919)
at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:265)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardService.startInternal(StandardService.java:432)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.startup.Catalina.start(Catalina.java:772)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:476)
Could the startup fail due to the session error ? Or is this some race condition in filters/services being loaded ?
In the past when this happened we tried to restart tomcat but we had to try multiple times before Jenkins wanted to start.
The solution we found was to touch the web.xml file to trigger a reload of the jenkins war file.
Current version of Jenkins is 2.414 and the operating system is RHEL 7 and RHEL 8.