-
Bug
-
Resolution: Fixed
-
Minor
-
None
-
Windows Server 2016
Java 1.8.0_221-b.11
Jenkins 2.202
IIS Reverse proxy configured per https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+IIS
CSRF Protection w/ Default Crumb Issuer
-
-
2.223
Auto-Upgrade Jenkins, select Restart Jenkins when complete...
Manual restart via https://<jenkins>/restart...
Logs (see below) show shutdown and restart occurs. However, during startup, an infinite cycle of exceptions (Jenkins.instance is missing) see below. Must manually restart windows service to successfully restart jenkins.
2019-10-30 19:58:18.332+0000 [id=154905] INFO h.model.UpdateCenter$DownloadJob#run: Starting the installation of jenkins.war on behalf of xxxxx 2019-10-30 19:58:18.332+0000 [id=154905] INFO h.model.UpdateCenter$DownloadJob#run: Starting the installation of jenkins.war on behalf of xxxxx 2019-10-30 19:58:18.532+0000 [id=154905] INFO h.m.UpdateCenter$UpdateCenterConfiguration#download: Downloading jenkins.war 2019-10-30 19:58:23.497+0000 [id=154905] INFO h.model.UpdateCenter$DownloadJob#run: Installation successful: jenkins.war 2019-10-30 20:01:07.466+0000 [id=151608] INFO hudson.model.UpdateCenter#doSafeRestart: Scheduling Jenkins reboot 2019-10-30 20:01:07.478+0000 [id=155008] INFO jenkins.model.Jenkins$23#run: Restart in 10 seconds 2019-10-30 20:01:17.479+0000 [id=155008] INFO jenkins.model.Jenkins$23#run: Restarting VM as requested by xxxxx 2019-10-30 20:01:17.481+0000 [id=155008] INFO jenkins.model.Jenkins#cleanUp: Stopping Jenkins 2019-10-30 20:01:17.529+0000 [id=155008] INFO jenkins.model.Jenkins$18#onAttained: Started termination 2019-10-30 20:01:17.542+0000 [id=155008] INFO jenkins.model.Jenkins$18#onAttained: Completed termination 2019-10-30 20:01:17.542+0000 [id=155008] INFO jenkins.model.Jenkins#_cleanUpDisconnectComputers: Starting node disconnection 2019-10-30 20:01:17.550+0000 [id=155008] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: safe-restart thread for xxxx terminated: java.nio.channels.ClosedChannelException 2019-10-30 20:01:17.557+0000 [id=155008] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: safe-restart thread for xxxx terminated: java.nio.channels.ClosedChannelException 2019-10-30 20:01:18.569+0000 [id=155008] INFO jenkins.model.Jenkins#_cleanUpShutdownPluginManager: Stopping plugin manager 2019-10-30 20:01:18.596+0000 [id=155008] INFO jenkins.model.Jenkins#_cleanUpPersistQueue: Persisting build queue 2019-10-30 20:01:18.602+0000 [id=155008] INFO jenkins.model.Jenkins#_cleanUpAwaitDisconnects: Waiting for node disconnection completion 2019-10-30 20:01:18.602+0000 [id=155008] INFO jenkins.model.Jenkins#cleanUp: Jenkins stopped 2019-10-30 20:01:22.820+0000 [id=151613] WARNING h.s.HttpSessionContextIntegrationFilter2#hasInvalidSessionSeed: Encountered IllegalStateException trying to get a user. System init may not have completed yet. Invalidating user session. 2019-10-30 20:01:27.573+0000 [id=152096] WARNING o.e.jetty.server.HttpChannel#handleException: /tcpSlaveAgentListener/java.lang.IllegalStateException: Jenkins.instance is missing. Read the documentation of Jenkins.getInstanceOrNull to see what you are doing wrong. at jenkins.model.Jenkins.get(Jenkins.java:777) at org.jenkinsci.plugins.matrixauth.AuthorizationContainer.hasPermission(AuthorizationContainer.java:169) at hudson.security.GlobalMatrixAuthorizationStrategy$AclImpl.hasPermission(GlobalMatrixAuthorizationStrategy.java:127) at hudson.security.SidACL._hasPermission(SidACL.java:70) at hudson.security.SidACL.hasPermission(SidACL.java:52) at hudson.security.ACL.checkPermission(ACL.java:72) at hudson.security.AccessControlled.checkPermission(AccessControlled.java:47) at jenkins.model.Jenkins.getTarget(Jenkins.java:4763) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:703) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:676) at hudson.init.impl.InstallUncaughtExceptionHandler.lambda$init$0(InstallUncaughtExceptionHandler.java:36) at org.kohsuke.stapler.compression.CompressionFilter.reportException(CompressionFilter.java:77) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:58) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:505) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804) at java.lang.Thread.run(Unknown Source)
Thank you csclutgen for the report. Due to the stack trace, the symptoms are different from
JENKINS-55070. Here the problem is revealed in AuthorizationContainer.java#L169.After a deeper investigation, the problem seems to be after the nullification of theInstance in Jenkins#L3400. There is a moment between the instance being null and the real restart (fraction of second) and during that time, the instance cannot be retrieved.
As the HudsonIsRestarting is only used as the last filter, it has no chance to avoid other computation yet. The problem is when a filter that is applied earlier, requires to have access to Jenkins.get().
I see two opportunities to correct the problem:
To elaborate on option 2, current flow of filters to servlet:
We could either bypass completely the filters (HudsonFilter + CrumbFilter + PluginServletFilter) and reach directly the app (Stapler), or disabling the configured filters. If we bypass the rest of the filter, we will have troubles to deliver the asset bounded to the restart page, but this could be done "inline" and avoid any other parallel call.
That could solve the
> WARNING h.s.HttpSessionContextIntegrationFilter2#hasInvalidSessionSeed: Encountered IllegalStateException trying to get a user. System init may not have completed yet. Invalidating user session.
But I was not able to reproduce the second warning, as I was not using the WindowsService. I imagine it's somewhat linked.