Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59992

"Jenkins.instance is missing" error when restarting Jenkins

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • core
    • None
    • 2.223

      Auto-Upgrade Jenkins, select Restart Jenkins when complete...

      Manual restart via https://<jenkins>/restart...

       

      Logs (see below) show shutdown and restart occurs. However, during startup, an infinite cycle of exceptions (Jenkins.instance is missing) see below. Must manually restart windows service to successfully restart jenkins.

       

      2019-10-30 19:58:18.332+0000 [id=154905] INFO h.model.UpdateCenter$DownloadJob#run: Starting the installation of jenkins.war on behalf of xxxxx
      2019-10-30 19:58:18.332+0000 [id=154905] INFO h.model.UpdateCenter$DownloadJob#run: Starting the installation of jenkins.war on behalf of xxxxx
      2019-10-30 19:58:18.532+0000 [id=154905] INFO h.m.UpdateCenter$UpdateCenterConfiguration#download: Downloading jenkins.war
      2019-10-30 19:58:23.497+0000 [id=154905] INFO h.model.UpdateCenter$DownloadJob#run: Installation successful: jenkins.war
      2019-10-30 20:01:07.466+0000 [id=151608] INFO hudson.model.UpdateCenter#doSafeRestart: Scheduling Jenkins reboot
      2019-10-30 20:01:07.478+0000 [id=155008] INFO jenkins.model.Jenkins$23#run: Restart in 10 seconds
      2019-10-30 20:01:17.479+0000 [id=155008] INFO jenkins.model.Jenkins$23#run: Restarting VM as requested by xxxxx
      2019-10-30 20:01:17.481+0000 [id=155008] INFO jenkins.model.Jenkins#cleanUp: Stopping Jenkins
      2019-10-30 20:01:17.529+0000 [id=155008] INFO jenkins.model.Jenkins$18#onAttained: Started termination
      2019-10-30 20:01:17.542+0000 [id=155008] INFO jenkins.model.Jenkins$18#onAttained: Completed termination
      2019-10-30 20:01:17.542+0000 [id=155008] INFO jenkins.model.Jenkins#_cleanUpDisconnectComputers: Starting node disconnection
      2019-10-30 20:01:17.550+0000 [id=155008] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: safe-restart thread for xxxx terminated: java.nio.channels.ClosedChannelException
      2019-10-30 20:01:17.557+0000 [id=155008] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: safe-restart thread for xxxx terminated: java.nio.channels.ClosedChannelException
      2019-10-30 20:01:18.569+0000 [id=155008] INFO jenkins.model.Jenkins#_cleanUpShutdownPluginManager: Stopping plugin manager
      2019-10-30 20:01:18.596+0000 [id=155008] INFO jenkins.model.Jenkins#_cleanUpPersistQueue: Persisting build queue
      2019-10-30 20:01:18.602+0000 [id=155008] INFO jenkins.model.Jenkins#_cleanUpAwaitDisconnects: Waiting for node disconnection completion
      2019-10-30 20:01:18.602+0000 [id=155008] INFO jenkins.model.Jenkins#cleanUp: Jenkins stopped
      2019-10-30 20:01:22.820+0000 [id=151613] WARNING h.s.HttpSessionContextIntegrationFilter2#hasInvalidSessionSeed: Encountered IllegalStateException trying to get a user. System init may not have completed yet. Invalidating user session.
      2019-10-30 20:01:27.573+0000 [id=152096] WARNING o.e.jetty.server.HttpChannel#handleException: /tcpSlaveAgentListener/java.lang.IllegalStateException: Jenkins.instance is missing. Read the documentation of Jenkins.getInstanceOrNull to see what you are doing wrong.
      	at jenkins.model.Jenkins.get(Jenkins.java:777)
      	at org.jenkinsci.plugins.matrixauth.AuthorizationContainer.hasPermission(AuthorizationContainer.java:169)
      	at hudson.security.GlobalMatrixAuthorizationStrategy$AclImpl.hasPermission(GlobalMatrixAuthorizationStrategy.java:127)
      	at hudson.security.SidACL._hasPermission(SidACL.java:70)
      	at hudson.security.SidACL.hasPermission(SidACL.java:52)
      	at hudson.security.ACL.checkPermission(ACL.java:72)
      	at hudson.security.AccessControlled.checkPermission(AccessControlled.java:47)
      	at jenkins.model.Jenkins.getTarget(Jenkins.java:4763)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:703)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:676)
      	at hudson.init.impl.InstallUncaughtExceptionHandler.lambda$init$0(InstallUncaughtExceptionHandler.java:36)
      	at org.kohsuke.stapler.compression.CompressionFilter.reportException(CompressionFilter.java:77)
      	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:58)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
      	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
      	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      	at org.eclipse.jetty.server.Server.handle(Server.java:505)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
      	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
      	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
      	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804)
      	at java.lang.Thread.run(Unknown Source)
      

       

       

          [JENKINS-59992] "Jenkins.instance is missing" error when restarting Jenkins

          Wadeck Follonier added a comment - - edited

          Thank you csclutgen for the report. Due to the stack trace, the symptoms are different from JENKINS-55070. Here the problem is revealed in AuthorizationContainer.java#L169.

          After a deeper investigation, the problem seems to be after the nullification of theInstance in Jenkins#L3400. There is a moment between the instance being null and the real restart (fraction of second) and during that time, the instance cannot be retrieved.

          As the HudsonIsRestarting is only used as the last filter, it has no chance to avoid other computation yet. The problem is when a filter that is applied earlier, requires to have access to Jenkins.get().

          I see two opportunities to correct the problem:

          1. Ensure every filter or class being potentially used by a filter, to not use Jenkins.get(), but instead Jenkins.getInstanceOrNull() and act correctly in case it's null. This could require a lots of work and hard time explaining the situation to people.
          2. Prevent the regular filters to be applied when the HudsonIsRestarting is the "app" in the servletContext.

          To elaborate on option 2, current flow of filters to servlet:

          • DiagnosticThreadNameFilter
          • CharacterEncodingFilter
          • CompressionFilter
          • HudsonFilter
            • ChainedServletFilter
              • HttpSessionContextIntegrationFilter2
              • BasicHeaderProcessor
              • AuthenticationProcessingFilter2
              • RememberMeProcessingFilter
              • AnonymousProcessingFilter
              • ExceptionTranslationFilter
              • UnwrapSecurityExceptionFilter
          • CrumbFilter
          • PluginServletFilter
            • UserLanguages
            • [...] << PluginServletFilter.addFilter (mainly for legacy plugins)
          • Stapler as the main Servlet

          We could either bypass completely the filters (HudsonFilter + CrumbFilter + PluginServletFilter) and reach directly the app (Stapler), or disabling the configured filters. If we bypass the rest of the filter, we will have troubles to deliver the asset bounded to the restart page, but this could be done "inline" and avoid any other parallel call.

          That could solve the
          > WARNING h.s.HttpSessionContextIntegrationFilter2#hasInvalidSessionSeed: Encountered IllegalStateException trying to get a user. System init may not have completed yet. Invalidating user session.

          But I was not able to reproduce the second warning, as I was not using the WindowsService. I imagine it's somewhat linked.

          Wadeck Follonier added a comment - - edited Thank you csclutgen for the report. Due to the stack trace, the symptoms are different from JENKINS-55070 . Here the problem is revealed in AuthorizationContainer.java#L169 . After a deeper investigation, the problem seems to be after the nullification of theInstance in Jenkins#L3400 . There is a moment between the instance being null and the real restart (fraction of second) and during that time, the instance cannot be retrieved. As the HudsonIsRestarting is only used as the last filter, it has no chance to avoid other computation yet. The problem is when a filter that is applied earlier, requires to have access to Jenkins.get() . I see two opportunities to correct the problem: Ensure every filter or class being potentially used by a filter, to not use Jenkins.get() , but instead Jenkins.getInstanceOrNull() and act correctly in case it's null. This could require a lots of work and hard time explaining the situation to people. Prevent the regular filters to be applied when the HudsonIsRestarting is the "app" in the servletContext. To elaborate on option 2, current flow of filters to servlet: DiagnosticThreadNameFilter CharacterEncodingFilter CompressionFilter HudsonFilter ChainedServletFilter HttpSessionContextIntegrationFilter2 BasicHeaderProcessor AuthenticationProcessingFilter2 RememberMeProcessingFilter AnonymousProcessingFilter ExceptionTranslationFilter UnwrapSecurityExceptionFilter CrumbFilter PluginServletFilter UserLanguages [...] << PluginServletFilter.addFilter (mainly for legacy plugins) Stapler as the main Servlet We could either bypass completely the filters (HudsonFilter + CrumbFilter + PluginServletFilter) and reach directly the app (Stapler), or disabling the configured filters. If we bypass the rest of the filter, we will have troubles to deliver the asset bounded to the restart page, but this could be done "inline" and avoid any other parallel call. That could solve the > WARNING h.s.HttpSessionContextIntegrationFilter2#hasInvalidSessionSeed: Encountered IllegalStateException trying to get a user. System init may not have completed yet. Invalidating user session. But I was not able to reproduce the second warning, as I was not using the WindowsService. I imagine it's somewhat linked.

          Wadeck Follonier added a comment - Symptom correction proposal: https://github.com/jenkinsci/jenkins/pull/4525

          Symptoms resolved in 2.223

          Wadeck Follonier added a comment - Symptoms resolved in 2.223

            wfollonier Wadeck Follonier
            csclutgen Craig Lutgen
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: