Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61990

StackOverflowError on boot related to QueueItemAuthenticatorConfiguration

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • Jenkins version 2.222.1,2.249.1
      matrix-auth 2.5,2.6.2
      matrix-project 1.14,1.17

      After latest container (LTS) and plugin updates, booting the jenkins results in huge stacktraces caused by a stackoverflow error. The result is that the Jenkins seems available, the queue is filled but items cannot be worked on, because a class has not been properly loaded on boot:

      java.lang.IllegalStateException: The class jenkins.security.QueueItemAuthenticatorConfiguration was not found, potentially not yet loadedjava.lang.IllegalStateException: The class jenkins.security.QueueItemAuthenticatorConfiguration was not found, potentially not yet loaded at hudson.ExtensionList.getInstance(ExtensionList.java:166) at jenkins.security.QueueItemAuthenticatorConfiguration.get(QueueItemAuthenticatorConfiguration.java:61) at jenkins.security.QueueItemAuthenticatorConfiguration$ProviderImpl.getAuthenticators(QueueItemAuthenticatorConfiguration.java:70) at jenkins.security.QueueItemAuthenticatorProvider$IteratorImpl.hasNext(QueueItemAuthenticatorProvider.java:44) at hudson.model.Queue$Item.authenticate(Queue.java:2332) 

      The relevant source of evil is the following stacktrace which I will attach in full as a file.

      To make this pop up in searches I paste an excerpt of the stack trace here.

      at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callReadResolve(SerializationMethodInvoker.java:66)
      	at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:268)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      	at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:390)
      	at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:328)
      	at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:267)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      	at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
      	at hudson.util.CopyOnWriteList$ConverterImpl.unmarshal(CopyOnWriteList.java:197)
      	at hudson.util.DescribableList$ConverterImpl.unmarshal(DescribableList.java:275)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      	at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:390)
      	at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:328)
      	at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:267)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
      	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
      	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
      	at hudson.util.XStream2.unmarshal(XStream2.java:161)
      	at hudson.util.XStream2.unmarshal(XStream2.java:132)
      	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
      	at hudson.XmlFile.unmarshal(XmlFile.java:180)
      	at hudson.XmlFile.unmarshal(XmlFile.java:163)
      	at hudson.model.Descriptor.load(Descriptor.java:914)
      	at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at hudson.ExtensionFinder$GuiceFinder$SezpozModule.onProvision(ExtensionFinder.java:601)
      	at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126)
      	at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68)
      	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:87)
      	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
      	at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
      	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
      	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
      	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
      	at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:440)
      	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
      	at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
      	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
      	at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
      	at hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:402)
      	at hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:393)
      	at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:344)
      	at hudson.ExtensionList.load(ExtensionList.java:381)
      	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:317)
      	at hudson.ExtensionList.getComponents(ExtensionList.java:183)
      	at hudson.DescriptorExtensionList.load(DescriptorExtensionList.java:194)
      	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:317)
      	at hudson.ExtensionList.iterator(ExtensionList.java:172)
      	at hudson.ExtensionList.getInstance(ExtensionList.java:162)
      	at jenkins.security.QueueItemAuthenticatorConfiguration.get(QueueItemAuthenticatorConfiguration.java:61)
      	at org.jenkinsci.plugins.authorizeproject.ProjectQueueItemAuthenticator.getConfigured(ProjectQueueItemAuthenticator.java:178)
      	at org.jenkinsci.plugins.authorizeproject.ProjectQueueItemAuthenticator.isConfigured(ProjectQueueItemAuthenticator.java:190)
      	at org.jenkinsci.plugins.authorizeproject.AuthorizeProjectStrategy.checkUnsecuredConfiguration(AuthorizeProjectStrategy.java:186)
      	at org.jenkinsci.plugins.authorizeproject.AuthorizeProjectStrategy.readResolve(AuthorizeProjectStrategy.java:176)
      	at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callReadResolve(SerializationMethodInvoker.java:66)
      	at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:268)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      	at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:390)
      	at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:328)
      	at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:267)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      	at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
      	at hudson.util.CopyOnWriteList$ConverterImpl.unmarshal(CopyOnWriteList.java:197)
      	at hudson.util.DescribableList$ConverterImpl.unmarshal(DescribableList.java:275)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      	at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:390)
      	at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:328)
      	at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:267)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
      	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
      	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
      	at hudson.util.XStream2.unmarshal(XStream2.java:161)
      	at hudson.util.XStream2.unmarshal(XStream2.java:132)
      	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
      	at hudson.XmlFile.unmarshal(XmlFile.java:180)
      	at hudson.XmlFile.unmarshal(XmlFile.java:163)
      	at hudson.model.Descriptor.load(Descriptor.java:914)
      	at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at hudson.ExtensionFinder$GuiceFinder$SezpozModule.onProvision(ExtensionFinder.java:601)
      	at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126)
      	at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68)
      	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:87)
      	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
      	at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
      	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
      	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
      	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
      	at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:440)
      	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
      	at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
      	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
      	at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
      	at hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:402)
      	at hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:393)
      	at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:344)
      	at hudson.ExtensionList.load(ExtensionList.java:381)
      	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:317)
      	at hudson.ExtensionList.getComponents(ExtensionList.java:183)
      	at hudson.DescriptorExtensionList.load(DescriptorExtensionList.java:194)
      	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:317)
      	at hudson.ExtensionList.iterator(ExtensionList.java:172)
      	at hudson.ExtensionList.getInstance(ExtensionList.java:162)
      	at jenkins.security.QueueItemAuthenticatorConfiguration.get(QueueItemAuthenticatorConfiguration.java:61)
      	at org.jenkinsci.plugins.authorizeproject.ProjectQueueItemAuthenticator.getConfigured(ProjectQueueItemAuthenticator.java:178)
      	at org.jenkinsci.plugins.authorizeproject.ProjectQueueItemAuthenticator.isConfigured(ProjectQueueItemAuthenticator.java:190)
      	at org.jenkinsci.plugins.authorizeproject.AuthorizeProjectStrategy.checkUnsecuredConfiguration(AuthorizeProjectStrategy.java:186)
      	at org.jenkinsci.plugins.authorizeproject.AuthorizeProjectStrategy.readResolve(AuthorizeProjectStrategy.java:176)
      	at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callReadResolve(SerializationMethodInvoker.java:66)
      	at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:268)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      	at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:390)
      	at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:328)
      	at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:267)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      	at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
      	at hudson.util.CopyOnWriteList$ConverterImpl.unmarshal(CopyOnWriteList.java:197)
      	at hudson.util.DescribableList$ConverterImpl.unmarshal(DescribableList.java:275)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      	at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:390)
      	at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:328)
      	at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:267)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      	at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
      	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
      	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
      	at hudson.util.XStream2.unmarshal(XStream2.java:161)
      	at hudson.util.XStream2.unmarshal(XStream2.java:132)
      	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
      	at hudson.XmlFile.unmarshal(XmlFile.java:180)
      	at hudson.XmlFile.unmarshal(XmlFile.java:163)
      	at hudson.model.Descriptor.load(Descriptor.java:914)
      	at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at hudson.ExtensionFinder$GuiceFinder$SezpozModule.onProvision(ExtensionFinder.java:601)
      	at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126)
      	at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68)
      	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:87)
      	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
      	at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
      	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
      	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
      	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
      	at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:440)
      	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
      	at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
      	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
      	at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
      	at hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:402)
      	at hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:393)
      	at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:344)
      	at hudson.ExtensionList.load(ExtensionList.java:381)
      	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:317)
      	at hudson.ExtensionList.getComponents(ExtensionList.java:183)
      

      Important note: I "fixed" my jenkins by manually emptying the queue and moving my init.groovy away. I.e., only this snippet (which has been working until now) is now causing this behavior:
      …
      After a reboot without this hook, everything works as expected. However, I really need this hook as a workaround for another bug in a different plugin. And as I said - this has been working for months before.

          [JENKINS-61990] StackOverflowError on boot related to QueueItemAuthenticatorConfiguration

          Tom Lorentsen added a comment -

          Had this problem on when upgrading to 2.319.1. Rolled back to snapshot twice! The workaround didn't appear to work for me

          Tom Lorentsen added a comment - Had this problem on when upgrading to 2.319.1. Rolled back to snapshot twice! The workaround didn't appear to work for me

          Ronny Perinke added a comment -

          not entirely sure it is helpful other than to say that it is nearing November and this is still not fixed. In any case, we are on Jenkins 2.263.4

          Now its May 2022 with Jenkins 2.332 and Access Control for Builds still mentions Authorize Project Plugin as the one to use. Or are there meanwhile other plugins that can do this?

          Ronny Perinke added a comment - not entirely sure it is helpful other than to say that it is nearing November and this is still not fixed. In any case, we are on Jenkins 2.263.4 Now its May 2022 with Jenkins 2.332 and Access Control for Builds still mentions Authorize Project Plugin as the one to use. Or are there meanwhile other plugins that can do this?

          Jesse Glick added a comment -

          https://github.com/jenkinsci/jenkins/pull/6222 and no, there is no real replacement.

          Jesse Glick added a comment - https://github.com/jenkinsci/jenkins/pull/6222 and no, there is no real replacement.

          Ana Trakhtman added a comment -

          Encountered this issue today and solved by removing queue.xml from /var/lib/jenkins.

          Ana Trakhtman added a comment - Encountered this issue today and solved by removing queue.xml from /var/lib/jenkins.

          Liam Baker added a comment - - edited

          Saw the same on Jenkins 2.452.1 We previously thought it was due to too many items on the queue (300+).
          However this time we stopped all jobs, removed all items from the queue, set Jenkins in "prepare for shutdown".
          Then rebooted the Jenkins Controller (reboot to install OS updates).

          On reboot Jenkins UI is throwing a 500 error with the following trace.

          java.lang.IllegalStateException: Jenkins.instance is missing. Read the documentation of Jenkins.getInstanceOrNull to see what you are doing wrong. 

          System logs contained the "stack overflow" mentioned on https://docs.cloudbees.com/docs/cloudbees-ci-kb/latest/client-and-managed-controllers/stackoverflowerror-on-boot-from-queueitemauthenticatorconfiguration.
          In addition there was

          java.lang.IllegalStateException: Singleton is called recursively returning different results

          Currently, we have a work around in the form of a script which removes all the plugins and their configuration files (keeping user settings and custom views) before rebuilding the docker container and loading the configuration from CASC. Jenkins starts cleanly from a 'fresh' container every time, but sometimes fails in this way following a reboot, container restart, or 'Safe Restart'.

          We had pinned the Matrix Auth at Version 3.1.10, as it took some time to refactor the job permissions. I will be interested to see, now we have unpinned this plug-in whether Jenkins survives reboot.

          Liam Baker added a comment - - edited Saw the same on Jenkins 2.452.1 We previously thought it was due to too many items on the queue (300+). However this time we stopped all jobs, removed all items from the queue, set Jenkins in "prepare for shutdown". Then rebooted the Jenkins Controller (reboot to install OS updates). On reboot Jenkins UI is throwing a 500 error with the following trace. java.lang.IllegalStateException: Jenkins.instance is missing. Read the documentation of Jenkins.getInstanceOrNull to see what you are doing wrong. System logs contained the "stack overflow" mentioned on https://docs.cloudbees.com/docs/cloudbees-ci-kb/latest/client-and-managed-controllers/stackoverflowerror-on-boot-from-queueitemauthenticatorconfiguration . In addition there was java.lang.IllegalStateException: Singleton is called recursively returning different results Currently, we have a work around in the form of a script which removes all the plugins and their configuration files (keeping user settings and custom views) before rebuilding the docker container and loading the configuration from CASC. Jenkins starts cleanly from a 'fresh' container every time, but sometimes fails in this way following a reboot, container restart, or 'Safe Restart'. We had pinned the Matrix Auth at Version 3.1.10, as it took some time to refactor the job permissions. I will be interested to see, now we have unpinned this plug-in whether Jenkins survives reboot.

          Liam Baker added a comment - - edited

          I was able to find the logs starting from Jenkins boot

          2024-07-02 11:07:22.135+0000 [id=1] INFO winstone.Logger#logInternal: Beginning extraction from war file
          2024-07-02 21:07:22.246 
          2024-07-02 11:07:22.246+0000 [id=1] WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath
          2024-07-02 21:07:22.302 
          2024-07-02 11:07:22.302+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: jetty-10.0.20; built: 2024-01-29T20:46:45.278Z; git: 3a745c71c23682146f262b99f4ddc4c1bc41630c; jvm 17.0.11+9
          2024-07-02 21:07:22.705 
          2024-07-02 11:07:22.705+0000 [id=1] INFO o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
          2024-07-02 21:07:22.763 
          2024-07-02 11:07:22.763+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: Session workerName=node0
          2024-07-02 21:07:23.393 
          2024-07-02 11:07:23.393+0000 [id=1] INFO hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/jenkins_home found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
          2024-07-02 21:07:23.529 
          2024-07-02 11:07:23.528+0000 [id=1] INFO o.e.j.s.handler.ContextHandler#doStart: Started w.@a619c2
          
          {Jenkins v2.452.1,/,file:///var/jenkins_home/war/,AVAILABLE} \{/var/jenkins_home/war}
          
          2024-07-02 21:07:23.538 
          2024-07-02 11:07:23.537+0000 [id=1] INFO o.e.j.server.AbstractConnector#doStart: Started ServerConnector@4f67eb2a
          
          {HTTP/1.1, (http/1.1)} \{0.0.0.0:8080}
          
          2024-07-02 21:07:23.549 
          2024-07-02 11:07:23.549+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: Started Server@272ed83b
          
          {STARTING}
          
          [10.0.20,sto=0] @2057ms
          2024-07-02 21:07:23.550 
          2024-07-02 11:07:23.550+0000 [id=34] INFO winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled
          2024-07-02 21:07:23.842 
          2024-07-02 11:07:23.842+0000 [id=41] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization
          2024-07-02 21:07:24.606 
          2024-07-02 11:07:24.606+0000 [id=42] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins
          2024-07-02 21:07:31.545 
          2024-07-02 11:07:31.545+0000 [id=54] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
          2024-07-02 21:07:31.574 
          2024-07-02 11:07:31.573+0000 [id=55] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins
          2024-07-02 21:07:31.586 
          2024-07-02 11:07:31.586+0000 [id=43] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
          2024-07-02 21:07:31.607 
          2024-07-02 11:07:31.606+0000 [id=40] INFO j.a.PrioritySorterPlugin#init1: Configuring the Priority Sorter ...
          

          So far, so normal, but the next line is the first of a 2700 line stack trace which begins with

          2024-07-02 21:07:31.919 
          at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
          2024-07-02 21:07:31.919 
          at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
          2024-07-02 21:07:31.919 
          at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
          2024-07-02 21:07:31.919 
          at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:454)
          2024-07-02 21:07:31.919 
          at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:350)
          2024-07-02 21:07:31.919 
          at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:289)
          2024-07-02 21:07:31.919 
          at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
          

          and ends with

          at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:335)
          2024-07-02 21:07:31.954 
          2024-07-02 11:07:31.935+0000 [id=37] WARNING h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate Key[type=jenkins.security.QueueItemAuthenticatorConfiguration, annotation=[none]]; skipping this component
          2024-07-02 21:07:31.954 
          java.lang.IllegalStateException: Singleton is called recursively returning different results
          

          This trace then repeats a few times per second for the next few minutes.|

          Could the `PrioritySorterPlugin` be connected to this?

          Liam Baker added a comment - - edited I was able to find the logs starting from Jenkins boot 2024-07-02 11:07:22.135+0000 [id=1] INFO winstone.Logger#logInternal: Beginning extraction from war file 2024-07-02 21:07:22.246 2024-07-02 11:07:22.246+0000 [id=1] WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath 2024-07-02 21:07:22.302 2024-07-02 11:07:22.302+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: jetty-10.0.20; built: 2024-01-29T20:46:45.278Z; git: 3a745c71c23682146f262b99f4ddc4c1bc41630c; jvm 17.0.11+9 2024-07-02 21:07:22.705 2024-07-02 11:07:22.705+0000 [id=1] INFO o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet 2024-07-02 21:07:22.763 2024-07-02 11:07:22.763+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: Session workerName=node0 2024-07-02 21:07:23.393 2024-07-02 11:07:23.393+0000 [id=1] INFO hudson.WebAppMain#contextInitialized: Jenkins home directory: / var /jenkins_home found at: EnvVars.masterEnvVars.get( "JENKINS_HOME" ) 2024-07-02 21:07:23.529 2024-07-02 11:07:23.528+0000 [id=1] INFO o.e.j.s.handler.ContextHandler#doStart: Started w.@a619c2 {Jenkins v2.452.1,/,file: /// var /jenkins_home/war/,AVAILABLE} \{/ var /jenkins_home/war} 2024-07-02 21:07:23.538 2024-07-02 11:07:23.537+0000 [id=1] INFO o.e.j.server.AbstractConnector#doStart: Started ServerConnector@4f67eb2a {HTTP/1.1, (http/1.1)} \{0.0.0.0:8080} 2024-07-02 21:07:23.549 2024-07-02 11:07:23.549+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: Started Server@272ed83b {STARTING} [10.0.20,sto=0] @2057ms 2024-07-02 21:07:23.550 2024-07-02 11:07:23.550+0000 [id=34] INFO winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled 2024-07-02 21:07:23.842 2024-07-02 11:07:23.842+0000 [id=41] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization 2024-07-02 21:07:24.606 2024-07-02 11:07:24.606+0000 [id=42] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins 2024-07-02 21:07:31.545 2024-07-02 11:07:31.545+0000 [id=54] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins 2024-07-02 21:07:31.574 2024-07-02 11:07:31.573+0000 [id=55] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins 2024-07-02 21:07:31.586 2024-07-02 11:07:31.586+0000 [id=43] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions 2024-07-02 21:07:31.607 2024-07-02 11:07:31.606+0000 [id=40] INFO j.a.PrioritySorterPlugin#init1: Configuring the Priority Sorter ... So far, so normal, but the next line is the first of a 2700 line stack trace which begins with 2024-07-02 21:07:31.919 at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74) 2024-07-02 21:07:31.919 at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72) 2024-07-02 21:07:31.919 at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68) 2024-07-02 21:07:31.919 at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:454) 2024-07-02 21:07:31.919 at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:350) 2024-07-02 21:07:31.919 at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:289) 2024-07-02 21:07:31.919 at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74) and ends with at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:335) 2024-07-02 21:07:31.954 2024-07-02 11:07:31.935+0000 [id=37] WARNING h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate Key[type=jenkins.security.QueueItemAuthenticatorConfiguration, annotation=[none]]; skipping this component 2024-07-02 21:07:31.954 java.lang.IllegalStateException: Singleton is called recursively returning different results This trace then repeats a few times per second for the next few minutes.| Could the `PrioritySorterPlugin` be connected to this?

          Liam Baker added a comment - - edited

          I have traced the exception 

          Singleton is called recursively returning different results

           to code in com.google.inject.internal.SingletonScope
          It may be related to https://github.com/google/guice/issues/1510

           

          Liam Baker added a comment - - edited I have traced the exception  Singleton is called recursively returning different results  to code in com.google.inject.internal.SingletonScope It may be related to https://github.com/google/guice/issues/1510  

          Liam Baker added a comment - - edited

          Hi, this is consistently reproducible in Jenkins 2.542.2 LTS
          If a resource or the Jenkins controller URL is queried while Jenkins is starting up, the stack overflow is triggered.
          If the controller, or just the jenkins container is restarted, and no query of any URL is made, jenkins will start without issue.

          Sometimes we get a full crash, other times Jenkins serves an error page for a while before crashing.

          Error loading configuration

           
          Invalid configuration elements for type:

           

          security

          Attribute was:

           

          queueItemAuthenticator

          Valid attributes are:

          • apiToken
          • apiTokenProperty
          • badgePlugin
          • copyartifact
          • crumb
          • envInject
          • gitHooks
          • gitHostKeyVerificationConfiguration
          • globalJobDslSecurityConfiguration
          • prism
          • sSHD
          • scriptApproval
          • updateSiteWarningsConfiguration

          Liam Baker added a comment - - edited Hi, this is consistently reproducible in Jenkins 2.542.2 LTS If a resource or the Jenkins controller URL is queried while Jenkins is starting up, the stack overflow is triggered. If the controller, or just the jenkins container is restarted, and no query of any URL is made, jenkins will start without issue. Sometimes we get a full crash, other times Jenkins serves an error page for a while before crashing. Error loading configuration   Invalid configuration elements for type:   security Attribute was:   queueItemAuthenticator Valid attributes are: apiToken apiTokenProperty badgePlugin copyartifact crumb envInject gitHooks gitHostKeyVerificationConfiguration globalJobDslSecurityConfiguration prism sSHD scriptApproval updateSiteWarningsConfiguration

          Sudarshan T N added a comment - - edited

          The same issue in 2.462.1 LTS.

          Hi mawinter69 , I apologize for bothering you. I was wondering if you might have any insight into this bug.

          Sudarshan T N added a comment - - edited The same issue in 2.462.1 LTS. Hi mawinter69 , I apologize for bothering you. I was wondering if you might have any insight into this bug.

          Sudarshan T N added a comment -

          do we get any fix for this issue?

          Sudarshan T N added a comment - do we get any fix for this issue?

            Unassigned Unassigned
            tumbl3w33d Sir Tumbleweed
            Votes:
            42 Vote for this issue
            Watchers:
            47 Start watching this issue

              Created:
              Updated: