• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • core
    • 2.458

      This request was executed by blocking the resource, thereby causing the jenknis to stop completely, all subsequent threads were waiting for its completion

      This seems to be a core problem in query processing

      "Handling POST /legacy/job/SberBoardML/ajaxExecutors from 10.86.4.55 : qtp739498517-366123 View/ajaxExecutors.jelly" #366123 prio=5 os_prio=0 tid=0x00007f0344a47000 nid=0x52b2 runnable [0x00007f016e2fb000]   java.lang.Thread.State: RUNNABLEat java.util.WeakHashMap.expungeStaleEntries(WeakHashMap.java:341)- locked <0x00000001c9e75c78> (a java.lang.ref.ReferenceQueue)at java.util.WeakHashMap.getTable(WeakHashMap.java:350)at java.util.WeakHashMap.get(WeakHashMap.java:397)at org.apache.commons.beanutils.WrapDynaClass.createDynaClass(WrapDynaClass.java:426)at org.apache.commons.beanutils.WrapDynaClass.createDynaClass(WrapDynaClass.java:405)at org.apache.commons.beanutils.WrapDynaBean.getDynaClass(WrapDynaBean.java:230)at org.apache.commons.beanutils.WrapDynaBean.<init>(WrapDynaBean.java:79)at org.apache.commons.beanutils.WrapDynaBean.<init>(WrapDynaBean.java:61)at org.apache.commons.beanutils.ConvertingWrapDynaBean.<init>(ConvertingWrapDynaBean.java:47)at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:246)at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)at org.apache.commons.jelly.tags.define.TaglibTag.doTag(TaglibTag.java:64)at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41)at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:56)at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:43)at org.kohsuke.stapler.Facet$1.dispatch(Facet.java:238)at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:815)at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:280)at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)at org.kohsuke.stapler.Stapler.invoke(Stapler.java:676)at org.kohsuke.stapler.Stapler.service(Stapler.java:238)at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:246)at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:92)at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)at hudson.plugins.locale.LocaleFilter.doFilter(LocaleFilter.java:42)at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114)at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128)at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:64)at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1701)at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1668)at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)at org.eclipse.jetty.server.Server.handle(Server.java:502)at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)at java.lang.Thread.run(Thread.java:748)
         Locked ownable synchronizers:- None
      

          [JENKINS-60997] Unexpected thread blocking

          Oleg Nenashev added a comment -

          vnenashev I doubt that you wanted to attach a screenshot of a private conversation in WhatsApp

          Oleg Nenashev added a comment - vnenashev I doubt that you wanted to attach a screenshot of a private conversation in WhatsApp

          Oleg Nenashev added a comment -

          Regarding the issue itself, there is no evidence that the provided thread is blocking anything.

          It would be great to have a thread dump from the instance

          Oleg Nenashev added a comment - Regarding the issue itself, there is no evidence that the provided thread is blocking anything. It would be great to have a thread dump from the instance

          Alex Taylor added a comment -

          Adding some additional things to this issue because I just saw this today after updating last friday oleg_nenashev:

          Issue:
          Jenkins UI becomes unusable. It does not show significant CPU usage (1 thread is maxing a single core but nothing else is really working hard), in the background Jenkins continues to build and execute, but the UI is unreachable. It is so similar to REST APIs flooding the UI that I diagnosed as such and after a thread dump I found I was wrong.

          Symptoms: There appears to be a single thread trying to remove a weakHashMap entry and it blocks all other threads doing something similar for at least 60 seconds in my case (the thread dump set will show the same). This did not happen before the upgrade when I went from 2.138.1 to 2.222.4 but that leaves a very large window of changes. Hopefully the thread dump can help narrow it down?

          Attaching the jenkinsHang script output which will include a series of thread dumps, top information done at nearly the same time, top -h information at the same time to see individual threads and a support bundle so you can see plugin versions and such but it appear to me to be Jenkins core. Maybe a stapler update or surrounding dependency?

          Script set: jenkinshangWithJstack.17309.output.tar.gz
          Bundle: support_2020-06-17_01.17.37.zip

          Alex Taylor added a comment - Adding some additional things to this issue because I just saw this today after updating last friday oleg_nenashev : Issue: Jenkins UI becomes unusable. It does not show significant CPU usage (1 thread is maxing a single core but nothing else is really working hard), in the background Jenkins continues to build and execute, but the UI is unreachable. It is so similar to REST APIs flooding the UI that I diagnosed as such and after a thread dump I found I was wrong. Symptoms: There appears to be a single thread trying to remove a weakHashMap entry and it blocks all other threads doing something similar for at least 60 seconds in my case (the thread dump set will show the same). This did not happen before the upgrade when I went from 2.138.1 to 2.222.4 but that leaves a very large window of changes. Hopefully the thread dump can help narrow it down? Attaching the jenkinsHang script output which will include a series of thread dumps, top information done at nearly the same time, top -h information at the same time to see individual threads and a support bundle so you can see plugin versions and such but it appear to me to be Jenkins core. Maybe a stapler update or surrounding dependency? Script set: jenkinshangWithJstack.17309.output.tar.gz Bundle: support_2020-06-17_01.17.37.zip

          Alex Taylor added a comment -

          Realized something this morning, this may have been triggered by a reload from disk. I had not restarted and I needed security to get re-enabled but did not want to do a full restart so I adjusted the config.xml on disk and then did a reload config from disk. This issue started not long after that which means they could be related in case this needs to be reproduced locally

          Alex Taylor added a comment - Realized something this morning, this may have been triggered by a reload from disk. I had not restarted and I needed security to get re-enabled but did not want to do a full restart so I adjusted the config.xml on disk and then did a reload config from disk. This issue started not long after that which means they could be related in case this needs to be reproduced locally

          Pierre Beitz added a comment -

          I saw this same issue on a totally different widget, maybe it's related to https://issues.apache.org/jira/browse/BEANUTILS-509 ?

          Pierre Beitz added a comment - I saw this same issue on a totally different widget, maybe it's related to https://issues.apache.org/jira/browse/BEANUTILS-509  ?

          Alex Taylor added a comment -

          Looks like there is a PR out there: https://github.com/apache/commons-beanutils/pull/56 which may fix this. Then we would need to pull in the newer apache libraries so that is neat

          Alex Taylor added a comment - Looks like there is a PR out there: https://github.com/apache/commons-beanutils/pull/56 which may fix this. Then we would need to pull in the newer apache libraries so that is neat

          FTR, I've seen today a 2.289.3 Jenkins controller overwhelmed by threads like this:

          "Handling GET /jenkins/job/XXXXXXXXXXXX/configure from X.X.X.X : Jetty (winstone)-16075 Job/configure.jelly WorkflowJob/configure-entries.jelly CpsScmFlowDefinition/DescriptorImpl/config.jelly GitSCM/DescriptorImpl/config.jelly" #16075 prio=5 os_prio=0 tid=0x00007ff94400d800 nid=0x4345 runnable [0x00007ff90aae7000]
             java.lang.Thread.State: RUNNABLE
          	at java.util.WeakHashMap.eq(WeakHashMap.java:287)
          	at java.util.WeakHashMap.get(WeakHashMap.java:401)
          	at org.apache.commons.beanutils.WrapDynaClass.createDynaClass(WrapDynaClass.java:426)
          	at org.apache.commons.beanutils.WrapDynaClass.createDynaClass(WrapDynaClass.java:405)
          	at org.apache.commons.beanutils.WrapDynaBean.getDynaClass(WrapDynaBean.java:230)
          	at org.apache.commons.beanutils.WrapDynaBean.<init>(WrapDynaBean.java:79)
          	at org.apache.commons.beanutils.WrapDynaBean.<init>(WrapDynaBean.java:61)
          	at org.apache.commons.beanutils.ConvertingWrapDynaBean.<init>(ConvertingWrapDynaBean.java:47)
          	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:246)
          	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:100)
          	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:100)
          	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
          	at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
          	at org.kohsuke.stapler.jelly.groovy.JellyBuilder.doInvokeMethod(JellyBuilder.java:272)
          	at org.kohsuke.stapler.jelly.groovy.Namespace.invokeMethod(Namespace.java:56)
          	at org.kohsuke.stapler.jelly.groovy.Namespace$ProxyImpl.invokeMethod(Namespace.java:101)
          	at sun.reflect.GeneratedMethodAccessor884.invoke(Unknown Source)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at org.kohsuke.stapler.jelly.groovy.Namespace$ProxyImpl.invoke(Namespace.java:83)
          	at com.sun.proxy.$Proxy89.invokeMethod(Unknown Source)
          	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
          	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
          	at hudson.plugins.git.UserRemoteConfig.config$_run_closure2.doCall(config.groovy:11)
          	at hudson.plugins.git.UserRemoteConfig.config$_run_closure2.doCall(config.groovy)
          ...
          

          It had happened soon after Jenkins being restarted (for a 2.277.4 to 2.289.3 upgrade), with two "almost simultaneous" requests to some /jenkins/job/XXX/configure pages.

          BEANUTILS-509 is indeed very likely the right upstream bug report.

          As for commons-beanutils #56, I don't think it fixes BEANUTILS-509, but the ConcurrentWeakKeyHashMap collection it introduces could probably be used there too:

          Thomas de Grenier de Latour added a comment - FTR, I've seen today a 2.289.3 Jenkins controller overwhelmed by threads like this: "Handling GET /jenkins/job/XXXXXXXXXXXX/configure from X.X.X.X : Jetty (winstone)-16075 Job/configure.jelly WorkflowJob/configure-entries.jelly CpsScmFlowDefinition/DescriptorImpl/config.jelly GitSCM/DescriptorImpl/config.jelly" #16075 prio=5 os_prio=0 tid=0x00007ff94400d800 nid=0x4345 runnable [0x00007ff90aae7000] java.lang. Thread .State: RUNNABLE at java.util.WeakHashMap.eq(WeakHashMap.java:287) at java.util.WeakHashMap.get(WeakHashMap.java:401) at org.apache.commons.beanutils.WrapDynaClass.createDynaClass(WrapDynaClass.java:426) at org.apache.commons.beanutils.WrapDynaClass.createDynaClass(WrapDynaClass.java:405) at org.apache.commons.beanutils.WrapDynaBean.getDynaClass(WrapDynaBean.java:230) at org.apache.commons.beanutils.WrapDynaBean.<init>(WrapDynaBean.java:79) at org.apache.commons.beanutils.WrapDynaBean.<init>(WrapDynaBean.java:61) at org.apache.commons.beanutils.ConvertingWrapDynaBean.<init>(ConvertingWrapDynaBean.java:47) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:246) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:100) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:100) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.kohsuke.stapler.jelly.groovy.JellyBuilder.doInvokeMethod(JellyBuilder.java:272) at org.kohsuke.stapler.jelly.groovy.Namespace.invokeMethod(Namespace.java:56) at org.kohsuke.stapler.jelly.groovy.Namespace$ProxyImpl.invokeMethod(Namespace.java:101) at sun.reflect.GeneratedMethodAccessor884.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.kohsuke.stapler.jelly.groovy.Namespace$ProxyImpl.invoke(Namespace.java:83) at com.sun.proxy.$Proxy89.invokeMethod(Unknown Source) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) at hudson.plugins.git.UserRemoteConfig.config$_run_closure2.doCall(config.groovy:11) at hudson.plugins.git.UserRemoteConfig.config$_run_closure2.doCall(config.groovy) ... It had happened soon after Jenkins being restarted (for a 2.277.4 to 2.289.3 upgrade), with two "almost simultaneous" requests to some /jenkins/job/XXX/configure pages. BEANUTILS-509 is indeed very likely the right upstream bug report. As for commons-beanutils #56 , I don't think it fixes BEANUTILS-509, but the ConcurrentWeakKeyHashMap collection it introduces could probably be used there too: WrapDynaClass.java#L135 (in 1.9.x) WrapDynaClass.java#L119 (in 2.x)  

          Jon Sten added a comment - - edited

          Also seen this, three days after upgrading to 2.289.3 (from 2.289.1). All computed folders (e.g. multibranch pipeline jobs in our case) fail to load. Slightly different stacktrace in our case, we fail later in WeakHashMap#get():

          "Handling GET /jenkins/job/xxx/job/xxx/job/xxx/job/xxxx/ from 1.2.3.4 : Jetty (winstone)-4912298 View/index.jelly View/sidepanel.jelly AbstractFolder/tasks-top.jelly" Id=4912298 Group=main RUNNABLE
          	at java.util.WeakHashMap.get(WeakHashMap.java:403)
          	at org.apache.commons.beanutils.WrapDynaClass.createDynaClass(WrapDynaClass.java:426)
          	at org.apache.commons.beanutils.WrapDynaClass.createDynaClass(WrapDynaClass.java:405)
          	at org.apache.commons.beanutils.WrapDynaBean.getDynaClass(WrapDynaBean.java:230)
          	at org.apache.commons.beanutils.WrapDynaBean.<init>(WrapDynaBean.java:79)
          	at org.apache.commons.beanutils.WrapDynaBean.<init>(WrapDynaBean.java:61)
          	at org.apache.commons.beanutils.ConvertingWrapDynaBean.<init>(ConvertingWrapDynaBean.java:47)
          	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:246)
          	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
          	at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
          	at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:171)
          	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
          	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          	at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
          	at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
          	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
          	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:100)
          	....
          

          Jon Sten added a comment - - edited Also seen this, three days after upgrading to 2.289.3 (from 2.289.1). All computed folders (e.g. multibranch pipeline jobs in our case) fail to load. Slightly different stacktrace in our case, we fail later in WeakHashMap#get(): "Handling GET /jenkins/job/xxx/job/xxx/job/xxx/job/xxxx/ from 1.2.3.4 : Jetty (winstone)-4912298 View/index.jelly View/sidepanel.jelly AbstractFolder/tasks-top.jelly" Id=4912298 Group=main RUNNABLE at java.util.WeakHashMap.get(WeakHashMap.java:403) at org.apache.commons.beanutils.WrapDynaClass.createDynaClass(WrapDynaClass.java:426) at org.apache.commons.beanutils.WrapDynaClass.createDynaClass(WrapDynaClass.java:405) at org.apache.commons.beanutils.WrapDynaBean.getDynaClass(WrapDynaBean.java:230) at org.apache.commons.beanutils.WrapDynaBean.<init>(WrapDynaBean.java:79) at org.apache.commons.beanutils.WrapDynaBean.<init>(WrapDynaBean.java:61) at org.apache.commons.beanutils.ConvertingWrapDynaBean.<init>(ConvertingWrapDynaBean.java:47) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:246) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95) at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:171) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:100) ....

          John Lengeling added a comment - - edited

          We had one of our 80+ Jenkins instances become unresponsive via the GUI but was still up.   High CPU usage and lots of BLOCK threads having the same kind of thread dump information on commons.beanutils.   We were able to get a full thread dump of the system, but here is just one example:

          We are running 2.387.3. Any progress in getting the common.beanutil maintainers to backport this defect or to cut a new release?

          "Handling GET /login from 192.168.168.145 : Jetty (winstone)-99033 Jenkins/login.jelly" #99033 prio=5 os_prio=0 cpu=2517.70ms elapsed=199856.31s allocated=256M defined_classes=0 tid=0x00007fe9780d9800 nid=0x3ee1 waiting for monitor entry  [0x00007fe988719000]
             java.lang.Thread.State: BLOCKED (on object monitor)
                  at java.util.WeakHashMap.expungeStaleEntries(java.base@11.0.19/WeakHashMap.java:319)
                  - waiting to lock <0x0000000682c534a8> (a java.lang.ref.ReferenceQueue)
                  at java.util.WeakHashMap.getTable(java.base@11.0.19/WeakHashMap.java:350)
                  at java.util.WeakHashMap.get(java.base@11.0.19/WeakHashMap.java:398)
                  at org.apache.commons.beanutils.WrapDynaClass.createDynaClass(WrapDynaClass.java:426)
                  at org.apache.commons.beanutils.WrapDynaClass.createDynaClass(WrapDynaClass.java:405)
                  at org.apache.commons.beanutils.WrapDynaBean.getDynaClass(WrapDynaBean.java:230)
                  at org.apache.commons.beanutils.WrapDynaBean.<init>(WrapDynaBean.java:79)
                  at org.apache.commons.beanutils.WrapDynaBean.<init>(WrapDynaBean.java:61)
                  at org.apache.commons.beanutils.ConvertingWrapDynaBean.<init>(ConvertingWrapDynaBean.java:47)
                  at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:242)
                  at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
                  at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                  at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
                  at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:122)
                  at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
                  at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:99)
                  at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:66)
                  at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:55)
                  at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:56)
                  at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:43)
                  at org.kohsuke.stapler.Facet$1.dispatch(Facet.java:240)
                  at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
                  at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
                  at org.kohsuke.stapler.Stapler.invoke(Stapler.java:698)
                  at org.kohsuke.stapler.Stapler.service(Stapler.java:248)
                  at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
                  at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
                  at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)
                  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:157)
                  at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
                  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
                  at hudson.plugins.locale.LocaleFilter.doFilter(LocaleFilter.java:42)
                  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
                  at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:112)
                  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
                  at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129)
                  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
                  at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:81)
                  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
                  at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
                  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
                  at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
                  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
                  at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
                  at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
                  at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
                  at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:121)
                  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
                  at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
                  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
                  at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:160)
                  at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
                  at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
                  at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:160)
                  at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
                  at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94)
                  at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
                  at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
                  at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
                  at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
                  at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
                          at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:110)
                  at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
                  at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227)
                  at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
                  at jenkins.security.BasicHeaderProcessor.success(BasicHeaderProcessor.java:139)
                  at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:86)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
                  at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117)
                  at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
                  at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63)
                  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
                  at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111)
                  at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172)
                  at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
                  at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
                  at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
                  at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
                  at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
                  at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86)
                  at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
                  at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
                  at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
                  at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
                  at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
                  at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38)
                  at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
                  at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
                  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
                  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
                  at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:549)
                  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
                  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
                  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)
                  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
                  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383)
                  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
                  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
                  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
                  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
                  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305)
                  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
                  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
                  at org.eclipse.jetty.server.Server.handle(Server.java:563)
                  at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)
                  at org.eclipse.jetty.server.HttpChannel$$Lambda$280/0x00000008003f5c40.dispatch(Unknown Source)
                  at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)
                  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)
                  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
                  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
                  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
                  at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
                  at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416)
                  at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385)
                  at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272)
                  at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140)
                  at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy$$Lambda$252/0x00000008003dc040.run(Unknown Source)
                  at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
                  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
                  at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
                  at java.lang.Thread.run(java.base@11.0.19/Thread.java:829)
             Locked ownable synchronizers:
                  - None
          

           

          John Lengeling added a comment - - edited We had one of our 80+ Jenkins instances become unresponsive via the GUI but was still up.   High CPU usage and lots of BLOCK threads having the same kind of thread dump information on commons.beanutils.   We were able to get a full thread dump of the system, but here is just one example: We are running 2.387.3. Any progress in getting the common.beanutil maintainers to backport this defect or to cut a new release? "Handling GET /login from 192.168.168.145 : Jetty (winstone)-99033 Jenkins/login.jelly" #99033 prio=5 os_prio=0 cpu=2517.70ms elapsed=199856.31s allocated=256M defined_classes=0 tid=0x00007fe9780d9800 nid=0x3ee1 waiting for monitor entry [0x00007fe988719000] java.lang. Thread .State: BLOCKED (on object monitor) at java.util.WeakHashMap.expungeStaleEntries(java.base@11.0.19/WeakHashMap.java:319) - waiting to lock <0x0000000682c534a8> (a java.lang.ref.ReferenceQueue) at java.util.WeakHashMap.getTable(java.base@11.0.19/WeakHashMap.java:350) at java.util.WeakHashMap.get(java.base@11.0.19/WeakHashMap.java:398) at org.apache.commons.beanutils.WrapDynaClass.createDynaClass(WrapDynaClass.java:426) at org.apache.commons.beanutils.WrapDynaClass.createDynaClass(WrapDynaClass.java:405) at org.apache.commons.beanutils.WrapDynaBean.getDynaClass(WrapDynaBean.java:230) at org.apache.commons.beanutils.WrapDynaBean.<init>(WrapDynaBean.java:79) at org.apache.commons.beanutils.WrapDynaBean.<init>(WrapDynaBean.java:61) at org.apache.commons.beanutils.ConvertingWrapDynaBean.<init>(ConvertingWrapDynaBean.java:47) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:242) at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:122) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:99) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:66) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:55) at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:56) at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:43) at org.kohsuke.stapler.Facet$1.dispatch(Facet.java:240) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:698) at org.kohsuke.stapler.Stapler.service(Stapler.java:248) at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:157) at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at hudson.plugins.locale.LocaleFilter.doFilter(LocaleFilter.java:42) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:112) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:81) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:121) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:160) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:160) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94) at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:110) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at jenkins.security.BasicHeaderProcessor.success(BasicHeaderProcessor.java:139) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:86) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:549) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) at org.eclipse.jetty.server.Server.handle(Server.java:563) at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505) at org.eclipse.jetty.server.HttpChannel$$Lambda$280/0x00000008003f5c40.dispatch(Unknown Source) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$ new $0(AdaptiveExecutionStrategy.java:140) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy$$Lambda$252/0x00000008003dc040.run(Unknown Source) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078) at java.lang. Thread .run(java.base@11.0.19/ Thread .java:829) Locked ownable synchronizers: - None  

          Basil Crow added a comment -

          2.458 contains a workaround for this upstream issue. I will not be monitoring this ticket, and any further issues should be discussed in a separate ticket (with complete stack trace).

          Basil Crow added a comment - 2.458 contains a workaround for this upstream issue. I will not be monitoring this ticket, and any further issues should be discussed in a separate ticket (with complete stack trace).

            basil Basil Crow
            vnenashev Владислав Ненашев
            Votes:
            2 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated:
              Resolved: