Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65901

Thread deadlock since workflow-api-plugin-2.44

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Duplicate
    • Component/s: workflow-api-plugin
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      Since workflow-api v2.44, we have been experiencing thread deadlocks on Jenkins, which result in Jenkins becoming unresponsive.

      This appears likely to be a result of this PR: https://github.com/jenkinsci/workflow-api-plugin/pull/153

      We don't have repro steps for this issue unfortunately, but have found that the issue stopped occurring when we removed the BlueOcean plugins (based on the requests involved in the deadlocks), so we believe they exercise the appropriate paths.

      We are in the process of rolling back to 2.42 to confirm, but as we can't reliably repro, it will take a couple of days to get good certainty that that resolves the issue.

      Example stack traces for deadlocked threads (Apologies for the ellipses, it's an issue in our monitoring platform. Hopefully it's clear enough what's going on):
      **

      Deadlock: Deadlocked thread: Handling GET /blue/rest/organizations/jenkins/pipelines/STEP-Landonline-Web-Survey/branches/SURVEY-2248-prepare-parcel-linking-front-end/runs/13/nodes/ from 10.80.10.90 : Jetty (winstone)-7775
      ...sci.plugins.workflow.cps.CpsFlowExecution$Timing.close(CpsFlowExecution.java:426)
      ...low.cps.CpsFlowExecution$TimingFlowNodeStorage.getNode(CpsFlowExecution.java:1817)
      ...enkinsci.plugins.workflow.cps.CpsFlowExecution.getNode(CpsFlowExecution.java:1191)
      ....graph.StandardGraphLookupView.findEnclosingBlockStart(StandardGraphLookupView.java:185)
      ...w$EnclosingBlocksIterable$EnclosingBlocksIterator.next(GraphLookupView.java:80)
      ...w$EnclosingBlocksIterable$EnclosingBlocksIterator.next(GraphLookupView.java:63)
      ...sci.plugins.workflow.graphanalysis.FilteratorImpl.next(FilteratorImpl.java:74)
      ...i.plugins.workflow.graphanalysis.FilteratorImpl.<init>(FilteratorImpl.java:53)
      ...i.plugins.workflow.graphanalysis.FilteratorImpl.filter(FilteratorImpl.java:45)
      ...i.plugins.workflow.graphanalysis.FilteratorImpl.filter(FilteratorImpl.java:36)
      ...dk.internal.reflect.GeneratedMethodAccessor1105.invoke(Unknown Source)
      ...k.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
             java.base@11.0.9/java.lang.reflect.Method.invoke(Unknown Source)
           org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
                        groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
                       groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
                       groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
      ...odehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:47)
      ...codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
      org.jenkinsci.plugins.pipeline.modeldefinition.Utils.findStageFlowNodes(Utils.groovy:246)
      ...modeldefinition.Utils$findStageFlowNodes$18.callStatic(Unknown Source)
      org.jenkinsci.plugins.pipeline.modeldefinition.Utils.stageHasStatusOf(Utils.groovy:366)
      ....RestartDeclarativePipelineAction.getRestartableStages(RestartDeclarativePipelineAction.java:162)
      ...cean.rest.impl.pipeline.PipelineNodeImpl.isRestartable(PipelineNodeImpl.java:185)
      java.base@11.0.9/java.lang.invoke.LambdaForm$DMH/0x00000008012ac440.invokeVirtual(LambdaForm$DMH)
      java.base@11.0.9/java.lang.invoke.LambdaForm$MH/0x00000008022b1840.invoke(LambdaForm$MH)
      java.base@11.0.9/java.lang.invoke.LambdaForm$MH/0x0000000800458440.invoke_MT(LambdaForm$MH)
      ...ueocean.commons.stapler.export.MethodProperty.getValue(MethodProperty.java:72)
      ...an.commons.stapler.export.ExportInterceptor$1.getValue(ExportInterceptor.java:46)
      ...ons.stapler.Export$BlueOceanExportInterceptor.getValue(Export.java:196)
      ...kins.blueocean.commons.stapler.export.Property.writeTo(Property.java:136)
      ...ocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228)
      ...ocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:224)
      ...jenkins.blueocean.commons.stapler.export.Model.writeTo(Model.java:199)
         io.jenkins.blueocean.commons.stapler.Export.writeOne(Export.java:177)
      ...kins.blueocean.commons.stapler.Export.serveExposedBean(Export.java:165)
           io.jenkins.blueocean.commons.stapler.Export.doJson(Export.java:108)
      ...st.pageable.PagedResponse$Processor$1.generateResponse(PagedResponse.java:70)
      ...tapler.HttpResponseRenderer$Default.handleHttpResponse(HttpResponseRenderer.java:124)
      ....stapler.HttpResponseRenderer$Default.generateResponse(HttpResponseRenderer.java:69)
                  org.kohsuke.stapler.Function.renderResponse(Function.java:164)
      ...kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:147)
                 org.kohsuke.stapler.IndexDispatcher.dispatch(IndexDispatcher.java:27)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                   org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:220)
             org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                     org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:457)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                   org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:220)
             org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                     org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:457)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                   org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:220)
             org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                     org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:457)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                   org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:220)
             org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                     org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:457)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                     org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:457)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                     org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:457)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:735)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                     org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:457)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:694)
                          org.kohsuke.stapler.Stapler.service(Stapler.java:240)
                       javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
               org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
      ...eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
                   hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
      ...ns.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
                   hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
               jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:76)
                   hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      ...metry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129)
                   hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
           io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
                   hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      ...ueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
                   hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      ...jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
                   hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
                  jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
                   hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
           hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:64)
                   hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
                     hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
              org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      ...rg.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
                    hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:159)
              org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      ...rg.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
              hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:92)
      jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
              hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
              hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      ...ecurity.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
      ...ecurity.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
              hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      ....authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105)
              hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      ...ion.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
      ...ion.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:92)
              hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      ...cation.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:218)
      ...cation.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:212)
              hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
               jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
              hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      ....web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
      ....web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
      ...security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:62)
              hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
                hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:109)
                        hudson.security.HudsonFilter.doFilter(HudsonFilter.java:168)
              org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      ...rg.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      ...kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:51)
              org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      ...rg.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
                 hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
              org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      ...rg.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      ...rg.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
              org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      ...rg.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
            jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:36)
              org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      ...rg.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
            org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
        org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
            org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
      org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
      ....eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
      ...g.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
      ....eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
      ...g.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
      ...g.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
             org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
      ...rg.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
      ...g.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
      ...rg.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
        org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
                       org.eclipse.jetty.server.Server.handle(Server.java:516)
         org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
      ...er.HttpChannel$$Lambda$281/0x000000080065bc40.dispatch(Unknown Source)
                org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
                  org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
           org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:279)
      ...pse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
                   org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
                   org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
      ...ipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
      ...se.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
      ...e.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
      ....eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
      ....util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
        org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
      ....eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
                        java.base@11.0.9/java.lang.Thread.run(Unknown Source)
      Deadlock: Deadlocked thread: Handling GET /blue/rest/organizations/jenkins/pipelines/STEP-Landonline-Web-Survey/branches/SURVEY-2248-prepare-parcel-linking-front-end/runs/13/nodes/ from 10.80.10.90 : Jetty (winstone)-7775
      ...ugins.workflow.graph.StandardGraphLookupView.onNewHead(StandardGraphLookupView.java:41)
      ....plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1473)
      ...nsci.plugins.workflow.cps.CpsThreadGroup.notifyNewHead(CpsThreadGroup.java:472)
      org.jenkinsci.plugins.workflow.cps.FlowHead.setNewHead(FlowHead.java:157)
      ...sci.plugins.workflow.cps.CpsFlowExecution.onProgramEnd(CpsFlowExecution.java:1255)
        org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:424)
      ...nkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
      ...g.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
      ...g.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
      ...insci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
         java.base@11.0.9/java.util.concurrent.FutureTask.run(Unknown Source)
              hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
           jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
          jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
      ....9/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
         java.base@11.0.9/java.util.concurrent.FutureTask.run(Unknown Source)
      ....0.9/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      ...0.9/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
                        java.base@11.0.9/java.lang.Thread.run(Unknown Source)

        Attachments

          Issue Links

            Activity

            Hide
            domruf Dominik Ruf added a comment -

            We also had deadlocked threads since we updated the jenkins plugins last weekend.

            The web interface still worked, but more and more jobs stopped.

            We also suspect workflow-api-plugin as the root of the problem and therefore rolled it back to 2.42.

            Show
            domruf Dominik Ruf added a comment - We also had deadlocked threads since we updated the jenkins plugins last weekend. The web interface still worked, but more and more jobs stopped. We also suspect workflow-api-plugin as the root of the problem and therefore rolled it back to 2.42.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              nzdjb Dave Ball
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: