Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65901

Thread deadlock since workflow-api-plugin-2.44

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Duplicate
    • Component/s: workflow-api-plugin
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      Since workflow-api v2.44, we have been experiencing thread deadlocks on Jenkins, which result in Jenkins becoming unresponsive.

      This appears likely to be a result of this PR: https://github.com/jenkinsci/workflow-api-plugin/pull/153

      We don't have repro steps for this issue unfortunately, but have found that the issue stopped occurring when we removed the BlueOcean plugins (based on the requests involved in the deadlocks), so we believe they exercise the appropriate paths.

      We are in the process of rolling back to 2.42 to confirm, but as we can't reliably repro, it will take a couple of days to get good certainty that that resolves the issue.

      Example stack traces for deadlocked threads (Apologies for the ellipses, it's an issue in our monitoring platform. Hopefully it's clear enough what's going on):
      **

      Deadlock: Deadlocked thread: Handling GET /blue/rest/organizations/jenkins/pipelines/STEP-Landonline-Web-Survey/branches/SURVEY-2248-prepare-parcel-linking-front-end/runs/13/nodes/ from 10.80.10.90 : Jetty (winstone)-7775
      ...sci.plugins.workflow.cps.CpsFlowExecution$Timing.close(CpsFlowExecution.java:426)
      ...low.cps.CpsFlowExecution$TimingFlowNodeStorage.getNode(CpsFlowExecution.java:1817)
      ...enkinsci.plugins.workflow.cps.CpsFlowExecution.getNode(CpsFlowExecution.java:1191)
      ....graph.StandardGraphLookupView.findEnclosingBlockStart(StandardGraphLookupView.java:185)
      ...w$EnclosingBlocksIterable$EnclosingBlocksIterator.next(GraphLookupView.java:80)
      ...w$EnclosingBlocksIterable$EnclosingBlocksIterator.next(GraphLookupView.java:63)
      ...sci.plugins.workflow.graphanalysis.FilteratorImpl.next(FilteratorImpl.java:74)
      ...i.plugins.workflow.graphanalysis.FilteratorImpl.<init>(FilteratorImpl.java:53)
      ...i.plugins.workflow.graphanalysis.FilteratorImpl.filter(FilteratorImpl.java:45)
      ...i.plugins.workflow.graphanalysis.FilteratorImpl.filter(FilteratorImpl.java:36)
      ...dk.internal.reflect.GeneratedMethodAccessor1105.invoke(Unknown Source)
      ...k.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
             java.base@11.0.9/java.lang.reflect.Method.invoke(Unknown Source)
           org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
                        groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
                       groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
                       groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
      ...odehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:47)
      ...codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
      org.jenkinsci.plugins.pipeline.modeldefinition.Utils.findStageFlowNodes(Utils.groovy:246)
      ...modeldefinition.Utils$findStageFlowNodes$18.callStatic(Unknown Source)
      org.jenkinsci.plugins.pipeline.modeldefinition.Utils.stageHasStatusOf(Utils.groovy:366)
      ....RestartDeclarativePipelineAction.getRestartableStages(RestartDeclarativePipelineAction.java:162)
      ...cean.rest.impl.pipeline.PipelineNodeImpl.isRestartable(PipelineNodeImpl.java:185)
      java.base@11.0.9/java.lang.invoke.LambdaForm$DMH/0x00000008012ac440.invokeVirtual(LambdaForm$DMH)
      java.base@11.0.9/java.lang.invoke.LambdaForm$MH/0x00000008022b1840.invoke(LambdaForm$MH)
      java.base@11.0.9/java.lang.invoke.LambdaForm$MH/0x0000000800458440.invoke_MT(LambdaForm$MH)
      ...ueocean.commons.stapler.export.MethodProperty.getValue(MethodProperty.java:72)
      ...an.commons.stapler.export.ExportInterceptor$1.getValue(ExportInterceptor.java:46)
      ...ons.stapler.Export$BlueOceanExportInterceptor.getValue(Export.java:196)
      ...kins.blueocean.commons.stapler.export.Property.writeTo(Property.java:136)
      ...ocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228)
      ...ocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:224)
      ...jenkins.blueocean.commons.stapler.export.Model.writeTo(Model.java:199)
         io.jenkins.blueocean.commons.stapler.Export.writeOne(Export.java:177)
      ...kins.blueocean.commons.stapler.Export.serveExposedBean(Export.java:165)
           io.jenkins.blueocean.commons.stapler.Export.doJson(Export.java:108)
      ...st.pageable.PagedResponse$Processor$1.generateResponse(PagedResponse.java:70)
      ...tapler.HttpResponseRenderer$Default.handleHttpResponse(HttpResponseRenderer.java:124)
      ....stapler.HttpResponseRenderer$Default.generateResponse(HttpResponseRenderer.java:69)
                  org.kohsuke.stapler.Function.renderResponse(Function.java:164)
      ...kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:147)
                 org.kohsuke.stapler.IndexDispatcher.dispatch(IndexDispatcher.java:27)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                   org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:220)
             org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                     org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:457)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                   org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:220)
             org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                     org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:457)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                   org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:220)
             org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                     org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:457)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                   org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:220)
             org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                     org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:457)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                     org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:457)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                     org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:457)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:735)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                     org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:457)
                        org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
                           org.kohsuke.stapler.Stapler.invoke(Stapler.java:694)
                          org.kohsuke.stapler.Stapler.service(Stapler.java:240)
                       javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
               org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
      ...eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
                   hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
      ...ns.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
                   hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
               jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:76)
                   hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      ...metry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129)
                   hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
           io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
                   hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      ...ueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
                   hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      ...jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
                   hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
                  jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
                   hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
           hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:64)
                   hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
                     hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
              org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      ...rg.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
                    hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:159)
              org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      ...rg.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
              hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:92)
      jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
              hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
              hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      ...ecurity.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
      ...ecurity.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
              hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      ....authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105)
              hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      ...ion.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
      ...ion.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:92)
              hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      ...cation.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:218)
      ...cation.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:212)
              hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
               jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
              hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      ....web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
      ....web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
      ...security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:62)
              hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
                hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:109)
                        hudson.security.HudsonFilter.doFilter(HudsonFilter.java:168)
              org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      ...rg.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      ...kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:51)
              org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      ...rg.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
                 hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
              org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      ...rg.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      ...rg.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
              org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      ...rg.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
            jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:36)
              org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      ...rg.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
            org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
        org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
            org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
      org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
      ....eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
      ...g.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
      ....eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
      ...g.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
      ...g.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
             org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
      ...rg.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
      ...g.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
      ...rg.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
        org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
                       org.eclipse.jetty.server.Server.handle(Server.java:516)
         org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
      ...er.HttpChannel$$Lambda$281/0x000000080065bc40.dispatch(Unknown Source)
                org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
                  org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
           org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:279)
      ...pse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
                   org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
                   org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
      ...ipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
      ...se.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
      ...e.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
      ....eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
      ....util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
        org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
      ....eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
                        java.base@11.0.9/java.lang.Thread.run(Unknown Source)
      Deadlock: Deadlocked thread: Handling GET /blue/rest/organizations/jenkins/pipelines/STEP-Landonline-Web-Survey/branches/SURVEY-2248-prepare-parcel-linking-front-end/runs/13/nodes/ from 10.80.10.90 : Jetty (winstone)-7775
      ...ugins.workflow.graph.StandardGraphLookupView.onNewHead(StandardGraphLookupView.java:41)
      ....plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1473)
      ...nsci.plugins.workflow.cps.CpsThreadGroup.notifyNewHead(CpsThreadGroup.java:472)
      org.jenkinsci.plugins.workflow.cps.FlowHead.setNewHead(FlowHead.java:157)
      ...sci.plugins.workflow.cps.CpsFlowExecution.onProgramEnd(CpsFlowExecution.java:1255)
        org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:424)
      ...nkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
      ...g.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
      ...g.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
      ...insci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
         java.base@11.0.9/java.util.concurrent.FutureTask.run(Unknown Source)
              hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
           jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
          jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
      ....9/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
         java.base@11.0.9/java.util.concurrent.FutureTask.run(Unknown Source)
      ....0.9/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      ...0.9/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
                        java.base@11.0.9/java.lang.Thread.run(Unknown Source)

        Attachments

          Issue Links

            Activity

            nzdjb Dave Ball created issue -
            Hide
            domruf Dominik Ruf added a comment -

            We also had deadlocked threads since we updated the jenkins plugins last weekend.

            The web interface still worked, but more and more jobs stopped.

            We also suspect workflow-api-plugin as the root of the problem and therefore rolled it back to 2.42.

            Show
            domruf Dominik Ruf added a comment - We also had deadlocked threads since we updated the jenkins plugins last weekend. The web interface still worked, but more and more jobs stopped. We also suspect workflow-api-plugin as the root of the problem and therefore rolled it back to 2.42.
            domruf Dominik Ruf made changes -
            Field Original Value New Value
            Link This issue relates to JENKINS-65885 [ JENKINS-65885 ]
            jglick Jesse Glick made changes -
            Resolution Duplicate [ 3 ]
            Status Open [ 1 ] Resolved [ 5 ]
            bitwiseman Liam Newman made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              nzdjb Dave Ball
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: