Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45861

SCM pipelines get stuck and go into a zombie unkillable state

      I am running Jenkins ver. 2.60.1 with git client 2.4.6 and git plugin 3.4.1

      I am running a pipeline based on a Jenkinsfile from SCM.

      I have noticed that sometimes, my pipeline seems stuck.

      I go to the "/restart" page and initiate a restart. Later, I still see some of the pipelines in a stuck state. For example, in the console, I see:

      Started by user myghmachineuser
      Checking out git ${GITHUB_REPOSITORY} into /var/jenkins_home/workspace/redacted@script to read Jenkinsfile-nodeploy.groovy
       > git rev-parse --is-inside-work-tree # timeout=10
      Fetching changes from the remote Git repository
       > git config remote.origin.url git@github.com:redacted/redacted.git # timeout=10
      Fetching upstream changes from git@github.com:redacted/redacted.git
       > git --version # timeout=10
      using GIT_SSH to set credentials 
       > git fetch --tags --progress git@github.com:redacted/redacted.git +refs/heads/*:refs/remotes/origin/*
       > git rev-parse 3f96f044eaec0fabf966d702fe727560bcbf9dbf^{commit} # timeout=10
      Checking out Revision 3f96f044eaec0fabf966d702fe727560bcbf9dbf (detached)
      Commit message: "revert service-lib"
       > git config core.sparsecheckout # timeout=10
       > git checkout -f 3f96f044eaec0fabf966d702fe727560bcbf9dbf
       > git rev-list 7f3be5084de22472b8c2a9de339f588a6427915f # timeout=10
      Click here to forcibly kill entire build
      Click here to forcibly kill entire build
      Click here to forcibly kill entire build

      When I click on the forcibly kill links, nothing happens. This is a great example of a pipeline run that seems to be a zombie that will not go away.

      I looked in the Jenkins master logs and I see:

      Jul 28, 2017 3:09:46 PM WARNING org.jenkinsci.plugins.workflow.job.WorkflowRun finish
      doorstep-django-nodeploy #9037 failed to start
      org.jenkinsci.plugins.workflow.steps.FlowInterruptedException
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.doKill(WorkflowRun.java:417)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.doStop(WorkflowRun.java:804)
      	at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
      	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343)
      	at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:52)
      	at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
      	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184)
      	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117)
      	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:129)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
      	at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
      	at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:248)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
      	at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
      	at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
      	at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:51)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
      	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:237)
      	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:209)
      	at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
      	at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:113)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
      	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:138)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:92)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.server.Server.handle(Server.java:499)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
      	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
      	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:748)

       

          [JENKINS-45861] SCM pipelines get stuck and go into a zombie unkillable state

          Mark Waite added a comment -

          I'm making a wild guess that jglick is better suited to understand this request than I am.

          When I have jobs that I attempt to kill from job page, sometimes they report that message in the log, and then I click the "Forcibly kill" link in the log.

          I'm not sure if workflow-durable-task plugin is the best choice for assignment, but I think it is better than the git plugin.

          Mark Waite added a comment - I'm making a wild guess that jglick is better suited to understand this request than I am. When I have jobs that I attempt to kill from job page, sometimes they report that message in the log, and then I click the "Forcibly kill" link in the log. I'm not sure if workflow-durable-task plugin is the best choice for assignment, but I think it is better than the git plugin.

          Jesse Glick added a comment -

          Well WorkflowRun.execution must have been set by this point, else doTerm would not have printed the message it did, though otherwise it looks like the call to FlowDefinition.create is still in progress (hung in GitSCM I would presume). Without a thread dump or a known way to reproduce I am not sure how to diagnose further.

          Jesse Glick added a comment - Well WorkflowRun.execution must have been set by this point, else doTerm would not have printed the message it did, though otherwise it looks like the call to FlowDefinition.create is still in progress (hung in GitSCM I would presume). Without a thread dump or a known way to reproduce I am not sure how to diagnose further.

            Unassigned Unassigned
            piratejohnny Jon B
            Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: