Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45861

SCM pipelines get stuck and go into a zombie unkillable state

XMLWordPrintable

      I am running Jenkins ver. 2.60.1 with git client 2.4.6 and git plugin 3.4.1

      I am running a pipeline based on a Jenkinsfile from SCM.

      I have noticed that sometimes, my pipeline seems stuck.

      I go to the "/restart" page and initiate a restart. Later, I still see some of the pipelines in a stuck state. For example, in the console, I see:

      Started by user myghmachineuser
      Checking out git ${GITHUB_REPOSITORY} into /var/jenkins_home/workspace/redacted@script to read Jenkinsfile-nodeploy.groovy
       > git rev-parse --is-inside-work-tree # timeout=10
      Fetching changes from the remote Git repository
       > git config remote.origin.url git@github.com:redacted/redacted.git # timeout=10
      Fetching upstream changes from git@github.com:redacted/redacted.git
       > git --version # timeout=10
      using GIT_SSH to set credentials 
       > git fetch --tags --progress git@github.com:redacted/redacted.git +refs/heads/*:refs/remotes/origin/*
       > git rev-parse 3f96f044eaec0fabf966d702fe727560bcbf9dbf^{commit} # timeout=10
      Checking out Revision 3f96f044eaec0fabf966d702fe727560bcbf9dbf (detached)
      Commit message: "revert service-lib"
       > git config core.sparsecheckout # timeout=10
       > git checkout -f 3f96f044eaec0fabf966d702fe727560bcbf9dbf
       > git rev-list 7f3be5084de22472b8c2a9de339f588a6427915f # timeout=10
      Click here to forcibly kill entire build
      Click here to forcibly kill entire build
      Click here to forcibly kill entire build

      When I click on the forcibly kill links, nothing happens. This is a great example of a pipeline run that seems to be a zombie that will not go away.

      I looked in the Jenkins master logs and I see:

      Jul 28, 2017 3:09:46 PM WARNING org.jenkinsci.plugins.workflow.job.WorkflowRun finish
      doorstep-django-nodeploy #9037 failed to start
      org.jenkinsci.plugins.workflow.steps.FlowInterruptedException
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.doKill(WorkflowRun.java:417)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.doStop(WorkflowRun.java:804)
      	at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
      	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343)
      	at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:52)
      	at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
      	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184)
      	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117)
      	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:129)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
      	at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
      	at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:248)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
      	at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
      	at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
      	at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:51)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
      	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:237)
      	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:209)
      	at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
      	at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:113)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
      	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:138)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:92)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.server.Server.handle(Server.java:499)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
      	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
      	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:748)

       

            Unassigned Unassigned
            piratejohnny Jon B
            Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: