Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-28183

Hard killed job's stage blocks stage in following jobs

      Build #480 of my job hung in Stage 'Imaging' and I had to do a hard kill using BUILD_URL/doDelete

      Now all builds of my job hang on:

      Running: Imaging
      Entering stage Imaging
      Waiting for builds [480]

      Restarting Jenkins does not help.

          [JENKINS-28183] Hard killed job's stage blocks stage in following jobs

          Anshu Arya created issue -
          Anshu Arya made changes -
          Description Original: Build #480 of my job hung in Stage 'Imaging' and I had to do a hard kill using BUILD_URL/doDelete

          Now all builds of my job hang on:

          Running: Imaging
          Entering stage Imaging
          Waiting for builds [480]
          New: Build #480 of my job hung in Stage 'Imaging' and I had to do a hard kill using BUILD_URL/doDelete

          Now all builds of my job hang on:

          Running: Imaging
          Entering stage Imaging
          Waiting for builds [480]

          Restarting Jenkins does not help.
          Jesse Glick made changes -
          Link New: This issue is blocking JENKINS-25550 [ JENKINS-25550 ]

          Jesse Glick added a comment -

          Hmm, you might need to edit $JENKINS_HOME/org.jenkinsci.plugins.workflow.support.steps.StageStep.xml as a last resort.

          I am surprised at this report since there is a cleanUp function specifically to remove entries corresponding to builds that disappeared somehow without a clean exit. For some reason it is not getting called here, or is failing to detect the deleted build. It should have printed a warning to your log at some point saying Cleaning up apparently deleted job-name#480. I wonder if your job was renamed/moved at any point, or anything else unusual occurred?

          Jesse Glick added a comment - Hmm, you might need to edit $JENKINS_HOME/org.jenkinsci.plugins.workflow.support.steps.StageStep.xml as a last resort. I am surprised at this report since there is a cleanUp function specifically to remove entries corresponding to builds that disappeared somehow without a clean exit. For some reason it is not getting called here, or is failing to detect the deleted build. It should have printed a warning to your log at some point saying Cleaning up apparently deleted job-name#480 . I wonder if your job was renamed/moved at any point, or anything else unusual occurred?

          kishorerp added a comment -

          Can we please have a timeline by which this issue would be fixed.
          We cannot afford end users or anyone for that matter to manually kill a job by doing some changes to files in jenkins server.
          Context?
          https://groups.google.com/forum/#!topic/jenkinsci-dev/k4VLUAVfFjA

          kishorerp added a comment - Can we please have a timeline by which this issue would be fixed. We cannot afford end users or anyone for that matter to manually kill a job by doing some changes to files in jenkins server. Context? https://groups.google.com/forum/#!topic/jenkinsci-dev/k4VLUAVfFjA
          James Nord made changes -
          Link New: This issue is related to JENKINS-29754 [ JENKINS-29754 ]
          Jesse Glick made changes -
          Link New: This issue is duplicated by JENKINS-29754 [ JENKINS-29754 ]
          Jesse Glick made changes -
          Link Original: This issue is related to JENKINS-29754 [ JENKINS-29754 ]

          Jesse Glick added a comment -

          There are many bugs and it is impossible to give a timeline for all of them. If a particular bug is important to you, you can try to fix it yourself, or if you are a CloudBees customer you can request a fix via a support ticket.

          Jesse Glick added a comment - There are many bugs and it is impossible to give a timeline for all of them. If a particular bug is important to you, you can try to fix it yourself, or if you are a CloudBees customer you can request a fix via a support ticket.

          jglick I hit the same issue by aborting a build (abort, then clicking the link at the end of the console). After the weekend I noticed that a new build got stuck waiting on an aborted build. Searching for Cleaning up apparently deleted showed no results (in jenkins.err.log)

          In the log I found: Build 259 is the most recent build, triggered after the a restart(LTS patch update, should be unrelated), still waiting. Build 258 hung waiting on 256 for two days (I only noticed that after the fact because waiting builds are only shown on the jobs page and not on /). The logs below document my cancelling of 258, not sure they are helpful.

          Mär 21, 2016 11:04:22 AM org.jenkinsci.plugins.workflow.support.steps.StageStepExecution cancel
          WARNING: cannot cancel dead CpsStepContext[223]:Owner[jobname/master/258:null] or CpsStepContext[223]:Ownerjobname/master/259:jobname/master #259
          Mär 21, 2016 11:08:52 AM org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$6 onSuccess
          WARNING: Failed to abort CpsFlowExecutionOwner[jobname/master/258:jobname/master #258]
          java.lang.UnsupportedOperationException
          at org.jenkinsci.plugins.workflow.support.steps.StageStepExecution.stop(StageStepExecution.java:66)
          at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$6.onSuccess(CpsFlowExecution.java:760)
          at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$6.onSuccess(CpsFlowExecution.java:755)
          at org.jenkinsci.plugins.workflow.support.concurrent.Futures$1.run(Futures.java:150)
          at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
          at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
          at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:105)
          at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:155)
          at org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:160)
          at org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:90)
          at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.interrupt(CpsFlowExecution.java:755)
          at org.jenkinsci.plugins.workflow.job.WorkflowRun$2.interrupt(WorkflowRun.java:230)
          at hudson.model.Executor.interrupt(Executor.java:227)
          at hudson.model.Executor.interrupt(Executor.java:197)
          at hudson.model.Executor.interrupt(Executor.java:187)
          at hudson.model.Executor.interrupt(Executor.java:173)
          at hudson.model.Executor.doStop(Executor.java:867)
          at org.jenkinsci.plugins.workflow.job.WorkflowRun.doStop(WorkflowRun.java:629)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
          at java.lang.reflect.Method.invoke(Unknown Source)
          at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
          at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:46)
          at org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:399)
          at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
          at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
          at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121)
          at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          at org.kohsuke.stapler.MetaClass$13.dispatch(MetaClass.java:411)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
          at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
          at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
          at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
          at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
          at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:129)
          at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:49)
          at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:44)
          at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:106)
          at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:44)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:129)
          at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:129)
          at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:123)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
          at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
          at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
          at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
          at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
          at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
          at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
          at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
          at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
          at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
          at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
          at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
          at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
          at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
          at org.eclipse.jetty.server.Server.handle(Server.java:370)
          at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
          at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
          at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
          at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
          at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
          at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
          at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
          at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
          at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at java.lang.Thread.run(Unknown Source)

          Mär 21, 2016 11:09:09 AM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
          INFO: jobname/master #258 completed: ABORTED

          Ing. Christoph Obexer added a comment - jglick I hit the same issue by aborting a build (abort, then clicking the link at the end of the console). After the weekend I noticed that a new build got stuck waiting on an aborted build. Searching for Cleaning up apparently deleted showed no results (in jenkins.err.log) In the log I found: Build 259 is the most recent build, triggered after the a restart(LTS patch update, should be unrelated), still waiting. Build 258 hung waiting on 256 for two days (I only noticed that after the fact because waiting builds are only shown on the jobs page and not on /). The logs below document my cancelling of 258, not sure they are helpful. Mär 21, 2016 11:04:22 AM org.jenkinsci.plugins.workflow.support.steps.StageStepExecution cancel WARNING: cannot cancel dead CpsStepContext [223] :Owner [jobname/master/258:null] or CpsStepContext [223] :Owner jobname/master/259:jobname/master #259 Mär 21, 2016 11:08:52 AM org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$6 onSuccess WARNING: Failed to abort CpsFlowExecution Owner[jobname/master/258:jobname/master #258] java.lang.UnsupportedOperationException at org.jenkinsci.plugins.workflow.support.steps.StageStepExecution.stop(StageStepExecution.java:66) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$6.onSuccess(CpsFlowExecution.java:760) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$6.onSuccess(CpsFlowExecution.java:755) at org.jenkinsci.plugins.workflow.support.concurrent.Futures$1.run(Futures.java:150) at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253) at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149) at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:105) at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:155) at org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:160) at org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:90) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.interrupt(CpsFlowExecution.java:755) at org.jenkinsci.plugins.workflow.job.WorkflowRun$2.interrupt(WorkflowRun.java:230) at hudson.model.Executor.interrupt(Executor.java:227) at hudson.model.Executor.interrupt(Executor.java:197) at hudson.model.Executor.interrupt(Executor.java:187) at hudson.model.Executor.interrupt(Executor.java:173) at hudson.model.Executor.doStop(Executor.java:867) at org.jenkinsci.plugins.workflow.job.WorkflowRun.doStop(WorkflowRun.java:629) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298) at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:46) at org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:399) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$13.dispatch(MetaClass.java:411) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:129) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:49) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:44) at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:106) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:44) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:129) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:129) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:123) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Mär 21, 2016 11:09:09 AM org.jenkinsci.plugins.workflow.job.WorkflowRun finish INFO: jobname/master #258 completed: ABORTED

            jglick Jesse Glick
            anshuarya Anshu Arya
            Votes:
            7 Vote for this issue
            Watchers:
            16 Start watching this issue

              Created:
              Updated:
              Resolved: