Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43413

RejectedExecutionException at SingleLaneExecutorService when loading CpsFlowExecution after restart

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • workflow-cps-plugin

      One one of the Jenkins instances I see the following behavior after the Jenkins restart:

      2017-04-05 10:56:30.569+0000 [id=72]	WARNING	o.j.p.w.f.FlowExecutionList$ItemListenerImpl$1#onFailure: Failed to load CpsFlowExecution[Owner[ *SECRET* #15]]java.util.concurrent.RejectedExecutionException
      	at hudson.remoting.SingleLaneExecutorService.execute(SingleLaneExecutorService.java:100)
      	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
      	at com.google.common.util.concurrent.ForwardingExecutorService.submit(ForwardingExecutorService.java:110)
      	at jenkins.util.InterceptingExecutorService.submit(InterceptingExecutorService.java:49)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4.onSuccess(CpsFlowExecution.java:648)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4.onSuccess(CpsFlowExecution.java:644)
      	at org.jenkinsci.plugins.workflow.support.concurrent.Futures$1.run(Futures.java:150)
      

      According to the remoting Code-base, it happens when the Executor Service is shutting down due to whatever reason. I am not sure why it could be shutting down in Pipeline CPS Plugin during the startup, but this issue should be gracefully handled.

      Diagnostics PR on the remoting side: https://github.com/jenkinsci/remoting/pull/156.

          [JENKINS-43413] RejectedExecutionException at SingleLaneExecutorService when loading CpsFlowExecution after restart

          Jesse Glick added a comment -

          Nothing to do with Remoting really.

          Saw something like this in 2.46.1 with up-to-date plugins after running thousands of builds of a job discarding 50 old builds:

          echo 'OK'
          build job: JOB_NAME, wait: false
          

          After /exit and restarting, there were a bunch of running builds, all hung. When trying to manually interrupt one, get

          …org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$6 onFailure
          WARNING: Failed to interrupt steps in Owner[p/9569:p #9569]
          java.util.concurrent.RejectedExecutionException
          	at hudson.remoting.SingleLaneExecutorService.execute(SingleLaneExecutorService.java:100)
          	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
          	at com.google.common.util.concurrent.ForwardingExecutorService.submit(ForwardingExecutorService.java:110)
          	at jenkins.util.InterceptingExecutorService.submit(InterceptingExecutorService.java:49)
          	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4.onSuccess(CpsFlowExecution.java:655)
          	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4.onSuccess(CpsFlowExecution.java:651)
          	at org.jenkinsci.plugins.workflow.support.concurrent.Futures$1.run(Futures.java:150)
          	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
          	at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
          	at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:105)
          	at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:155)
          	at org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:160)
          	at org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:90)
          	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.runInCpsVmThread(CpsFlowExecution.java:651)
          	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.interrupt(CpsFlowExecution.java:861)
          	at org.jenkinsci.plugins.workflow.job.WorkflowRun$2$1.run(WorkflowRun.java:276)
          	at …
          

          The weird thing is the logs:

          …
          [Pipeline] End of Pipeline
          Finished: SUCCESS
          Resuming build at … after Jenkins restart
          

          In other words, they should have been complete already (there was only ever supposed to be one build of this project running at a time), so why were they marked as in progress?

          Not sure if it is reproducible.

          Jesse Glick added a comment - Nothing to do with Remoting really. Saw something like this in 2.46.1 with up-to-date plugins after running thousands of builds of a job discarding 50 old builds: echo 'OK' build job: JOB_NAME, wait: false After /exit and restarting, there were a bunch of running builds, all hung. When trying to manually interrupt one, get …org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$6 onFailure WARNING: Failed to interrupt steps in Owner[p/9569:p #9569] java.util.concurrent.RejectedExecutionException at hudson.remoting.SingleLaneExecutorService.execute(SingleLaneExecutorService.java:100) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at com.google.common.util.concurrent.ForwardingExecutorService.submit(ForwardingExecutorService.java:110) at jenkins.util.InterceptingExecutorService.submit(InterceptingExecutorService.java:49) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4.onSuccess(CpsFlowExecution.java:655) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4.onSuccess(CpsFlowExecution.java:651) at org.jenkinsci.plugins.workflow.support.concurrent.Futures$1.run(Futures.java:150) at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253) at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149) at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:105) at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:155) at org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:160) at org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:90) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.runInCpsVmThread(CpsFlowExecution.java:651) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.interrupt(CpsFlowExecution.java:861) at org.jenkinsci.plugins.workflow.job.WorkflowRun$2$1.run(WorkflowRun.java:276) at … The weird thing is the logs: … [Pipeline] End of Pipeline Finished: SUCCESS Resuming build at … after Jenkins restart In other words, they should have been complete already (there was only ever supposed to be one build of this project running at a time), so why were they marked as in progress? Not sure if it is reproducible.

          Patrick Rose added a comment -

          Same thing here:

          [Pipeline] End of Pipeline
          java.util.concurrent.RejectedExecutionException: Task com.google.common.eventbus.AsyncEventBus$1@3119aa0c rejected from java.util.concurrent.ThreadPoolExecutor@51dba0f7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 11758]
          	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
          	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
          	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
          	at com.google.common.eventbus.AsyncEventBus.dispatch(AsyncEventBus.java:90)
          	at com.google.common.eventbus.AsyncEventBus.dispatchQueuedEvents(AsyncEventBus.java:81)
          	at com.google.common.eventbus.EventBus.post(EventBus.java:264)
          	at org.jenkinsci.plugins.pubsub.GuavaPubsubBus$1.publish(GuavaPubsubBus.java:70)
          	at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:141)
          	at io.jenkins.blueocean.events.PipelineEventListener.publishEvent(PipelineEventListener.java:214)
          	at io.jenkins.blueocean.events.PipelineEventListener.onNewHead(PipelineEventListener.java:77)
          	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1221)
          	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:407)
          	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
          	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
          	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          	at java.lang.Thread.run(Thread.java:745)
          Finished: FAILURE
          Resuming build at Fri Sep 15 17:55:27 BST 2017 after Jenkins restart

          Usual trick of pausing/resuming doesn't seem to be working because the build has finished but Jenkins doesn't seem to know that.

          Patrick Rose added a comment - Same thing here: [Pipeline] End of Pipeline java.util.concurrent.RejectedExecutionException: Task com.google.common.eventbus.AsyncEventBus$1@3119aa0c rejected from java.util.concurrent.ThreadPoolExecutor@51dba0f7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 11758] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) at com.google.common.eventbus.AsyncEventBus.dispatch(AsyncEventBus.java:90) at com.google.common.eventbus.AsyncEventBus.dispatchQueuedEvents(AsyncEventBus.java:81) at com.google.common.eventbus.EventBus.post(EventBus.java:264) at org.jenkinsci.plugins.pubsub.GuavaPubsubBus$1.publish(GuavaPubsubBus.java:70) at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:141) at io.jenkins.blueocean.events.PipelineEventListener.publishEvent(PipelineEventListener.java:214) at io.jenkins.blueocean.events.PipelineEventListener.onNewHead(PipelineEventListener.java:77) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1221) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:407) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) Finished: FAILURE Resuming build at Fri Sep 15 17:55:27 BST 2017 after Jenkins restart Usual trick of pausing/resuming doesn't seem to be working because the build has finished but Jenkins doesn't seem to know that.

          We have the same issue on Jenkins 2.89.4 and workflow-cps-plugin 2.48. Jobs starting after Jenkins reboot even if we cleaning up workspace directory on Master.

          In threadDump no mentions about these jobs. Is there a way to stop these phantom jobs?

          Denis Koryavov added a comment - We have the same issue on Jenkins 2.89.4 and workflow-cps-plugin 2.48. Jobs starting after Jenkins reboot even if we cleaning up workspace directory on Master. In threadDump no mentions about these jobs. Is there a way to stop these phantom jobs?

          Sam Van Oort added a comment -

          oleg_nenashev This should be resolved by the workflow-cps 2.50 and workflow-job 2.21 releases

          Sam Van Oort added a comment - oleg_nenashev This should be resolved by the workflow-cps 2.50 and workflow-job 2.21 releases

          Yo-An Lin added a comment -

          This is still happening in jenkins 2.150.1 with all updated plugin. (The time 2018-12-7)

          Yo-An Lin added a comment - This is still happening in jenkins 2.150.1 with all updated plugin. (The time 2018-12-7)

          Jesse Glick added a comment -

          Unclear if this specific bug was definitively tracked down and covered by the round of fixes.

          Jesse Glick added a comment - Unclear if this specific bug was definitively tracked down and covered by the round of fixes.

            Unassigned Unassigned
            oleg_nenashev Oleg Nenashev
            Votes:
            2 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: