[JENKINS-43413] RejectedExecutionException at SingleLaneExecutorService when loading CpsFlowExecution after restart

Type: Bug
Resolution: Unresolved
Priority: Major
Component/s: workflow-cps-plugin
Labels:
- maybe-fixed-by-durability-megafix
- regression
Environment:

Hide
* Jenkins 2.32.2
* workflow-aggregator:2.4 'Pipeline'
  * workflow-api:2.8 'Pipeline: API'
  * workflow-basic-steps:2.3 'Pipeline: Basic Steps'
  * workflow-cps:2.24 'Pipeline: Groovy'
  * workflow-cps-checkpoint:2.4 'CloudBees Pipeline: Groovy Checkpoint Plugin'
  * workflow-cps-global-lib:2.5 'Pipeline: Shared Groovy Libraries'
  * workflow-durable-task-step:2.8 'Pipeline: Nodes and Processes'
  * workflow-job:2.9 'Pipeline: Job'
  * workflow-multibranch:2.9.2 'Pipeline: Multibranch'
  * workflow-scm-step:2.3 'Pipeline: SCM Step'
  * workflow-step-api:2.7 'Pipeline: Step API'
  * workflow-support:2.12 'Pipeline: Supporting APIs'

Show
* Jenkins 2.32.2 * workflow-aggregator:2.4 'Pipeline'   * workflow-api:2.8 'Pipeline: API'   * workflow-basic-steps:2.3 'Pipeline: Basic Steps'   * workflow-cps:2.24 'Pipeline: Groovy'   * workflow-cps-checkpoint:2.4 'CloudBees Pipeline: Groovy Checkpoint Plugin'   * workflow-cps-global-lib:2.5 'Pipeline: Shared Groovy Libraries'   * workflow-durable-task-step:2.8 'Pipeline: Nodes and Processes'   * workflow-job:2.9 'Pipeline: Job'   * workflow-multibranch:2.9.2 'Pipeline: Multibranch'   * workflow-scm-step:2.3 'Pipeline: SCM Step'   * workflow-step-api:2.7 'Pipeline: Step API'   * workflow-support:2.12 'Pipeline: Supporting APIs'

Similar Issues:
Powered by SuggestiMate

Show

One one of the Jenkins instances I see the following behavior after the Jenkins restart:

2017-04-05 10:56:30.569+0000 [id=72]	WARNING	o.j.p.w.f.FlowExecutionList$ItemListenerImpl$1#onFailure: Failed to load CpsFlowExecution[Owner[ *SECRET* #15]]java.util.concurrent.RejectedExecutionException
	at hudson.remoting.SingleLaneExecutorService.execute(SingleLaneExecutorService.java:100)
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
	at com.google.common.util.concurrent.ForwardingExecutorService.submit(ForwardingExecutorService.java:110)
	at jenkins.util.InterceptingExecutorService.submit(InterceptingExecutorService.java:49)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4.onSuccess(CpsFlowExecution.java:648)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4.onSuccess(CpsFlowExecution.java:644)
	at org.jenkinsci.plugins.workflow.support.concurrent.Futures$1.run(Futures.java:150)

According to the remoting Code-base, it happens when the Executor Service is shutting down due to whatever reason. I am not sure why it could be shutting down in Pipeline CPS Plugin during the startup, but this issue should be gracefully handled.

Diagnostics PR on the remoting side: https://github.com/jenkinsci/remoting/pull/156.

links to

CloudBees Internal CD-28

Oleg Nenashev created issue - 2017-04-07 06:53

Jesse Glick added a comment - 2017-04-10 16:45

Nothing to do with Remoting really.

Saw something like this in 2.46.1 with up-to-date plugins after running thousands of builds of a job discarding 50 old builds:

echo 'OK'
build job: JOB_NAME, wait: false

After /exit and restarting, there were a bunch of running builds, all hung. When trying to manually interrupt one, get

…org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$6 onFailure
WARNING: Failed to interrupt steps in Owner[p/9569:p #9569]
java.util.concurrent.RejectedExecutionException
	at hudson.remoting.SingleLaneExecutorService.execute(SingleLaneExecutorService.java:100)
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
	at com.google.common.util.concurrent.ForwardingExecutorService.submit(ForwardingExecutorService.java:110)
	at jenkins.util.InterceptingExecutorService.submit(InterceptingExecutorService.java:49)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4.onSuccess(CpsFlowExecution.java:655)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4.onSuccess(CpsFlowExecution.java:651)
	at org.jenkinsci.plugins.workflow.support.concurrent.Futures$1.run(Futures.java:150)
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
	at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
	at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:105)
	at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:155)
	at org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:160)
	at org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:90)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.runInCpsVmThread(CpsFlowExecution.java:651)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.interrupt(CpsFlowExecution.java:861)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$2$1.run(WorkflowRun.java:276)
	at …

The weird thing is the logs:

…
[Pipeline] End of Pipeline
Finished: SUCCESS
Resuming build at … after Jenkins restart

In other words, they should have been complete already (there was only ever supposed to be one build of this project running at a time), so why were they marked as in progress?

Not sure if it is reproducible.

Jesse Glick added a comment - 2017-04-10 16:45 Nothing to do with Remoting really. Saw something like this in 2.46.1 with up-to-date plugins after running thousands of builds of a job discarding 50 old builds: echo 'OK' build job: JOB_NAME, wait: false After /exit and restarting, there were a bunch of running builds, all hung. When trying to manually interrupt one, get …org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$6 onFailure WARNING: Failed to interrupt steps in Owner[p/9569:p #9569] java.util.concurrent.RejectedExecutionException at hudson.remoting.SingleLaneExecutorService.execute(SingleLaneExecutorService.java:100) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at com.google.common.util.concurrent.ForwardingExecutorService.submit(ForwardingExecutorService.java:110) at jenkins.util.InterceptingExecutorService.submit(InterceptingExecutorService.java:49) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4.onSuccess(CpsFlowExecution.java:655) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4.onSuccess(CpsFlowExecution.java:651) at org.jenkinsci.plugins.workflow.support.concurrent.Futures$1.run(Futures.java:150) at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253) at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149) at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:105) at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:155) at org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:160) at org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:90) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.runInCpsVmThread(CpsFlowExecution.java:651) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.interrupt(CpsFlowExecution.java:861) at org.jenkinsci.plugins.workflow.job.WorkflowRun$2$1.run(WorkflowRun.java:276) at … The weird thing is the logs: … [Pipeline] End of Pipeline Finished: SUCCESS Resuming build at … after Jenkins restart In other words, they should have been complete already (there was only ever supposed to be one build of this project running at a time), so why were they marked as in progress? Not sure if it is reproducible.

Jesse Glick made changes - 2017-04-10 16:45

Component/s	Original: pipeline [ 21692 ]
Labels	Original: pipeline remoting

Patrick Rose added a comment - 2017-09-18 10:16

Same thing here:

[Pipeline] End of Pipeline
java.util.concurrent.RejectedExecutionException: Task com.google.common.eventbus.AsyncEventBus$1@3119aa0c rejected from java.util.concurrent.ThreadPoolExecutor@51dba0f7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 11758]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
	at com.google.common.eventbus.AsyncEventBus.dispatch(AsyncEventBus.java:90)
	at com.google.common.eventbus.AsyncEventBus.dispatchQueuedEvents(AsyncEventBus.java:81)
	at com.google.common.eventbus.EventBus.post(EventBus.java:264)
	at org.jenkinsci.plugins.pubsub.GuavaPubsubBus$1.publish(GuavaPubsubBus.java:70)
	at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:141)
	at io.jenkins.blueocean.events.PipelineEventListener.publishEvent(PipelineEventListener.java:214)
	at io.jenkins.blueocean.events.PipelineEventListener.onNewHead(PipelineEventListener.java:77)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1221)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:407)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Finished: FAILURE
Resuming build at Fri Sep 15 17:55:27 BST 2017 after Jenkins restart

Usual trick of pausing/resuming doesn't seem to be working because the build has finished but Jenkins doesn't seem to know that.

Patrick Rose added a comment - 2017-09-18 10:16 Same thing here: [Pipeline] End of Pipeline java.util.concurrent.RejectedExecutionException: Task com.google.common.eventbus.AsyncEventBus$1@3119aa0c rejected from java.util.concurrent.ThreadPoolExecutor@51dba0f7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 11758] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) at com.google.common.eventbus.AsyncEventBus.dispatch(AsyncEventBus.java:90) at com.google.common.eventbus.AsyncEventBus.dispatchQueuedEvents(AsyncEventBus.java:81) at com.google.common.eventbus.EventBus.post(EventBus.java:264) at org.jenkinsci.plugins.pubsub.GuavaPubsubBus$1.publish(GuavaPubsubBus.java:70) at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:141) at io.jenkins.blueocean.events.PipelineEventListener.publishEvent(PipelineEventListener.java:214) at io.jenkins.blueocean.events.PipelineEventListener.onNewHead(PipelineEventListener.java:77) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1221) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:407) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) Finished: FAILURE Resuming build at Fri Sep 15 17:55:27 BST 2017 after Jenkins restart Usual trick of pausing/resuming doesn't seem to be working because the build has finished but Jenkins doesn't seem to know that.

CloudBees Inc. made changes - 2017-12-05 08:22

Remote Link

New: This issue links to "CloudBees Internal CD-28 (Web Link)" [ 19039 ]

Denis Koryavov added a comment - 2018-04-25 10:32

We have the same issue on Jenkins 2.89.4 and workflow-cps-plugin 2.48. Jobs starting after Jenkins reboot even if we cleaning up workspace directory on Master.

In threadDump no mentions about these jobs. Is there a way to stop these phantom jobs?

Denis Koryavov added a comment - 2018-04-25 10:32 We have the same issue on Jenkins 2.89.4 and workflow-cps-plugin 2.48. Jobs starting after Jenkins reboot even if we cleaning up workspace directory on Master. In threadDump no mentions about these jobs. Is there a way to stop these phantom jobs?

Sam Van Oort made changes - 2018-05-02 21:45

Labels

New: maybe-fixed-by-durability-megafix regression

Sam Van Oort added a comment - 2018-05-02 21:45

oleg_nenashev This should be resolved by the workflow-cps 2.50 and workflow-job 2.21 releases

Sam Van Oort added a comment - 2018-05-02 21:45 oleg_nenashev This should be resolved by the workflow-cps 2.50 and workflow-job 2.21 releases

Sam Van Oort made changes - 2018-05-02 21:45

Resolution		New: Fixed [ 1 ]
Status	Original: Open [ 1 ]	New: Closed [ 6 ]

Yo-An Lin added a comment - 2018-12-12 21:15

This is still happening in jenkins 2.150.1 with all updated plugin. (The time 2018-12-7)

Yo-An Lin added a comment - 2018-12-12 21:15 This is still happening in jenkins 2.150.1 with all updated plugin. (The time 2018-12-7)

Assignee:: Unassigned

Reporter:: Oleg Nenashev

Votes:: 2 Vote for this issue

Watchers:: 8 Start watching this issue

Created:: 2017-04-07 06:53

Updated:: 2019-01-11 23:04

Jenkins

Details

Description

Attachments

Issue Links

Activity

Collapse comment: Jesse Glick added a comment - 2017-04-10 16:45

Expand comment: Jesse Glick added a comment - 2017-04-10 16:45

Collapse comment: Patrick Rose added a comment - 2017-09-18 10:16

Expand comment: Patrick Rose added a comment - 2017-09-18 10:16

Collapse comment: Denis Koryavov added a comment - 2018-04-25 10:32

Expand comment: Denis Koryavov added a comment - 2018-04-25 10:32

Collapse comment: Sam Van Oort added a comment - 2018-05-02 21:45

Expand comment: Sam Van Oort added a comment - 2018-05-02 21:45

Collapse comment: Yo-An Lin added a comment - 2018-12-12 21:15

Expand comment: Yo-An Lin added a comment - 2018-12-12 21:15

People

Dates