[JENKINS-71692] Pipeline sometimes leaks Execution on heavyweight executors

Type: Bug
Resolution: Fixed
Priority: Minor
Component/s: workflow-cps-plugin
Labels:
None
Environment:
core:2.401
workflow-durable-task:1247.v7f9dfea_b_4fd0

Similar Issues:
Powered by SuggestiMate

Show
Released As:
workflow-cps 3785.vee73da_b_9544e

I have seen instances where pipeline async executions still exist on heavyweight executors although the build has completed. The build may have ended up in SUCCESS or FAILURE, it does not matter. But intermittently, the execution is still kind of orphaned and holding an heavyweight executor.

When running a groovy script dumpExecutors.groovy, the out put shows the following:

LABEL
 Executor #0(0)
  OneOff? false
  Active? true
  Likely Stuck? false
  Progress: 99
  Interrupted? false
  Busy? true
  Elasped Time? 14752068
  Current Work Unit? hudson.model.queue.WorkUnit@1a477ff1[work=part of FOLDER » JOB #11 release 3ce8636615 O20010]
  Causes Of Interruption? []
  Idle Start Milliseconds? 1689274902515
  Asynchronous Execution: org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable$1
  Executable: PlaceholderExecutable:ExecutorStepExecution.PlaceholderTask{runId=FOLDER/JOB#11,label=LABEL,context=CpsStepContext[4:node]:Owner[FOLDER/JOB/11:FOLDER/JOB #11],cookie=e5c90ade-8077-4668-8d86-c1a612d0a8b7,auth=null}
   Executable: PlaceholderExecutable:ExecutorStepExecution.PlaceholderTask{runId=FOLDER/JOB#11,label=LABEL,context=CpsStepContext[4:node]:Owner[FOLDER/JOB/11:FOLDER/JOB #11],cookie=e5c90ade-8077-4668-8d86-c1a612d0a8b7,auth=null}
   Executable (class): class org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable
   Executable (url): job/FOLDER/job/JOB/11/
   Executable (parent): ExecutorStepExecution.PlaceholderTask{runId=FOLDER/JOB#11,label=LABEL,context=CpsStepContext[4:node]:Owner[FOLDER/JOB/11:FOLDER/JOB #11],cookie=e5c90ade-8077-4668-8d86-c1a612d0a8b7,auth=null}
   Executable (parent class): class org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask
   Executable (parent ownerTask): org.jenkinsci.plugins.workflow.job.WorkflowJob@60e140cb[FOLDER/JOB]
   Executable (parent url): job/FOLDER/job/JOB/11/
   Executable (parent runId): FOLDER/JOB#11
   Run Id: FOLDER/JOB #11
   Run URL: job/FOLDER/job/JOB/11/
   Run Result: FAILURE

I recently was able to reproduce the same result (having an orphaned execution that holds an heavyweight executor) with a pipeline like the following:

node {
    stage('Main') {
        outerloop: {
            for (int i = 0; i < 2; i++) {
                if (i > 0) {
                	echo "${i}"
                } else {
                    break outerloop
                }
            }
        }
    }
}

Unclear if this capture the main cause of the problem or just one of them.

In that particular case, the exception shown in Jenkins logs is:

2023-08-29 11:07:53.285+0000 [id=127098]	WARNING	o.j.p.w.cps.CpsVmExecutorService#reportProblem: Unexpected exception in CPS VM thread: CpsFlowExecution[Owner[my-job/1:my-job #1]]
java.lang.IllegalStateException: unexpected break statement
	at com.cloudbees.groovy.cps.impl.CallEnv.getBreakAddress(CallEnv.java:101)
	at com.cloudbees.groovy.cps.impl.ProxyEnv.getBreakAddress(ProxyEnv.java:52)
	at com.cloudbees.groovy.cps.impl.ProxyEnv.getBreakAddress(ProxyEnv.java:52)
	at com.cloudbees.groovy.cps.impl.ProxyEnv.getBreakAddress(ProxyEnv.java:52)
	at com.cloudbees.groovy.cps.impl.ProxyEnv.getBreakAddress(ProxyEnv.java:52)
	at com.cloudbees.groovy.cps.impl.LoopBlockScopeEnv.getBreakAddress(LoopBlockScopeEnv.java:29)
	at com.cloudbees.groovy.cps.impl.ProxyEnv.getBreakAddress(ProxyEnv.java:52)
	at com.cloudbees.groovy.cps.impl.ProxyEnv.getBreakAddress(ProxyEnv.java:52)
	at com.cloudbees.groovy.cps.impl.ProxyEnv.getBreakAddress(ProxyEnv.java:52)
	at com.cloudbees.groovy.cps.impl.BreakBlock.eval(BreakBlock.java:21)
	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:152)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:146)
	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136)
	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:146)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:187)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:422)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:330)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:294)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
	at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

dumpExecutors.groovy
3 kB
2023-07-21 07:53

is duplicated by

JENKINS-70267 Internal pipeline crashes leave stuck executors

Closed

relates to

JENKINS-51568 Pipeline jobs hanging in Build Executor even if it is finished

Open

JENKINS-71617 Labels in implicitly returned closures cause a NullPointerException during Groovy compilation

Open

JENKINS-60507 Pipeline stuck when allocating machine | node block appears to be neither running nor scheduled

Reopened

JENKINS-70267 Internal pipeline crashes leave stuck executors

Closed

links to

jenkinsci/workflow-cps-plugin#780

(1 links to)

Allan BURDAJEWICZ created issue - 2023-07-21 07:53

Allan BURDAJEWICZ made changes - 2023-07-21 07:53

Attachment

New: dumpExecutors.groovy [ 60862 ]

Jesse Glick made changes - 2023-07-21 14:03

Link

New: This issue relates to JENKINS-51568 [ JENKINS-51568 ]

Jesse Glick made changes - 2023-07-21 14:03

Link

New: This issue relates to JENKINS-71617 [ JENKINS-71617 ]

Jesse Glick made changes - 2023-07-21 14:19

Link

New: This issue relates to JENKINS-60507 [ JENKINS-60507 ]

Allan BURDAJEWICZ made changes - 2023-08-29 11:24

Description

Original: I have seen instances where pipeline async executions still exist on heavyweight executors although the build has completed. The build may have ended up in SUCCESS or FAILURE, it does not matter. But intermittently, the execution is still kind of orphaned and holding an heavyweight executor.

When running a groovy script [^dumpExecutors.groovy], the out put shows the following:

{code}
LABEL
Executor #0(0)
  OneOff? false
  Active? true
  Likely Stuck? false
  Progress: 99
  Interrupted? false
  Busy? true
  Elasped Time? 14752068
  Current Work Unit? hudson.model.queue.WorkUnit@1a477ff1[work=part of FOLDER » JOB #11 release 3ce8636615 O20010]
  Causes Of Interruption? []
  Idle Start Milliseconds? 1689274902515
  Asynchronous Execution: org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable$1
  Executable: PlaceholderExecutable:ExecutorStepExecution.PlaceholderTask{runId=FOLDER/JOB#11,label=LABEL,context=CpsStepContext[4:node]:Owner[FOLDER/JOB/11:FOLDER/JOB #11],cookie=e5c90ade-8077-4668-8d86-c1a612d0a8b7,auth=null}
   Executable: PlaceholderExecutable:ExecutorStepExecution.PlaceholderTask{runId=FOLDER/JOB#11,label=LABEL,context=CpsStepContext[4:node]:Owner[FOLDER/JOB/11:FOLDER/JOB #11],cookie=e5c90ade-8077-4668-8d86-c1a612d0a8b7,auth=null}
   Executable (class): class org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable
   Executable (url): job/FOLDER/job/JOB/11/
   Executable (parent): ExecutorStepExecution.PlaceholderTask{runId=FOLDER/JOB#11,label=LABEL,context=CpsStepContext[4:node]:Owner[FOLDER/JOB/11:FOLDER/JOB #11],cookie=e5c90ade-8077-4668-8d86-c1a612d0a8b7,auth=null}
   Executable (parent class): class org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask
   Executable (parent ownerTask): org.jenkinsci.plugins.workflow.job.WorkflowJob@60e140cb[FOLDER/JOB]
   Executable (parent url): job/FOLDER/job/JOB/11/
   Executable (parent runId): FOLDER/JOB#11
   Run Id: FOLDER/JOB #11
   Run URL: job/FOLDER/job/JOB/11/
   Run Result: FAILURE
{code}

I recently was able to reproduce the same result (having an orphaned execution that holds an heavyweight executor) with a pipeline like the following:

{code}
node {
    stage('Main') {
        outerloop: {
            for (int i = 0; i < 2; i++) {
                if (i > 0) {
                 echo "${i}"
                } else {
                    break outerloop
                }
            }
        }
    }
}
{code}

Unclear if this capture the main cause of the problem or just one of them.

New: I have seen instances where pipeline async executions still exist on heavyweight executors although the build has completed. The build may have ended up in SUCCESS or FAILURE, it does not matter. But intermittently, the execution is still kind of orphaned and holding an heavyweight executor.

When running a groovy script [^dumpExecutors.groovy], the out put shows the following:

{code}
LABEL
Executor #0(0)
  OneOff? false
  Active? true
  Likely Stuck? false
  Progress: 99
  Interrupted? false
  Busy? true
  Elasped Time? 14752068
  Current Work Unit? hudson.model.queue.WorkUnit@1a477ff1[work=part of FOLDER » JOB #11 release 3ce8636615 O20010]
  Causes Of Interruption? []
  Idle Start Milliseconds? 1689274902515
  Asynchronous Execution: org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable$1
  Executable: PlaceholderExecutable:ExecutorStepExecution.PlaceholderTask{runId=FOLDER/JOB#11,label=LABEL,context=CpsStepContext[4:node]:Owner[FOLDER/JOB/11:FOLDER/JOB #11],cookie=e5c90ade-8077-4668-8d86-c1a612d0a8b7,auth=null}
   Executable: PlaceholderExecutable:ExecutorStepExecution.PlaceholderTask{runId=FOLDER/JOB#11,label=LABEL,context=CpsStepContext[4:node]:Owner[FOLDER/JOB/11:FOLDER/JOB #11],cookie=e5c90ade-8077-4668-8d86-c1a612d0a8b7,auth=null}
   Executable (class): class org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable
   Executable (url): job/FOLDER/job/JOB/11/
   Executable (parent): ExecutorStepExecution.PlaceholderTask{runId=FOLDER/JOB#11,label=LABEL,context=CpsStepContext[4:node]:Owner[FOLDER/JOB/11:FOLDER/JOB #11],cookie=e5c90ade-8077-4668-8d86-c1a612d0a8b7,auth=null}
   Executable (parent class): class org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask
   Executable (parent ownerTask): org.jenkinsci.plugins.workflow.job.WorkflowJob@60e140cb[FOLDER/JOB]
   Executable (parent url): job/FOLDER/job/JOB/11/
   Executable (parent runId): FOLDER/JOB#11
   Run Id: FOLDER/JOB #11
   Run URL: job/FOLDER/job/JOB/11/
   Run Result: FAILURE
{code}

I recently was able to reproduce the same result (having an orphaned execution that holds an heavyweight executor) with a pipeline like the following:

{code}
node {
    stage('Main') {
        outerloop: {
            for (int i = 0; i < 2; i++) {
                if (i > 0) {
                 echo "${i}"
                } else {
                    break outerloop
                }
            }
        }
    }
}
{code}

Unclear if this capture the main cause of the problem or just one of them.

In that particular case, the exception shown in Jenkins logs is:

{code}
2023-08-29 11:07:53.285+0000 [id=127098] WARNING o.j.p.w.cps.CpsVmExecutorService#reportProblem: Unexpected exception in CPS VM thread: CpsFlowExecution[Owner[my-job/1:my-job #1]]
java.lang.IllegalStateException: unexpected break statement
at com.cloudbees.groovy.cps.impl.CallEnv.getBreakAddress(CallEnv.java:101)
at com.cloudbees.groovy.cps.impl.ProxyEnv.getBreakAddress(ProxyEnv.java:52)
at com.cloudbees.groovy.cps.impl.ProxyEnv.getBreakAddress(ProxyEnv.java:52)
at com.cloudbees.groovy.cps.impl.ProxyEnv.getBreakAddress(ProxyEnv.java:52)
at com.cloudbees.groovy.cps.impl.ProxyEnv.getBreakAddress(ProxyEnv.java:52)
at com.cloudbees.groovy.cps.impl.LoopBlockScopeEnv.getBreakAddress(LoopBlockScopeEnv.java:29)
at com.cloudbees.groovy.cps.impl.ProxyEnv.getBreakAddress(ProxyEnv.java:52)
at com.cloudbees.groovy.cps.impl.ProxyEnv.getBreakAddress(ProxyEnv.java:52)
at com.cloudbees.groovy.cps.impl.ProxyEnv.getBreakAddress(ProxyEnv.java:52)
at com.cloudbees.groovy.cps.impl.BreakBlock.eval(BreakBlock.java:21)
at com.cloudbees.groovy.cps.Next.step(Next.java:83)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:152)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:146)
at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136)
at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:146)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:187)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:422)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:330)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:294)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
{code}

Devin Nusbaum made changes - 2023-08-31 18:44

Remote Link

New: This issue links to "jenkinsci/workflow-cps-plugin#780 (Web Link)" [ 28919 ]

Devin Nusbaum made changes - 2023-08-31 20:25

Link

New: This issue relates to ~~JENKINS-70267~~ [ ~~JENKINS-70267~~ ]

Devin Nusbaum made changes - 2023-08-31 21:57

Link

New: This issue is duplicated by ~~JENKINS-70267~~ [ ~~JENKINS-70267~~ ]

Devin Nusbaum made changes - 2023-08-31 22:01

Assignee

New: Devin Nusbaum [ dnusbaum ]

Devin Nusbaum made changes - 2023-08-31 22:01

Status

Original: Open [ 1 ]

New: In Progress [ 3 ]

Assignee:: Devin Nusbaum

Reporter:: Allan BURDAJEWICZ

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Created:: 2023-07-21 07:53

Updated:: 2023-09-05 21:26

Resolved:: 2023-09-05 21:26

Jenkins

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates