Hello,

I finally reproduced the issue with pipeline stuck on our instance.

how to reproduce it:

1. create pipeline with build trigger step
2. run pipeline
3. kill/restart jenkins during execution of job which was triggered by the pipeline job (execution has to be started). The goal is to interrupt it without calling listener about stopping/failing execution (which usually can happen by restart or falling jenkins instance)
4. after start of jenkins, there is pipeline job endlessly waiting for finishing the execution of triggered job and calling from listener about completion of the job execution.

 

If I understand correctly the issue happens due this code:

when execution of the step is called the resume value of execution thread is set on null

https://github.com/jenkinsci/workflow-cps-plugin/blob/workflow-cps-2.49/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java#L173

the thread is not runnable until resume value is non null.

https://github.com/jenkinsci/workflow-cps-plugin/blob/workflow-cps-2.49/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java#L138

until isRunnable returns true, thread group do not continue execution of this thread (and pipeline script) and expects that thread is waiting for completion of something (which in aforementioned case will never happen).

https://github.com/jenkinsci/workflow-cps-plugin/blob/workflow-cps-2.49/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java#L331

 

In case of build step trigger, the listener should fire that everything is completed and set it in context which causes that resume value is no longer null and new run of thread group is scheduled.

https://github.com/jenkinsci/pipeline-build-step-plugin/blob/pipeline-build-step-2.7/src/main/java/org/jenkinsci/plugins/workflow/support/steps/build/BuildTriggerListener.java#L47

https://github.com/jenkinsci/workflow-cps-plugin/blob/workflow-cps-2.49/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsStepContext.java#L333

https://github.com/jenkinsci/workflow-cps-plugin/blob/workflow-cps-2.49/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsStepContext.java#L340

https://github.com/jenkinsci/workflow-cps-plugin/blob/workflow-cps-2.49/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsStepContext.java#L443

https://github.com/jenkinsci/workflow-cps-plugin/blob/workflow-cps-2.49/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java#L261

 

If I understand it correctly, currently everything is dependable on calling listener which does not have to be called by build in some unexpected situation or bug. It is not mistake in pipeline but I think that pipeline (or at least build trigger step) should count with such situation and handle it better. Because when the pipeline is in such state, this state is saved and no matter how many times the Jenkins is restarted it hangs forever until someone interrupt it form UI. 

I thought about two solutions:

build trigger solution - periodically check if job is still executing (no very nice solution)

pipeline solution - extends api from something like - stillNotCompleted method, where it will periodically ask the build step if we are still waiting for some completion. 

 

What do you think about it? Do you have any better idea how to solve it?