-
Bug
-
Resolution: Unresolved
-
Trivial
I doubt this one is reproductible.
Going to yourjiraurlhere/computer/api/json?pretty=true&tree=computer[oneOffExecutors[likelyStuck,currentExecutable[result,url]]]{0}
gives you the jobs currently running.
One of my job is marked as "likely stuck", but his state is result is "SUCCESS" (and has been "SUCCESS" since 2h30, making me doubt about the veracity of the "likely stuck".
The job isn't running either. It's completed, but is still somehow showing as "likely stuck".
- is related to
-
-
- Resolved
-
- relates to
-
-
- Resolved
-
-
JENKINS-60348 Followup - Investigate root cause of idle executors
-
- Open
-
-
-
- Resolved
-
- links to
[JENKINS-45571] "likely stuck" job is not actually stuck.
Stark Gabriel
created issue -
From another source I got a few logs of something that looks a lot like this – could you check whether the listed builds appear in the Jenkins system log as having failed to resume after restart, or similar?
The best theory we have is that something is causing the OneOffExecutors to not be cleaned up correctly, and that it might be related to resuming pipelines at startup. zeal_iskander Are you still seeing this issue? If so, what versions of the workflow-cps and workflow-job plugins do you have installed? Do you see any log messages about the builds that completed but are showing as likely stuck?
I wouldn't know, I don't work at that company anymore. Sorry!
Stark Gabriel
added a comment - - edited I wouldn't know, I don't work at that company anymore. Sorry! 
Sam Van Oort
made changes -
| Component/s | New: workflow-cps-plugin [ 21713 ] | |
| Component/s | New: workflow-job-plugin [ 21716 ] |
Sam Van Oort
made changes -
| Assignee | Original: Stark Gabriel [ zeal_iskander ] | New: Sam Van Oort [ svanoort ] |
Sam Van Oort
added a comment - dnusbaum danielbeck IIUC what is being described, this actually maps to a really obnoxious bug I've been investigating for several weeks that has been blocking release of a fix/improvement to persistence (with threading implications due to the use of synchronization during I/O).
It seems to be on the Pipeline end itself though – it relates to how the Pipeline job interacts with the OneOffExecutor created when it throws an AsynchronousExecution upon running. The Pipeline may even get marked as completed, but somehow the listener that terminates the Pipeline is not invoked – I'm simplifying grossly here, of course, in reality there's a very complex asynchronous chain of events with a complex threading model underlying all this.
The behavior can be traced to the upon-resume situation if state was incompletely persisted AFAICT, but it requires somewhat precise timing to trigger the events.
Some situations that cause this behavior have probably been solved by prior fixes, but clearly not all of them.
Sam Van Oort
added a comment - dnusbaum danielbeck IIUC what is being described, this actually maps to a really obnoxious bug I've been investigating for several weeks that has been blocking release of a fix/improvement to persistence (with threading implications due to the use of synchronization during I/O).
It seems to be on the Pipeline end itself though – it relates to how the Pipeline job interacts with the OneOffExecutor created when it throws an AsynchronousExecution upon running. The Pipeline may even get marked as completed, but somehow the listener that terminates the Pipeline is not invoked – I'm simplifying grossly here, of course, in reality there's a very complex asynchronous chain of events with a complex threading model underlying all this.
The behavior can be traced to the upon-resume situation if state was incompletely persisted AFAICT, but it requires somewhat precise timing to trigger the events.
Some situations that cause this behavior have probably been solved by prior fixes, but clearly not all of them. | Remote Link | New: This issue links to "jenkinsci/workflow-cps-plugin#234 (Web Link)" [ 21237 ] |
Builds are successful until they're not, even while running. It doesn't mean it finished. In fact, while an executor is in use, it's very likely to not be finished.