Status: Resolved (View Workflow)
workflow-cps plugin 2.47 (upgraded from 2.46)
Using workflow-cps version 2.46 and latest version of Jenkins (2.114) and all other installed plugins, everything works fine. After upgrading workflow-cps to 2.47 and restarting, Jenkins hangs on the "Please wait while Jenkins is getting ready to work" page. I let it sit for about 10 minutes, but no change. It usually starts up in less than two minutes.
Our theme is also not applied, so it appears that the Simple Theme Plugin is not loaded by the time it hangs.
Reverting workflow-cps to 2.46 manually gets it to start up fine.
There doesn't appear to be anything interesting in the jenkins.log, and the jenkins.err.log doesn't appear to be present at all unless I'm looking in the wrong location.
Any advice on where to look for further information?
- is related to
JENKINS-50663 Deadlock on jenkins startup
Thanks for the quick response svanoort. I'll grab the thread dump and upload here as soon as I can - I have to try not to disrupt users of this instance too much if possible.
Yes, the workflow-job plugin was also updated this morning. It appears that disk total usage (R/W combined) spiked to about 24% around the upgrade (I can try to get more specific metrics if necessary). Right now it's around 2-5% and it looks like that's about the same as what it was before upgrading. I've left the workflow-job plugin upgraded, the only plugin I have reverted to an older version at this moment is workflow-cps.
Thread dump: threaddump.txt
I took a glance at it and it mentions a deadlock, is that what you're looking for?
nv035674 Ah, thank you, that helps a lot – It looks like the root cause of your problem is actually https://issues.jenkins-ci.org/browse/JENKINS-50663 which is caused by the Sonar Plugin triggering a deadlock. You should be able to resolve the issue by downgrading sonar to 2.6.1 and can freely upgrade workflow-cps again at the same time.
The workflow-cps upgrade appears to be a red herring – probably since this is a deadlock it just so happens that the timing of the startup process was a bit different between restarts so you got a deadlock in one case but not the other.
I'm closing this one as a duplicate since it appears to be caused by an already-identified bug in the sonar plugin – will be happy to re-open if there proves to be more going on than that (just reply back and re-open it).
Thanks for the investigation. Sorry to point blame the wrong direction, but it's definitely curious that downgrading the workflow-cps plugin "fixes" it. I'll keep an eye on the other Jira for a fix.
nv035674 I'm sorry to hear that. Could you please grab and attach a Thread dump – link here: https://helpx.adobe.com/experience-manager/kb/TakeThreadDump.html
Also, did you also update workflow-job plugin? If so, do you see higher disk activity on the system? This may be a result of one-time migrations of loaded build records to persist completed status. If not, could you please try with both workflow-job and workflow-cps updated?