To help unattended reocvery from startup failure, introduce the following functionalities:
- If Jenkins fails to complete the bootup sequence, produce a marker file that captures the failure. If the boot sequence succeeds, remove this file.
- If Jenkins fails to complete the bootup sequence, evaluate $JENKINS_HOME/boot-failure.groovy.d/* in the lexicological order
These user-supplied script can capture additional details, raise alerts, restart Jenkins (possibly after doing any other corrective actions.)
We should also count the repeated failures to boot, so that these scripts can try progressively aggressive measures.