During the past days I've been restarting a CI farm controller a few times for maintenance, and for too many of those times I saw this error in logs:

java.lang.IllegalStateException: The class jenkins.security.QueueItemAuthenticatorConfiguration was not found, potentially not yet loaded
        at hudson.ExtensionList.getInstance(ExtensionList.java:166) 

When this situation strikes, server has to be restarted since it can not launch any jobs (spewing the same error into log/stderr).

I think I have seen such message earlier (several years ago) too, and IIRC back then it was about a race condition in server start-up, whether some components poked into core before or after auth classes were initialized (not per se an issue with Queue, IIRC).

Here I tried to rule out impact from the outside (with Swarm clients aggressively reconnecting if server is lost), by stopping the reverse-proxy web server which publishes Jenkins to the world, takes care of HTTPS, etc. Right on the subsequent restart Jenkins went up well and was usable.

So while not 100% convinced that swarm-plugin is at fault, I tend to think so at the moment. Perhaps it should not accept connections (do some heavy logic about it) until it makes sure the server core has initialized sufficiently.

If my guesses are correct, the stopping of webserver (or manipulation of firewall etc.) allow a workaround. So this is not a show-stopper for servers where an admin is available to hold its hand during restart... Otherwise, fairly annoying and SLA-breaking.