@Brian: The git plugin does support a "fast remote polling", using the git ls-remote command, which allows fetching the latest revisions from the remote URL without actually requiring a workspace to keep an entire clone of the repository. That then only requires that the git plugin keep track of the last built revisions, and if the revision doesn't match, it would trigger a new build.
That said, the git plugin currently does not appear to save this option properly, as every time I check that box, it appears to get lost the next time I look at the configuration.
The really annoying part of this behavior is that the SCM poller is already starting to do work before the slaves even have a chance to come online (for persistent slaves). So pretty much every restart of Jenkins in my experience, causes all SCM polling jobs to schedule new builds, even though nothing changed and there's no reason to start a build. That has the downside that all my slaves are hosed for at least 5-10 minutes after a startup.
To make matters worse, using the "Purge Build Queue" plugin to clear out the build queue on startup, actually does more harm than good, because it appears to leave the SCM pollers in an unstable state, and eventually one or more slaves seems to become unresponsive, and it's just a cascading effect from there.