Manage Jenkins --> Restrict project naming --> Strategy: Pattern --> Name Pattern: <your regex>

When trying to save the configuration, the regex should be checked for correctness (e.g. matching parens)

Alternatively this could even be done in real time on the configuration page.

[Workaround: any regex errors will appear - and need to be fixed immediately - when you try to configure/create a job]