To solve the UI nightmare somewhat it would be nice if the "Multi-Project Throttle Category" checkboxes were hidden / disabled if "Throttle this project alone" is selected.
More general, what about a UI like this ("()" indicating radio buttons, "[]" indicating checkboxes):
—
Apply thread limits from () project only () project, then categories
Maximum total concurrent builds _____________________________________________
Maximum concurrent builds per node _____________________________________________
Throttle category membership [] category name 1 [] category name 2 ...
—
Similar to what I suggested in the beginning, the last line should only be display if the "project, then categories" radio button is selected.
Amusingly, in older versions of the plugin, that was possible - it'd check the per-project throttle values first, and if they were 0, it'd fall back to category throttle values, but it was a UI nightmare and kind of a pain on the backend. I'll see what I can do.