Amusingly, in older versions of the plugin, that was possible - it'd check the per-project throttle values first, and if they were 0, it'd fall back to category throttle values, but it was a UI nightmare and kind of a pain on the backend. I'll see what I can do.

To solve the UI nightmare somewhat it would be nice if the "Multi-Project Throttle Category" checkboxes were hidden / disabled if "Throttle this project alone" is selected.

More general, what about a UI like this ("()" indicating radio buttons, "[]" indicating checkboxes):

—
Apply thread limits from () project only () project, then categories
Maximum total concurrent builds _____________________________________________
Maximum concurrent builds per node _____________________________________________
Throttle category membership [] category name 1 [] category name 2 ...
—

Similar to what I suggested in the beginning, the last line should only be display if the "project, then categories" radio button is selected.

Pull request submitted for this: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/20

Currently no UI changes, but it will allow
() Throttle by project alone
() Also throttle by one or more categories

per existing code, if project values are 0, then no throttling by project will occur.

all feedback welcome.
Geoff