Throwing in my 2 cents here.
Our automation jobs don't currently handle cancellation well. Currently a proper solution won't be made for the short or medium term, so I've tried to disable cancel permissions for all but 1 group who have been trained to perform all required cleanup after cancelling a job.
Your plugin does indeed have a "Cancel" permission under the "Jobs" group. Not granting that permission seems to remove the ability to cancel jobs from the queue and slave sidebars on Jenkins, but alternative cancel buttons can still be found when clicking into the jobs themselves.
So it looks like you're restricting the cancellation of jobs, but not in all places.
There's no permission in Jenkins that represents "privilege to cancel a build", so I'm not sure which permission you are referring to.
If you disagree, please reopen the ticket and attach the screenshots of the matrix configuration UI, so that we can verify where it came from.