Git polling builds same branch multiple times when 'Execute concurrent builds if necessary' turned on

We have a job setup to build pull request branches based on a naming convention on the branches. The branch specifier is /pr/. We have 'Execute concurrent builds if necessary' turned on for this job and sometimes Jenkins will kick off 4-5 builds for the same branch at the same time and fill up the executors. I've tried adding a quiet period but that didn't seem to help.