It doesn't look like removing options for what to build PR/Fork/Origin Branches/etc works, after adding them. Specifically, I added to the default, then later removed them, all on the Org config, and the child branches only ever reflected the change of addition, and didn't remove the options as was done on the containing org folder.