When configuring the performance plugin, the check box for "Fail build when result files are not present" does not stay checked after a save and a reload of the project configuration.

Based on what I have seen the behavior may be as follows:

If the box is checked and the configuration saved, the setting works. However, if the configuration is re-loaded, it re-loads with the box unchecked, if it is then saved again (for some other changes), the setting is undone.