Steps to reproduce:

1. Install Jenkins 1.554.1 via official Jenkins repo on Ubuntu 13.10 (distro is most likely unrelated anyway).
2. Enable security either with Active Directory or with Jenkins' own user database (prolly occurs with other methods, too). Make sure that Markup Formatter is set to Raw HTML.
3. Login with a user.
4. Create a new freestyle job. Check "This build is parameterized", add a String or Boolean parameter with any name and save the job.
5. Go back to job configuration and delete the parameter.

Symptom:

When you click "Delete" the parameter disappears as it should, but when you click either on Apply or Save nothing happens. All you can do is navigate to somewhere else but when you return to this job the parameter will be there again.

Workaround:

Change markup formatter to Escaped HTML.