This is a regression between 2.269 and 2.270, likely https://github.com/jenkinsci/jenkins/pull/5063

While the popup is closing, a square "popup box" is visible at the top right of where the real popup located.

While the animation was never great (e.g. the triangle pointing at the button was always disconnected from the popup while it appears), this makes it look worse.