There should be an option to skip the build when the change set is empty.
I expected this, as
"only build when the changes are (not) matching expression X"
should imply
"empty change set never matches X"