There was a message on Gitter from a user, Junholv OBO. I tried reproducing his issue, but my Jenkins had an invalid server name for Zeppelin.

However, the job was still marked as successful. Instead I think it must be marked as failure.