I have a test application that exited with error code 0 (which was wrong and is fixed now). It did not run any tests and created the attached Junit report basically stating that 0 tests ran, 0 tests passed and 0 tests failed.

I would suggest to set the verdict of the test to at least unstable or give the possibility to configure that at least 1 test has to be executed and otherwise fail the job.