I have a Maven 2/3 project in Jenkins. Some days ago I discovered that the tests in the project couldn't be run, but Jenkins had marked the last build as successful.

In the suite file, one developer had added the name of a class that didn't exist (wrong package). This means that TestNG failed early since it couldn't load the class. In the Jenkins log I could see the failure:

04:02:42 -------------------------------------------------------
04:02:42 T E S T S
04:02:42 -------------------------------------------------------
04:02:42 Running TestSuite
04:02:42 [ERROR] There are test failures.

But shortly thereafter in the log:

04:02:46 projectSucceeded ...

I worked around the problem (i.e., that Jenkins doesn't fail the build despite the error) by adding -Dmaven.test.failure.ignore=false to MAVEN_OPTS.

I believe that Jenkins sets maven.test.failure.ignore to true in order to be able to mark a project as unstable instead of failing. If that is the case, then it's clearly a bug that it doesn't detect that no test cases were run whatsoever.