Details
-
Type:
Bug
-
Status: Reopened (View Workflow)
-
Priority:
Critical
-
Resolution: Unresolved
-
Component/s: testng-plugin
-
Labels:None
-
Environment:Jenkins 2.2
TestNG Results Plugin 1.12
-
Similar Issues:
Description
After updating the TestNG plugin to 1.12, the new feature from JENKINS-20985 caused our tests to pass regardless of failures/skips in our TestNG results. By default, the threshold is set to 100% if users have not configured the plugin yet.
* Helps resolve XML configs for versions before 1.11 when these new config options were introduced. * See https://wiki.jenkins-ci.org/display/JENKINS/Hint+on+retaining+backward+compatibility * @return resolved object */ protected Object readResolve() { if (unstableSkips == null) { unstableSkips = unstableOnSkippedTests ? 0 : 100; } if (failedFails == null) { failedFails = 100; } if (failedSkips == null) { failedSkips = 100; } if (thresholdMode == null) { thresholdMode = 2; } return this; }
This means that by default, a test will require MORE THAN 100% skips/failures in order to mark the build as unstable/failed from TestNG results. It would make more sense if the default was set to 0, and the thresholdMode set to 1. This way, a single failure/skip will cause the build to be unstable/fail like in previous versions.
Attachments
Issue Links
- is related to
-
JENKINS-34711 NullPointerException after Update to 1.11 of plugin
-
- Resolved
-
-
JENKINS-20985 TestNG plugin should has a possibility to use thresholds for failed tests
-
- Resolved
-
- relates to
-
JENKINS-42819 Percentage Threshold Is Broken for 100%
-
- Open
-
Nalin Makar I have testng plugin 1.14 installed but found that my build result still shown as successful even though all tests have been skipped. The threshold mode was set to % and threshold to mark build as unstable for skipped tests were set as 0.
I found the piece of code that does the checking and found the issue could be caused by the wrong default value. They are set to 100 which I think should be 0.
--------------------------
testng-plugin-plugin/src/main/java/hudson/plugins/testng/Publisher.java
//number of skips that will trigger "Unstable"
private Integer unstableSkips = 100;
//number of skips that will trigger "Failed"
private Integer failedSkips = 100;
} else if (results.getSkipCount() > unstableSkips) {
logger.println(String.format("%d tests were skipped, which exceeded threshold of %d. Marking build as UNSTABLE",
results.getSkipCount(), unstableSkips));
build.setResult(Result.UNSTABLE);
As a result, the unstable skips on Jenkins job has to be > 100% to mark the build as unstable. But the number will not pass the validate method that makes sure the number is between 1 and 100.
Can you take a look? Thanks.