Details
-
Type:
Bug
-
Status: Reopened (View Workflow)
-
Priority:
Critical
-
Resolution: Unresolved
-
Component/s: testng-plugin
-
Labels:None
-
Environment:Jenkins 2.2
TestNG Results Plugin 1.12
-
Similar Issues:
Description
After updating the TestNG plugin to 1.12, the new feature from JENKINS-20985 caused our tests to pass regardless of failures/skips in our TestNG results. By default, the threshold is set to 100% if users have not configured the plugin yet.
* Helps resolve XML configs for versions before 1.11 when these new config options were introduced. * See https://wiki.jenkins-ci.org/display/JENKINS/Hint+on+retaining+backward+compatibility * @return resolved object */ protected Object readResolve() { if (unstableSkips == null) { unstableSkips = unstableOnSkippedTests ? 0 : 100; } if (failedFails == null) { failedFails = 100; } if (failedSkips == null) { failedSkips = 100; } if (thresholdMode == null) { thresholdMode = 2; } return this; }
This means that by default, a test will require MORE THAN 100% skips/failures in order to mark the build as unstable/failed from TestNG results. It would make more sense if the default was set to 0, and the thresholdMode set to 1. This way, a single failure/skip will cause the build to be unstable/fail like in previous versions.
Attachments
Issue Links
- is related to
-
-
- Resolved
-
-
JENKINS-20985 TestNG plugin should has a possibility to use thresholds for failed tests
-
- Resolved
-
- relates to
-
-
- Open
-
Activity
Hey Chris Everling,
You're correct that the original behavior wouldn't FAIL a build for TestNG skips/fails, and that the default threshold for FAIL should be 100%. However if you look at the screenshot I posted, the default threshold for UNSTABLE was set to 100% which resulted in some notoriously bad tests to suddenly pass. I just tested the fix that nalin_makar implemented in 1.113, and the default UNSTABLE threshold for TestNG fails was correctly set to 0%. The default TestNG skip for UNSTABLE was still set to 100% which I think is correct with previous behavior (need confirmation).
Cam Spencer
added a comment - Hey Chris Everling ,
You're correct that the original behavior wouldn't FAIL a build for TestNG skips/fails, and that the default threshold for FAIL should be 100%. However if you look at the screenshot I posted, the default threshold for UNSTABLE was set to 100% which resulted in some notoriously bad tests to suddenly pass. I just tested the fix that nalin_makar implemented in 1.113, and the default UNSTABLE threshold for TestNG fails was correctly set to 0%. The default TestNG skip for UNSTABLE was still set to 100% which I think is correct with previous behavior (need confirmation). I think I resolved it. Fixed in v1.13.
Nalin Makar
added a comment - I think I resolved it. Fixed in v1.13. Code changed in jenkins
User: nullin
Path:
README
src/main/java/hudson/plugins/testng/Publisher.java
http://jenkins-ci.org/commit/testng-plugin-plugin/ed80688950e8b0c2bfbf6399c9956bcb15e9fcc0
Log:
Fixed: JENKINS-34822 - TestNG plugin has incorrect default threshold values
Nalin Makar I have testng plugin 1.14 installed but found that my build result still shown as successful even though all tests have been skipped. The threshold mode was set to % and threshold to mark build as unstable for skipped tests were set as 0.
I found the piece of code that does the checking and found the issue could be caused by the wrong default value. They are set to 100 which I think should be 0.
--------------------------
testng-plugin-plugin/src/main/java/hudson/plugins/testng/Publisher.java
//number of skips that will trigger "Unstable"
private Integer unstableSkips = 100;
//number of skips that will trigger "Failed"
private Integer failedSkips = 100;
} else if (results.getSkipCount() > unstableSkips) {
logger.println(String.format("%d tests were skipped, which exceeded threshold of %d. Marking build as UNSTABLE",
results.getSkipCount(), unstableSkips));
build.setResult(Result.UNSTABLE);
As a result, the unstable skips on Jenkins job has to be > 100% to mark the build as unstable. But the number will not pass the validate method that makes sure the number is between 1 and 100.
Can you take a look? Thanks.