[JENKINS-34822] TestNG plugin has incorrect default threshold values

Type: Bug
Resolution: Unresolved
Priority: Critical
Component/s: testng-plugin
Labels:
None
Environment:
Jenkins 2.2
TestNG Results Plugin 1.12

Similar Issues:
Powered by SuggestiMate

Show

After updating the TestNG plugin to 1.12, the new feature from ~~JENKINS-20985~~ caused our tests to pass regardless of failures/skips in our TestNG results. By default, the threshold is set to 100% if users have not configured the plugin yet.

    * Helps resolve XML configs for versions before 1.11 when these new config options were introduced.
    * See https://wiki.jenkins-ci.org/display/JENKINS/Hint+on+retaining+backward+compatibility
    * @return resolved object
     */
   protected Object readResolve() {
      if (unstableSkips == null) {
         unstableSkips = unstableOnSkippedTests ? 0 : 100;
      }
      if (failedFails == null) {
         failedFails = 100;
      }
      if (failedSkips == null) {
         failedSkips = 100;
      }
      if (thresholdMode == null) {
         thresholdMode = 2;
      }
      return this;
   }

This means that by default, a test will require MORE THAN 100% skips/failures in order to mark the build as unstable/failed from TestNG results. It would make more sense if the default was set to 0, and the thresholdMode set to 1. This way, a single failure/skip will cause the build to be unstable/fail like in previous versions.

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

TestNG Plugin.jpg
153 kB
2016-05-13 19:50

is related to

JENKINS-34711 NullPointerException after Update to 1.11 of plugin

Closed

JENKINS-20985 TestNG plugin should has a possibility to use thresholds for failed tests

Closed

relates to

JENKINS-42819 Percentage Threshold Is Broken for 100%

Open

Cam Spencer created issue - 2016-05-13 19:46

Cam Spencer made changes - 2016-05-13 19:47

Link

New: This issue is related to ~~JENKINS-20985~~ [ ~~JENKINS-20985~~ ]

Cam Spencer made changes - 2016-05-13 19:47

Description

Original: After updating the TestNG plugin to 1.12, the new feature from ~~JENKINS-20985~~ caused our tests to pass regardless of failures/skips in our TestNG results. By default, the threshold is set to 100% if users have not set this up yet.

{code:java}
    * Helps resolve XML configs for versions before 1.11 when these new config options were introduced.
    * See https://wiki.jenkins-ci.org/display/JENKINS/Hint+on+retaining+backward+compatibility
    * @return resolved object
     */
   protected Object readResolve() {
      if (unstableSkips == null) {
         unstableSkips = unstableOnSkippedTests ? 0 : 100;
      }
      if (failedFails == null) {
         failedFails = 100;
      }
      if (failedSkips == null) {
         failedSkips = 100;
      }
      if (thresholdMode == null) {
         thresholdMode = 2;
      }
      return this;
   }
{code}

This means that by default, a test will require MORE THAN 100% skips/failures in order to mark the build as unstable/failed from TestNG results. It would make more sense if the default was set to 1% to be more inline to what previous versions would do.

New: After updating the TestNG plugin to 1.12, the new feature from ~~JENKINS-20985~~ caused our tests to pass regardless of failures/skips in our TestNG results. By default, the threshold is set to 100% if users have not configured the plugin yet.

{code:java}
    * Helps resolve XML configs for versions before 1.11 when these new config options were introduced.
    * See https://wiki.jenkins-ci.org/display/JENKINS/Hint+on+retaining+backward+compatibility
    * @return resolved object
     */
   protected Object readResolve() {
      if (unstableSkips == null) {
         unstableSkips = unstableOnSkippedTests ? 0 : 100;
      }
      if (failedFails == null) {
         failedFails = 100;
      }
      if (failedSkips == null) {
         failedSkips = 100;
      }
      if (thresholdMode == null) {
         thresholdMode = 2;
      }
      return this;
   }
{code}

This means that by default, a test will require MORE THAN 100% skips/failures in order to mark the build as unstable/failed from TestNG results. It would make more sense if the default was set to 1% to be more inline to what previous versions would do.

Cam Spencer made changes - 2016-05-13 19:50

Attachment

New: TestNG Plugin.jpg [ 32714 ]

Nalin Makar made changes - 2016-05-13 19:52

Priority

Original: Major [ 3 ]

New: Critical [ 2 ]

Cam Spencer made changes - 2016-05-13 19:54

Description

Original: After updating the TestNG plugin to 1.12, the new feature from ~~JENKINS-20985~~ caused our tests to pass regardless of failures/skips in our TestNG results. By default, the threshold is set to 100% if users have not configured the plugin yet.

{code:java}
    * Helps resolve XML configs for versions before 1.11 when these new config options were introduced.
    * See https://wiki.jenkins-ci.org/display/JENKINS/Hint+on+retaining+backward+compatibility
    * @return resolved object
     */
   protected Object readResolve() {
      if (unstableSkips == null) {
         unstableSkips = unstableOnSkippedTests ? 0 : 100;
      }
      if (failedFails == null) {
         failedFails = 100;
      }
      if (failedSkips == null) {
         failedSkips = 100;
      }
      if (thresholdMode == null) {
         thresholdMode = 2;
      }
      return this;
   }
{code}

This means that by default, a test will require MORE THAN 100% skips/failures in order to mark the build as unstable/failed from TestNG results. It would make more sense if the default was set to 1% to be more inline to what previous versions would do.

New: After updating the TestNG plugin to 1.12, the new feature from ~~JENKINS-20985~~ caused our tests to pass regardless of failures/skips in our TestNG results. By default, the threshold is set to 100% if users have not configured the plugin yet.

{code:java}
    * Helps resolve XML configs for versions before 1.11 when these new config options were introduced.
    * See https://wiki.jenkins-ci.org/display/JENKINS/Hint+on+retaining+backward+compatibility
    * @return resolved object
     */
   protected Object readResolve() {
      if (unstableSkips == null) {
         unstableSkips = unstableOnSkippedTests ? 0 : 100;
      }
      if (failedFails == null) {
         failedFails = 100;
      }
      if (failedSkips == null) {
         failedSkips = 100;
      }
      if (thresholdMode == null) {
         thresholdMode = 2;
      }
      return this;
   }
{code}

This means that by default, a test will require MORE THAN 100% skips/failures in order to mark the build as unstable/failed from TestNG results. It would make more sense if the default was set to 1, and the thresholdMode set to 1. This way, a single failure/skip will cause the build to be unstable/fail like in previous versions.

Cam Spencer made changes - 2016-05-13 20:03

Description

Original: After updating the TestNG plugin to 1.12, the new feature from ~~JENKINS-20985~~ caused our tests to pass regardless of failures/skips in our TestNG results. By default, the threshold is set to 100% if users have not configured the plugin yet.

{code:java}
    * Helps resolve XML configs for versions before 1.11 when these new config options were introduced.
    * See https://wiki.jenkins-ci.org/display/JENKINS/Hint+on+retaining+backward+compatibility
    * @return resolved object
     */
   protected Object readResolve() {
      if (unstableSkips == null) {
         unstableSkips = unstableOnSkippedTests ? 0 : 100;
      }
      if (failedFails == null) {
         failedFails = 100;
      }
      if (failedSkips == null) {
         failedSkips = 100;
      }
      if (thresholdMode == null) {
         thresholdMode = 2;
      }
      return this;
   }
{code}

This means that by default, a test will require MORE THAN 100% skips/failures in order to mark the build as unstable/failed from TestNG results. It would make more sense if the default was set to 1, and the thresholdMode set to 1. This way, a single failure/skip will cause the build to be unstable/fail like in previous versions.

New: After updating the TestNG plugin to 1.12, the new feature from ~~JENKINS-20985~~ caused our tests to pass regardless of failures/skips in our TestNG results. By default, the threshold is set to 100% if users have not configured the plugin yet.

{code:java}
    * Helps resolve XML configs for versions before 1.11 when these new config options were introduced.
    * See https://wiki.jenkins-ci.org/display/JENKINS/Hint+on+retaining+backward+compatibility
    * @return resolved object
     */
   protected Object readResolve() {
      if (unstableSkips == null) {
         unstableSkips = unstableOnSkippedTests ? 0 : 100;
      }
      if (failedFails == null) {
         failedFails = 100;
      }
      if (failedSkips == null) {
         failedSkips = 100;
      }
      if (thresholdMode == null) {
         thresholdMode = 2;
      }
      return this;
   }
{code}

This means that by default, a test will require MORE THAN 100% skips/failures in order to mark the build as unstable/failed from TestNG results. It would make more sense if the default was set to 0, and the thresholdMode set to 1. This way, a single failure/skip will cause the build to be unstable/fail like in previous versions.

Nalin Makar made changes - 2016-05-15 23:21

Resolution		New: Fixed [ 1 ]
Status	Original: Open [ 1 ]	New: Resolved [ 5 ]

Cam Spencer made changes - 2016-05-20 17:47

Link

New: This issue is related to ~~JENKINS-34711~~ [ ~~JENKINS-34711~~ ]

R. Tyler Croy made changes - 2016-07-25 23:57

Workflow

Original: JNJira [ 171013 ]

New: JNJira + In-Review [ 199007 ]

vivian zhang made changes - 2017-05-19 21:59

Resolution	Original: Fixed [ 1 ]
Status	Original: Resolved [ 5 ]	New: Reopened [ 4 ]

Assignee:: Unassigned

Reporter:: Cam Spencer

Votes:: 1 Vote for this issue

Watchers:: 4 Start watching this issue

Created:: 2016-05-13 19:46

Updated:: 2021-11-10 04:34

Jenkins

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates