Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34822

TestNG plugin has incorrect default threshold values

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Reopened (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: testng-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.2
      TestNG Results Plugin 1.12
    • Similar Issues:

      Description

      After updating the TestNG plugin to 1.12, the new feature from JENKINS-20985 caused our tests to pass regardless of failures/skips in our TestNG results. By default, the threshold is set to 100% if users have not configured the plugin yet.

          * Helps resolve XML configs for versions before 1.11 when these new config options were introduced.
          * See https://wiki.jenkins-ci.org/display/JENKINS/Hint+on+retaining+backward+compatibility
          * @return resolved object
           */
         protected Object readResolve() {
            if (unstableSkips == null) {
               unstableSkips = unstableOnSkippedTests ? 0 : 100;
            }
            if (failedFails == null) {
               failedFails = 100;
            }
            if (failedSkips == null) {
               failedSkips = 100;
            }
            if (thresholdMode == null) {
               thresholdMode = 2;
            }
            return this;
         }
      

      This means that by default, a test will require MORE THAN 100% skips/failures in order to mark the build as unstable/failed from TestNG results. It would make more sense if the default was set to 0, and the thresholdMode set to 1. This way, a single failure/skip will cause the build to be unstable/fail like in previous versions.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              nullin Nalin Makar
              Reporter:
              cwcam Cam Spencer
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated: