Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34822

TestNG plugin has incorrect default threshold values

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • testng-plugin
    • None
    • Jenkins 2.2
      TestNG Results Plugin 1.12

      After updating the TestNG plugin to 1.12, the new feature from JENKINS-20985 caused our tests to pass regardless of failures/skips in our TestNG results. By default, the threshold is set to 100% if users have not configured the plugin yet.

          * Helps resolve XML configs for versions before 1.11 when these new config options were introduced.
          * See https://wiki.jenkins-ci.org/display/JENKINS/Hint+on+retaining+backward+compatibility
          * @return resolved object
           */
         protected Object readResolve() {
            if (unstableSkips == null) {
               unstableSkips = unstableOnSkippedTests ? 0 : 100;
            }
            if (failedFails == null) {
               failedFails = 100;
            }
            if (failedSkips == null) {
               failedSkips = 100;
            }
            if (thresholdMode == null) {
               thresholdMode = 2;
            }
            return this;
         }
      

      This means that by default, a test will require MORE THAN 100% skips/failures in order to mark the build as unstable/failed from TestNG results. It would make more sense if the default was set to 0, and the thresholdMode set to 1. This way, a single failure/skip will cause the build to be unstable/fail like in previous versions.

            Unassigned Unassigned
            cwcam Cam Spencer
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: