Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34822

TestNG plugin has incorrect default threshold values

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • testng-plugin
    • None
    • Jenkins 2.2
      TestNG Results Plugin 1.12

      After updating the TestNG plugin to 1.12, the new feature from JENKINS-20985 caused our tests to pass regardless of failures/skips in our TestNG results. By default, the threshold is set to 100% if users have not configured the plugin yet.

          * Helps resolve XML configs for versions before 1.11 when these new config options were introduced.
          * See https://wiki.jenkins-ci.org/display/JENKINS/Hint+on+retaining+backward+compatibility
          * @return resolved object
           */
         protected Object readResolve() {
            if (unstableSkips == null) {
               unstableSkips = unstableOnSkippedTests ? 0 : 100;
            }
            if (failedFails == null) {
               failedFails = 100;
            }
            if (failedSkips == null) {
               failedSkips = 100;
            }
            if (thresholdMode == null) {
               thresholdMode = 2;
            }
            return this;
         }
      

      This means that by default, a test will require MORE THAN 100% skips/failures in order to mark the build as unstable/failed from TestNG results. It would make more sense if the default was set to 0, and the thresholdMode set to 1. This way, a single failure/skip will cause the build to be unstable/fail like in previous versions.

          [JENKINS-34822] TestNG plugin has incorrect default threshold values

          Code changed in jenkins
          User: nullin
          Path:
          README
          src/main/java/hudson/plugins/testng/Publisher.java
          http://jenkins-ci.org/commit/testng-plugin-plugin/ed80688950e8b0c2bfbf6399c9956bcb15e9fcc0
          Log:
          Fixed: JENKINS-34822 - TestNG plugin has incorrect default threshold values

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: nullin Path: README src/main/java/hudson/plugins/testng/Publisher.java http://jenkins-ci.org/commit/testng-plugin-plugin/ed80688950e8b0c2bfbf6399c9956bcb15e9fcc0 Log: Fixed: JENKINS-34822 - TestNG plugin has incorrect default threshold values

          Nalin Makar added a comment -

          I think I resolved it. Fixed in v1.13.

          Nalin Makar added a comment - I think I resolved it. Fixed in v1.13.

          Cam Spencer added a comment -

          Hey chriseverling,

          You're correct that the original behavior wouldn't FAIL a build for TestNG skips/fails, and that the default threshold for FAIL should be 100%. However if you look at the screenshot I posted, the default threshold for UNSTABLE was set to 100% which resulted in some notoriously bad tests to suddenly pass. I just tested the fix that nalin_makar implemented in 1.113, and the default UNSTABLE threshold for TestNG fails was correctly set to 0%. The default TestNG skip for UNSTABLE was still set to 100% which I think is correct with previous behavior (need confirmation).

          Cam Spencer added a comment - Hey chriseverling , You're correct that the original behavior wouldn't FAIL a build for TestNG skips/fails, and that the default threshold for FAIL should be 100%. However if you look at the screenshot I posted, the default threshold for UNSTABLE was set to 100% which resulted in some notoriously bad tests to suddenly pass. I just tested the fix that nalin_makar implemented in 1.113, and the default UNSTABLE threshold for TestNG fails was correctly set to 0%. The default TestNG skip for UNSTABLE was still set to 100% which I think is correct with previous behavior (need confirmation).

          vivian zhang added a comment - - edited

          nullin I have testng plugin 1.14 installed but found that my build result still shown as successful even though all tests have been skipped.  The threshold mode was set to % and threshold to mark build as unstable for skipped tests were set as 0.

          I found the piece of code that does the checking and found the issue could be caused by the wrong default value.  They are set to 100 which I think should be 0. 

          --------------------------
          testng-plugin-plugin/src/main/java/hudson/plugins/testng/Publisher.java

             //number of skips that will trigger "Unstable"
             private Integer unstableSkips = 100;

             //number of skips that will trigger "Failed"
             private Integer failedSkips = 100;

           

          } else if (results.getSkipCount() > unstableSkips) {
             logger.println(String.format("%d tests were skipped, which exceeded threshold of %d. Marking     build as UNSTABLE",
             results.getSkipCount(), unstableSkips));
             build.setResult(Result.UNSTABLE);

           

          As a result, the unstable skips on Jenkins job has to be > 100% to mark the build as unstable.  But the number will not pass the validate method that makes sure the number is between 1 and 100.

          Can you take a look?  Thanks.

          vivian zhang added a comment - - edited nullin I have testng plugin 1.14 installed but found that my build result still shown as successful even though all tests have been skipped.  The threshold mode was set to % and threshold to mark build as unstable for skipped tests were set as 0. I found the piece of code that does the checking and found the issue could be caused by the wrong default value.  They are set to 100 which I think should be 0.  -------------------------- testng-plugin-plugin/src/main/java/hudson/plugins/testng/Publisher.java    //number of skips that will trigger "Unstable"    private Integer unstableSkips = 100;    //number of skips that will trigger "Failed"    private Integer failedSkips = 100;   } else if (results.getSkipCount() > unstableSkips) {    logger.println(String.format("%d tests were skipped, which exceeded threshold of %d. Marking     build as UNSTABLE",    results.getSkipCount(), unstableSkips));    build.setResult(Result.UNSTABLE);   As a result, the unstable skips on Jenkins job has to be > 100% to mark the build as unstable.  But the number will not pass the validate method that makes sure the number is between 1 and 100. Can you take a look?  Thanks.

            Unassigned Unassigned
            cwcam Cam Spencer
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: