Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-2988

Incorrect display of test results when using public static Test suite()

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • junit-plugin
    • None
    • Platform: All, OS: All

      If JUnit (3.x) tests use the suite() method to construct test suites from other
      test classes, and a given test class is used in more than one such suite, they
      can create results which are not displayed well by Hudson. junitResult.xml is
      fine, and so is /testReport/api/xml, but the HTML is confused.

      For example, given a WordLibraryTest.java which has one passing and one failing
      test, both writing something to stderr, and

      public class MetaTest {
      public static Test suite() {
      TestSuite all = new TestSuite();
      all.addTest(new WordLibraryTest("testBogosity"));
      all.addTest(new WordLibraryTest("testBogosity") {});
      return all;
      }
      }

      you will get the attached junitResult.xml. Note that the suite names reflect the
      class that was actually launched, whereas the class names in the test cases
      reflect the location of the test method.

      The HTML display gets messed up: WordLibraryTest looks to have three tests in
      it, MetaTest$1 one test. But two of the WordLibraryTest.testBogosity links
      actually point to the same place, and reflect just one of the actual results.
      One of the stderr results is not displayed, and the other displayed twice.

      For a realistic example look at:

      http://deadlock.netbeans.org/hudson/job/NB-Core-Build/lastBuild/testReport/org.openide.filesystems/XMLFileSystemTestHid/

      Here XMLFileSystemTestHid.java contains a bunch of test methods, and *Test.java
      in four different modules instantiate it (from suite()) with four different
      "filesystem" impls to be tested. Whereas you can see that each test case is run
      four times, and how long each took and whether it passed or failed, there is
      only one hyperlink created for each method - so you cannot get details most of
      the time.

      I have found that the following tiny patch makes things much better for such cases:

      Index: main/core/src/main/java/hudson/tasks/junit/PackageResult.java
      ===================================================================
      — main/core/src/main/java/hudson/tasks/junit/PackageResult.java (revision 15073)
      +++ main/core/src/main/java/hudson/tasks/junit/PackageResult.java (working copy)
      @@ -100,7 +100,9 @@
      }

      void add(CaseResult r) {

      • String n = r.getSimpleName(), sn = safe;
        + String fqn = r.getParent().getName();
        + String n = fqn.substring(fqn.lastIndexOf('.') + 1);
        + String sn = safe;
        ClassResult c = classes.get(sn);
        if(c==null)
        classes.put(sn,c=new ClassResult(this,n));

      With this patch, the HTML display groups tests into packages based on the
      package of the suite, not the test class, and everything appears OK. For
      programs that do not use suite() there should be no difference. The HTML display
      will show the test class name in All Failed Tests while linking to the right
      result using an HREF based on the suite name, which is odd but not a major problem.

      The problem is that I don't know what else, if anything, might need to be
      changed to make this a complete fix. Should CaseResult.getFullName() return
      parent.getName() + getName()? What about getSimpleName() and getPackageName()?
      (getClassName() is exported and should be left untouched.) These are not solely
      used in HTML display templates; TestResult.freeze uses getPackageName.

      Another concern is that some projects may use suite() just for grouping. For
      example, if you do not run your actual test classes as such, but just invoke
      <junit> (or whatever) on some suite classes, and any given actual test is
      included in at most one suite, then the current display is fine as it is and
      the patched display would probably be considered wrong - it would show a single
      suite class (in reality a placeholder) containing hundreds of test cases
      implemented in different classes.

      Also note that CaseResult(SuiteResult parent, Element testCase, String
      testCaseName) fails to set this.parent, and I'm not sure whether this is
      intentional or an oversight. Certainly several existing methods seem to assume
      that parent != null.

          [JENKINS-2988] Incorrect display of test results when using public static Test suite()

          Jesse Glick added a comment -

          Created an attachment (id=544)
          Sample report showing similar test cases coming from different suites

          Jesse Glick added a comment - Created an attachment (id=544) Sample report showing similar test cases coming from different suites

          Jesse Glick added a comment -

          A more general point is that (with or without my suggested patch) Hudson
          considers CaseResult.fullName to be a unique identifier, which is not
          necessarily the case. For example:

          import junit.framework.*;
          public class WeirdTest extends TestCase {
          private final String msg;
          public WeirdTest(String n, String msg)

          { super(n); this.msg = msg; }

          public void testOneThing()

          { fail(msg + " #1"); }

          public void testAnotherThing()

          { fail(msg + " #2"); }

          public static Test suite()

          { TestSuite suite = new TestSuite(); suite.addTest(new WeirdTest("testOneThing", "problem")); suite.addTest(new WeirdTest("testAnotherThing", "problem")); suite.addTest(new WeirdTest("testOneThing", "error")); suite.addTest(new WeirdTest("testAnotherThing", "error")); return suite; }

          }

          While /testReport/api/xml shows all the information correctly, /testReport HTML
          will never show you the stack traces resulting in "error #1" or "error #2". This
          is because Hudson creates URLs for case results that are not uniquified.

          Jesse Glick added a comment - A more general point is that (with or without my suggested patch) Hudson considers CaseResult.fullName to be a unique identifier, which is not necessarily the case. For example: import junit.framework.*; public class WeirdTest extends TestCase { private final String msg; public WeirdTest(String n, String msg) { super(n); this.msg = msg; } public void testOneThing() { fail(msg + " #1"); } public void testAnotherThing() { fail(msg + " #2"); } public static Test suite() { TestSuite suite = new TestSuite(); suite.addTest(new WeirdTest("testOneThing", "problem")); suite.addTest(new WeirdTest("testAnotherThing", "problem")); suite.addTest(new WeirdTest("testOneThing", "error")); suite.addTest(new WeirdTest("testAnotherThing", "error")); return suite; } } While /testReport/api/xml shows all the information correctly, /testReport HTML will never show you the stack traces resulting in "error #1" or "error #2". This is because Hudson creates URLs for case results that are not uniquified.

          Jesse Glick added a comment -

          Jesse Glick added a comment - Also reported as: http://www.netbeans.org/nonav/issues/show_bug.cgi?id=159859

          Code changed in hudson
          User: : jglick
          Path:
          trunk/hudson/main/core/src/main/java/hudson/tasks/junit/CaseResult.java
          trunk/hudson/main/core/src/main/java/hudson/tasks/junit/ClassResult.java
          trunk/hudson/main/core/src/main/java/hudson/tasks/junit/PackageResult.java
          trunk/hudson/main/core/src/main/java/hudson/tasks/junit/TabulatedResult.java
          trunk/hudson/main/core/src/main/java/hudson/tasks/junit/TestObject.java
          trunk/hudson/main/core/src/main/resources/hudson/tasks/junit/CaseResult/index.jelly
          http://fisheye4.cenqua.com/changelog/hudson/?cs=16202
          Log:
          [FIXED JENKINS-2988] Confused display of test results when using static Test suite().
          1. Uniquify relative URLs of test objects (plain, plain_2, plain_3, ...) so they can be browsed separately.
          2. Indicate the originating suite name if distinct from the test class name.

          SCM/JIRA link daemon added a comment - Code changed in hudson User: : jglick Path: trunk/hudson/main/core/src/main/java/hudson/tasks/junit/CaseResult.java trunk/hudson/main/core/src/main/java/hudson/tasks/junit/ClassResult.java trunk/hudson/main/core/src/main/java/hudson/tasks/junit/PackageResult.java trunk/hudson/main/core/src/main/java/hudson/tasks/junit/TabulatedResult.java trunk/hudson/main/core/src/main/java/hudson/tasks/junit/TestObject.java trunk/hudson/main/core/src/main/resources/hudson/tasks/junit/CaseResult/index.jelly http://fisheye4.cenqua.com/changelog/hudson/?cs=16202 Log: [FIXED JENKINS-2988] Confused display of test results when using static Test suite(). 1. Uniquify relative URLs of test objects (plain, plain_2, plain_3, ...) so they can be browsed separately. 2. Indicate the originating suite name if distinct from the test class name.

          Code changed in hudson
          User: : jglick
          Path:
          trunk/www/changelog.html
          http://fisheye4.cenqua.com/changelog/hudson/?cs=16203
          Log:
          JENKINS-2988 Noted.

          SCM/JIRA link daemon added a comment - Code changed in hudson User: : jglick Path: trunk/www/changelog.html http://fisheye4.cenqua.com/changelog/hudson/?cs=16203 Log: JENKINS-2988 Noted.

          Ulli Hafner added a comment -

          Does this fix #2184 and #2604, too?

          Ulli Hafner added a comment - Does this fix #2184 and #2604, too?

          Jesse Glick added a comment -
              • Issue 2184 has been marked as a duplicate of this issue. ***

          Jesse Glick added a comment - Issue 2184 has been marked as a duplicate of this issue. ***

          Jesse Glick added a comment -
              • Issue 2604 has been marked as a duplicate of this issue. ***

          Jesse Glick added a comment - Issue 2604 has been marked as a duplicate of this issue. ***

            Unassigned Unassigned
            jglick Jesse Glick
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: