If JUnit (3.x) tests use the suite() method to construct test suites from other
test classes, and a given test class is used in more than one such suite, they
can create results which are not displayed well by Hudson. junitResult.xml is
fine, and so is /testReport/api/xml, but the HTML is confused.

For example, given a WordLibraryTest.java which has one passing and one failing
test, both writing something to stderr, and

public class MetaTest {
public static Test suite() {
TestSuite all = new TestSuite();
all.addTest(new WordLibraryTest("testBogosity"));
all.addTest(new WordLibraryTest("testBogosity") {});
return all;
}
}

you will get the attached junitResult.xml. Note that the suite names reflect the
class that was actually launched, whereas the class names in the test cases
reflect the location of the test method.

The HTML display gets messed up: WordLibraryTest looks to have three tests in
it, MetaTest$1 one test. But two of the WordLibraryTest.testBogosity links
actually point to the same place, and reflect just one of the actual results.
One of the stderr results is not displayed, and the other displayed twice.

For a realistic example look at:

http://deadlock.netbeans.org/hudson/job/NB-Core-Build/lastBuild/testReport/org.openide.filesystems/XMLFileSystemTestHid/

Here XMLFileSystemTestHid.java contains a bunch of test methods, and *Test.java
in four different modules instantiate it (from suite()) with four different
"filesystem" impls to be tested. Whereas you can see that each test case is run
four times, and how long each took and whether it passed or failed, there is
only one hyperlink created for each method - so you cannot get details most of
the time.

I have found that the following tiny patch makes things much better for such cases:

Index: main/core/src/main/java/hudson/tasks/junit/PackageResult.java
===================================================================
— main/core/src/main/java/hudson/tasks/junit/PackageResult.java (revision 15073)
+++ main/core/src/main/java/hudson/tasks/junit/PackageResult.java (working copy)
@@ -100,7 +100,9 @@
}

void add(CaseResult r) {

• String n = r.getSimpleName(), sn = safe;
  + String fqn = r.getParent().getName();
  + String n = fqn.substring(fqn.lastIndexOf('.') + 1);
  + String sn = safe;
  ClassResult c = classes.get(sn);
  if(c==null)
  classes.put(sn,c=new ClassResult(this,n));

With this patch, the HTML display groups tests into packages based on the
package of the suite, not the test class, and everything appears OK. For
programs that do not use suite() there should be no difference. The HTML display
will show the test class name in All Failed Tests while linking to the right
result using an HREF based on the suite name, which is odd but not a major problem.

The problem is that I don't know what else, if anything, might need to be
changed to make this a complete fix. Should CaseResult.getFullName() return
parent.getName() + getName()? What about getSimpleName() and getPackageName()?
(getClassName() is exported and should be left untouched.) These are not solely
used in HTML display templates; TestResult.freeze uses getPackageName.

Another concern is that some projects may use suite() just for grouping. For
example, if you do not run your actual test classes as such, but just invoke
<junit> (or whatever) on some suite classes, and any given actual test is
included in at most one suite, then the current display is fine as it is and
the patched display would probably be considered wrong - it would show a single
suite class (in reality a placeholder) containing hundreds of test cases
implemented in different classes.

Also note that CaseResult(SuiteResult parent, Element testCase, String
testCaseName) fails to set this.parent, and I'm not sure whether this is
intentional or an oversight. Certainly several existing methods seem to assume
that parent != null.