It's JUnit Plugin 1.19 on ci.j.io.

I added a bit of logging to the junit plugin and when it gets to this point:

https://github.com/jenkinsci/junit-plugin/blob/master/src/main/java/hudson/tasks/junit/JUnitResultArchiver.java#L196

getFailCount() is returning 0.

I'll continue to dig in.

Maybe a missing tally() call in mergeResult()?

I added a call to tally() in the merge method of TestResult (which amounts to the same thing as putting it in mergeResult(). Rerunning...takes about 3 hours.

That's looks like it is the problem. With the call to tally() in TestResult.merge(), the failures show up correctly and the build is marked as unstable. Not sure if the location I put the call is correct for all use cases, if people think it is, I can create a PR for it for the junit-plugin.

This is the change I made to TestResult.java

    void merge(TestResult other) {
        for (SuiteResult suite : other.suites) {
            suite.setParent(null); // otherwise freeze ignores it
            add(suite);
        }
        tally();
    }

The call to tally() at the end is the addition.

That's where I'd have put it.

Should be straightforward to write a test for.

See https://github.com/jenkinsci/junit-plugin/pull/63

Code changed in jenkins
User: Alex Earl
Path:
src/test/java/hudson/tasks/junit/TestResultTest.java
src/test/resources/hudson/tasks/junit/JENKINS-41134/TestSuite_first.xml
src/test/resources/hudson/tasks/junit/JENKINS-41134/TestSuite_second.xml
http://jenkins-ci.org/commit/junit-plugin/c0f470f994693d50d78491bde7deb938903ccd7e
Log:
Add test for JENKINS-41134

Code changed in jenkins
User: Alex Earl
Path:
src/main/java/hudson/tasks/junit/TestResult.java
src/test/java/hudson/tasks/junit/TestResultTest.java
src/test/resources/hudson/tasks/junit/JENKINS-41134/TestSuite_first.xml
src/test/resources/hudson/tasks/junit/JENKINS-41134/TestSuite_second.xml
src/test/resources/hudson/tasks/junit/JENKINS-41134/TestSuite_second_dup_first.xml
http://jenkins-ci.org/commit/junit-plugin/631c41350f6eb24520af19083a9f2fe27163f572
Log:
Merge pull request #63 from slide/master

Fix JENKINS-41134

Compare: https://github.com/jenkinsci/junit-plugin/compare/90e7ddbda961...631c41350f6e