-
Bug
-
Resolution: Unresolved
-
Critical
-
None
The cause of this problem appears to be the `name` and `timestamp` attributes of a `<testsuite>` tag being equal. When processing the report file, if a `<testsuite>` tag is encountered where these attributes are the same as one that was already seen, the new tag is silently ignored.
This happens with the jUnit publisher and xUnit. It even happens if only the timestamps are equal, which is a nasty bug since timestamps shouldn't be used to uniquely identify a testsuite when they have different names. It looks like Jenkins publishes the first suite and ignores any other XML files with the same timestamp. If the timestamps are empty, they are all reported. Reproduced with Jenkins 1.595 with junit 1.2-beta4, and also reproduced with the latest Jenkins 1.627 and junit 1.8.
See the two attached test_result_*.xml to reproduce: Archive them via junit and only testFoo is reported. If you change the timestamp attribute of either, they both are reported.
[JENKINS-28490] jUnit discarding <testsuite> tags when certain attributes are equal
Kris Reeves
added a comment - I was going to answer, then I had a question, and that led me to an idea:
I created a new Jenkins job from a git repo whose only content was the input xml file, and generated a result from it; this is what I was uncertain how to do before I submitted this report. I was able to narrow down the problem to something specific.
When the test suite name AND timestamp of two `testsuite` tags are the same, one (the latter?) gets dropped.
The reason it was intermittent is that the tests were unit tests and ran fast, so sometimes the timestamp was the same, and sometimes it wasn't. While I was at it, I checked whether changing the testsuite `name` attribute had an effect, and it also did. Only when both the `name` and the `timestamp` attribute were the same did the problem occur. This suggests an immediate workaround but also confirms it quite clearly as a bug – an XML tag should be its own identity/entity, not treated as something like a key name into a hash table based on its attributes, which my best guess as to what's going on.
Kris Reeves
added a comment - I was going to answer, then I had a question, and that led me to an idea:
I created a new Jenkins job from a git repo whose only content was the input xml file, and generated a result from it; this is what I was uncertain how to do before I submitted this report. I was able to narrow down the problem to something specific.
When the test suite name AND timestamp of two `testsuite` tags are the same, one (the latter?) gets dropped.
The reason it was intermittent is that the tests were unit tests and ran fast, so sometimes the timestamp was the same, and sometimes it wasn't. While I was at it, I checked whether changing the testsuite `name` attribute had an effect, and it also did. Only when both the `name` and the `timestamp` attribute were the same did the problem occur. This suggests an immediate workaround but also confirms it quite clearly as a bug – an XML tag should be its own identity/entity, not treated as something like a key name into a hash table based on its attributes, which my best guess as to what's going on. Kris Reeves
added a comment - Any chance of this seeing a fix? This isn't specific to xUnit and happens when just publishing via the jUnit publisher. It also happens if only the timestamps are equal, which is a nasty bug since timestamps shouldn't be used to uniquely identify a testsuite when they have different names. We were noticing fluctuating numbers of tests and this took awhile to debug 
It looks like Jenkins publishes the first one and ignores any other XML files with the same timestamp. If the timestamps are empty, they are all reported. We are using Jenkins 1.595 with junit 1.2-beta4, and I also reproduced it with the latest Jenkins 1.627 and junit 1.8.
mcrooney
added a comment - - edited This isn't specific to xUnit and happens when just publishing via the jUnit publisher. It also happens if only the timestamps are equal, which is a nasty bug since timestamps shouldn't be used to uniquely identify a testsuite when they have different names. We were noticing fluctuating numbers of tests and this took awhile to debug It looks like Jenkins publishes the first one and ignores any other XML files with the same timestamp. If the timestamps are empty, they are all reported. We are using Jenkins 1.595 with junit 1.2-beta4, and I also reproduced it with the latest Jenkins 1.627 and junit 1.8. mcrooney
added a comment - Two files which reproduce the issue. Only testFoo is reported. If you change the timestamp attribute of either, they both are reported.
mcrooney
added a comment - Two files which reproduce the issue. Only testFoo is reported. If you change the timestamp attribute of either, they both are reported. 
Adrian Price
added a comment - I think I'm seeing the same thing. I have a Maven project with both the maven-surefire-plugin:test and gwt-maven-plugin:test producing results - most of the time the results from the latter don't get reported.
I don't understand why this bug is rated as 'minor' - the loss of test results and the inability to track test result trends across builds is surely a critical defect?
Adrian Price
added a comment - I think I'm seeing the same thing. I have a Maven project with both the maven-surefire-plugin:test and gwt-maven-plugin:test producing results - most of the time the results from the latter don't get reported.
I don't understand why this bug is rated as 'minor' - the loss of test results and the inability to track test result trends across builds is surely a critical defect? 
In the absence of a minimal reproduction case, just as a quick guess from looking at the first input file: does this still occur if each <testsuite> has a unique name?
i.e. not always "Mocha Tests"