-
Bug
-
Resolution: Unresolved
-
Critical
-
None
The cause of this problem appears to be the `name` and `timestamp` attributes of a `<testsuite>` tag being equal. When processing the report file, if a `<testsuite>` tag is encountered where these attributes are the same as one that was already seen, the new tag is silently ignored.
This happens with the jUnit publisher and xUnit. It even happens if only the timestamps are equal, which is a nasty bug since timestamps shouldn't be used to uniquely identify a testsuite when they have different names. It looks like Jenkins publishes the first suite and ignores any other XML files with the same timestamp. If the timestamps are empty, they are all reported. Reproduced with Jenkins 1.595 with junit 1.2-beta4, and also reproduced with the latest Jenkins 1.627 and junit 1.8.
See the two attached test_result_*.xml to reproduce: Archive them via junit and only testFoo is reported. If you change the timestamp attribute of either, they both are reported.
[JENKINS-28490] jUnit discarding <testsuite> tags when certain attributes are equal
Kris Reeves
created issue -
Kris Reeves
made changes -
| Environment |
New:
CentOS release 6.6 (Final) Linux jenkins 2.6.32-504.16.2.el6.x86_64 #1 SMP Wed Apr 22 06:48:29 UTC 2015 x86 OpenJDK Runtime Environment (rhel-2.5.5.1.el6_6-x86_64 u79-b14) OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode) Jenkins 1.611 xUnit plugin 1.96 jUnit plugin 1.6 |
Kris Reeves
added a comment - I was going to answer, then I had a question, and that led me to an idea:
I created a new Jenkins job from a git repo whose only content was the input xml file, and generated a result from it; this is what I was uncertain how to do before I submitted this report. I was able to narrow down the problem to something specific.
When the test suite name AND timestamp of two `testsuite` tags are the same, one (the latter?) gets dropped.
The reason it was intermittent is that the tests were unit tests and ran fast, so sometimes the timestamp was the same, and sometimes it wasn't. While I was at it, I checked whether changing the testsuite `name` attribute had an effect, and it also did. Only when both the `name` and the `timestamp` attribute were the same did the problem occur. This suggests an immediate workaround but also confirms it quite clearly as a bug – an XML tag should be its own identity/entity, not treated as something like a key name into a hash table based on its attributes, which my best guess as to what's going on.
Kris Reeves
added a comment - I was going to answer, then I had a question, and that led me to an idea:
I created a new Jenkins job from a git repo whose only content was the input xml file, and generated a result from it; this is what I was uncertain how to do before I submitted this report. I was able to narrow down the problem to something specific.
When the test suite name AND timestamp of two `testsuite` tags are the same, one (the latter?) gets dropped.
The reason it was intermittent is that the tests were unit tests and ran fast, so sometimes the timestamp was the same, and sometimes it wasn't. While I was at it, I checked whether changing the testsuite `name` attribute had an effect, and it also did. Only when both the `name` and the `timestamp` attribute were the same did the problem occur. This suggests an immediate workaround but also confirms it quite clearly as a bug – an XML tag should be its own identity/entity, not treated as something like a key name into a hash table based on its attributes, which my best guess as to what's going on. Kris Reeves
made changes -
| Description | Original: I have a project set to publish test results using the xUnit plugin, jUnit format. After noticing some irregularities in the graph, I found that apparently the "junitResult.xml" file that is being created kind of arbitrarily doesn't have one of the test suites in the input xml file. Replacing the timestamps with static text, a diff between the input files comes up clean; a diff between the output files reveals that it appears to be the same section that is getting excluded. I've attached the input and output files below. The section being excluded sometimes is the section at line 69. |
New:
I have a project set to publish test results using the xUnit plugin, jUnit format. After noticing some irregularities in the graph, I found that apparently the "junitResult.xml" file that is being created kind of arbitrarily doesn't have one of the test suites in the input xml file. Replacing the timestamps with static text, a diff between the input files comes up clean; a diff between the output files reveals that it appears to be the same section that is getting excluded. I've attached the input and output files below. The section being excluded sometimes is the section at line 69. Update: The cause of this problem appears to be the `name` and `timestamp` attributes of a `<testsuite>` tag being equal. When processing the report file, if a `<testsuite>` tag is encountered where these attributes are the same as one that was already seen, the new tag is silently(?) ignored. |
| Summary | Original: jUnit (?) randomly not including test suites | New: jUnit (?) discarding <testsuite> tags when certain attributes are equal |
Kris Reeves
added a comment - Any chance of this seeing a fix? This isn't specific to xUnit and happens when just publishing via the jUnit publisher. It also happens if only the timestamps are equal, which is a nasty bug since timestamps shouldn't be used to uniquely identify a testsuite when they have different names. We were noticing fluctuating numbers of tests and this took awhile to debug 
It looks like Jenkins publishes the first one and ignores any other XML files with the same timestamp. If the timestamps are empty, they are all reported. We are using Jenkins 1.595 with junit 1.2-beta4, and I also reproduced it with the latest Jenkins 1.627 and junit 1.8.
mcrooney
added a comment - - edited This isn't specific to xUnit and happens when just publishing via the jUnit publisher. It also happens if only the timestamps are equal, which is a nasty bug since timestamps shouldn't be used to uniquely identify a testsuite when they have different names. We were noticing fluctuating numbers of tests and this took awhile to debug It looks like Jenkins publishes the first one and ignores any other XML files with the same timestamp. If the timestamps are empty, they are all reported. We are using Jenkins 1.595 with junit 1.2-beta4, and I also reproduced it with the latest Jenkins 1.627 and junit 1.8. mcrooney
added a comment - Two files which reproduce the issue. Only testFoo is reported. If you change the timestamp attribute of either, they both are reported.
mcrooney
added a comment - Two files which reproduce the issue. Only testFoo is reported. If you change the timestamp attribute of either, they both are reported. mcrooney
made changes -
| Attachment | New: test_result_9114558122204227176.xml [ 30604 ] | |
| Attachment | New: test_result_1012678300206172099.xml [ 30605 ] |
mcrooney
made changes -
| Summary | Original: jUnit (?) discarding <testsuite> tags when certain attributes are equal | New: jUnit discarding <testsuite> tags when certain attributes are equal |
In the absence of a minimal reproduction case, just as a quick guess from looking at the first input file: does this still occur if each <testsuite> has a unique name?
i.e. not always "Mocha Tests"