-
Improvement
-
Resolution: Won't Do
-
Major
-
None
It would be nice to be able to categorize tests by providing an optional "category" parameter to the xunit publisher.
The specific use case I had in mind was to provide some ability to group test results based on stages in a Jenkins pipeline build. If the xunit test publisher accepted a "category" parameter, one could provide some unique name - like the name of the current build stage - as the category, allowing all the tests reported by that publisher to be grouped together and kept separate from the results produced by other stages.
In general, I think this feature would be handy for legacy and single-stage test suites as well. Suppose several teams are writing tests for a project and you want to group the tests from team 1 together, team 2, etc. You could then have spearate publish operations for the different xUnit report files, each with a custom "team" based category, so their results can be grouped together.
As for the visual representation, I was thinking that maybe a small row of tabs above or below the test report table would work, with one tab labelled "all" and each of the others labelled with the associated category name. In this way, if you wanted to see information on a specific group of tests you could just select the appropriate tab / category.
- depends on
-
-
- Resolved
-
[JENKINS-49344] Add support for categorizing tests to xunit plugin
Jakub Pawlinski
added a comment - My use case would be to separate x86 and x64 test results.
Doing them in same stage (due to missing dynamic stages in declarative) via dotCover.
Currently I get 2 results for each and cannot tell if it is x64 or x86
Jakub Pawlinski
added a comment - My use case would be to separate x86 and x64 test results.
Doing them in same stage (due to missing dynamic stages in declarative) via dotCover.
Currently I get 2 results for each and cannot tell if it is x64 or x86 How you distinguish the test cases x86 from x64? Based on report path ?
Only by reading log at the moment I assumed I will find solution when I will reach it, but found
JENKINS-45826 instead and that brought me here
Jakub Pawlinski
added a comment - - edited Only by reading log at the moment I assumed I will find solution when I will reach it, but found JENKINS-45826 instead and that brought me here 
Nils Carlson
added a comment - Has there been any movement on this issue? We have use-cases for this as well, regarding the possibility of categorizing component and unit tests for example.
Nils Carlson
added a comment - Has there been any movement on this issue? We have use-cases for this as well, regarding the possibility of categorizing component and unit tests for example. No, this has to be condered suspended since a rewrite of the report render is needed. This also could mean do not use anymore the junit plugin features.
As I had mentioned in JENKINS-49322, the reason I've set the initial priority of this improvement request to major is because at the moment if one has a multi-stage pipeline build that produces multiple xUnit test reports, the behavior is essentially broken. Any 2 test suites that happen to have the same test names (which has happened to us in production btw) run by the same pipeline build have name clashes in the xunit publisher. Similarly, when one has multiple test stages in a single pipeline, and each of which has hundreds or thousands of tests in them, it becomes prohibitively difficult to sift through the test reports to find out which parts of the test reports apply to which build stages, making debugging difficult if not impossible. This becomes even more of a problem in practice when multiple teams are working on different test suites, because then it is hard to assign blame to a specific team when certain tests fail.