I didn't notice this ticket until after I wrote https://github.com/jenkinsci/junit-plugin/pull/27 and https://github.com/jenkinsci/parallel-test-executor-plugin/pull/16

My suggested approach doesn't provide any visual difference in the test report, since suite names isn't rendered anywhere iiuc. But it serves a way for other parts of the flow to use that info, like the parallel test executor.

One possibility is to look up the nearest enclosing LabelAction and automatically include that in the test result metadata. Thus running JUnitResultArchiver inside label (JENKINS-26107) or a parallel branch would suffice to differentiate ambiguous test names.

Rendering results in a matrix requires more information. Perhaps the convention could be that a label in a format acceptable to hudson.matrix.Combination.fromString (for example, os=win,jdk=7) would be eligible for display by any supporting visualizations in a matrix cell.

The other possibility I see is to attach a new TestAction to the FlowNode executing/archiving the tests (preferably something that is general enough to apply beyond JUnit), which includes some metadata about tests & parameters. When scanning the flowgraph to generate UI rendering, nodes with these actions can be discovered and aggregated to the appropriate stage/labelled block/whatever.

This has a couple advantages:

1. It's simple to implement and fast at runtime, since only the node in question is touched, and you don't need to scan the FlowGraph every time to get the enclosing label/stage (a process than can be unpredictable slow and can trigger significant I/O)
2. It promotes JUnit/Test result archiving to a proper first-class pipeline step
3. Makes it easy to tie JUnit results to a specific point in time for a pipeline, with all the traceability that implies (i.e. ability to map out the steps preceding it, see the node label used, etc)

Yeah, I'd like to see something significantly more generic than just junit - or just test reports, for that matter. Artifacts, static analysis, etc all would be applicable - basically, any Recorder.

It is an interesting proposal but it has one major disadvantage: that it requires Pipeline-specific code in the junit plugin, or some new bridge plugin. The more straightforward proposal, which is just to archive test results but include an associated label (as in rsandell’s original patch), requires only minor modifications to junit. Ditto for other recorders.

I do not see any reason to think performance is a consideration here.

Making test archiving be a first-class step is already tracked in JENKINS-34148, which would be useful for example for scripts which want to inspect results without necessarily changing the overall build result, but I not think that is really what we need here. There is basic UI in the junit plugin which is responsible for displaying test results, and this has a corresponding API which a lot of other plugins need to refer to. Making a Pipeline-specific storage system and API would allow the stage view to show test results, with a total fork of the (rather complex!) JUnit UI, but would throw out integration with any of these others. This does not feel like a maintainable direction.

Note that one way or the other, we'll need to make changes to Junit and other similar plugins to make sure they don't just store all their data in a single per-run hardcoded file.

To capture discussion with abayer my proposal has been revised some: currently I think the best solution is to create a new Step for JUnit or other test results, but not attach an action to the FlowNode generated by this step exection. Instead, when run, the step will attach a reference to the FlowNodeId in the TestResult. Thus, each test result will be assignable to the exact point within the run where it was generated.

Implementation-side, this is what I'm thinking:

1. New step (not just a wrapper) for JUnit result archiving.
2. The step.run method in its execution will invoke a new version of the JUnitResultArchiver.perform method, which takes a (nullable) FlowNodeId parameter or similar.
3. The existing API will be modified to invoke the new API using a null FlowNodeId.
4. The test results will get a (nullable) field for the flowNodeId, probably on the TestAction or TestResult action. This allows us to associate the result with the time of generation.
5. The new JUnitResultArchiver perform method will set this flowNodeId field to store the information.
6. For non-pipeline builds, or archived results before this, we need to handle null values (and allow serializing/deserializing without the field set).
7. If we can find a good way, JUnitResultArchiver may will to look to see if it is inside a pipeline run and attach this field when the old perform API is called, by looking at the current heads for a FlowNode that might be an achiving step.

UI representation here TBD.

It is possibly worth calling the flowNodeId field something else, to allow for other uses and not make it 100% pipeline specific. I am not sure.