[JENKINS-27395] distinguish test reports filed from different parts of workflow

Jesse Glick added a comment - 2016-04-08 19:48

One possibility is to look up the nearest enclosing LabelAction and automatically include that in the test result metadata. Thus running JUnitResultArchiver inside label (~~JENKINS-26107~~) or a parallel branch would suffice to differentiate ambiguous test names.

Rendering results in a matrix requires more information. Perhaps the convention could be that a label in a format acceptable to hudson.matrix.Combination.fromString (for example, os=win,jdk=7) would be eligible for display by any supporting visualizations in a matrix cell.

Jesse Glick added a comment - 2016-04-08 19:48 One possibility is to look up the nearest enclosing LabelAction and automatically include that in the test result metadata. Thus running JUnitResultArchiver inside label ( JENKINS-26107 ) or a parallel branch would suffice to differentiate ambiguous test names. Rendering results in a matrix requires more information. Perhaps the convention could be that a label in a format acceptable to hudson.matrix.Combination.fromString (for example, os=win,jdk=7 ) would be eligible for display by any supporting visualizations in a matrix cell.

Sam Van Oort added a comment - 2016-04-27 18:32 - edited

The other possibility I see is to attach a new TestAction to the FlowNode executing/archiving the tests (preferably something that is general enough to apply beyond JUnit), which includes some metadata about tests & parameters. When scanning the flowgraph to generate UI rendering, nodes with these actions can be discovered and aggregated to the appropriate stage/labelled block/whatever.

This has a couple advantages:

It's simple to implement and fast at runtime, since only the node in question is touched, and you don't need to scan the FlowGraph every time to get the enclosing label/stage (a process than can be unpredictable slow and can trigger significant I/O)
It promotes JUnit/Test result archiving to a proper first-class pipeline step
Makes it easy to tie JUnit results to a specific point in time for a pipeline, with all the traceability that implies (i.e. ability to map out the steps preceding it, see the node label used, etc)

Sam Van Oort added a comment - 2016-04-27 18:32 - edited The other possibility I see is to attach a new TestAction to the FlowNode executing/archiving the tests (preferably something that is general enough to apply beyond JUnit), which includes some metadata about tests & parameters. When scanning the flowgraph to generate UI rendering, nodes with these actions can be discovered and aggregated to the appropriate stage/labelled block/whatever. This has a couple advantages: It's simple to implement and fast at runtime, since only the node in question is touched, and you don't need to scan the FlowGraph every time to get the enclosing label/stage (a process than can be unpredictable slow and can trigger significant I/O) It promotes JUnit/Test result archiving to a proper first-class pipeline step Makes it easy to tie JUnit results to a specific point in time for a pipeline, with all the traceability that implies (i.e. ability to map out the steps preceding it, see the node label used, etc)

Andrew Bayer added a comment - 2016-04-27 18:33

Yeah, I'd like to see something significantly more generic than just junit - or just test reports, for that matter. Artifacts, static analysis, etc all would be applicable - basically, any Recorder.

Andrew Bayer added a comment - 2016-04-27 18:33 Yeah, I'd like to see something significantly more generic than just junit - or just test reports, for that matter. Artifacts, static analysis, etc all would be applicable - basically, any Recorder .

Jesse Glick added a comment - 2016-05-02 14:59

It is an interesting proposal but it has one major disadvantage: that it requires Pipeline-specific code in the junit plugin, or some new bridge plugin. The more straightforward proposal, which is just to archive test results but include an associated label (as in rsandell’s original patch), requires only minor modifications to junit. Ditto for other recorders.

I do not see any reason to think performance is a consideration here.

Making test archiving be a first-class step is already tracked in ~~JENKINS-34148~~, which would be useful for example for scripts which want to inspect results without necessarily changing the overall build result, but I not think that is really what we need here. There is basic UI in the junit plugin which is responsible for displaying test results, and this has a corresponding API which a lot of other plugins need to refer to. Making a Pipeline-specific storage system and API would allow the stage view to show test results, with a total fork of the (rather complex!) JUnit UI, but would throw out integration with any of these others. This does not feel like a maintainable direction.

Jesse Glick added a comment - 2016-05-02 14:59 It is an interesting proposal but it has one major disadvantage: that it requires Pipeline-specific code in the junit plugin, or some new bridge plugin. The more straightforward proposal, which is just to archive test results but include an associated label (as in rsandell ’s original patch), requires only minor modifications to junit . Ditto for other recorders. I do not see any reason to think performance is a consideration here. Making test archiving be a first-class step is already tracked in JENKINS-34148 , which would be useful for example for scripts which want to inspect results without necessarily changing the overall build result, but I not think that is really what we need here. There is basic UI in the junit plugin which is responsible for displaying test results, and this has a corresponding API which a lot of other plugins need to refer to. Making a Pipeline-specific storage system and API would allow the stage view to show test results, with a total fork of the (rather complex!) JUnit UI, but would throw out integration with any of these others. This does not feel like a maintainable direction.

Andrew Bayer added a comment - 2016-05-02 15:09

Note that one way or the other, we'll need to make changes to Junit and other similar plugins to make sure they don't just store all their data in a single per-run hardcoded file.

Andrew Bayer added a comment - 2016-05-02 15:09 Note that one way or the other, we'll need to make changes to Junit and other similar plugins to make sure they don't just store all their data in a single per-run hardcoded file.

Sam Van Oort added a comment - 2016-05-02 15:11

To capture discussion with abayer my proposal has been revised some: currently I think the best solution is to create a new Step for JUnit or other test results, but not attach an action to the FlowNode generated by this step exection. Instead, when run, the step will attach a reference to the FlowNodeId in the TestResult. Thus, each test result will be assignable to the exact point within the run where it was generated.

Implementation-side, this is what I'm thinking:

1. New step (not just a wrapper) for JUnit result archiving.
2. The step.run method in its execution will invoke a new version of the JUnitResultArchiver.perform method, which takes a (nullable) FlowNodeId parameter or similar.
3. The existing API will be modified to invoke the new API using a null FlowNodeId.
4. The test results will get a (nullable) field for the flowNodeId, probably on the TestAction or TestResult action. This allows us to associate the result with the time of generation.
5. The new JUnitResultArchiver perform method will set this flowNodeId field to store the information.
6. For non-pipeline builds, or archived results before this, we need to handle null values (and allow serializing/deserializing without the field set).
7. If we can find a good way, JUnitResultArchiver may will to look to see if it is inside a pipeline run and attach this field when the old perform API is called, by looking at the current heads for a FlowNode that might be an achiving step.

UI representation here TBD.

It is possibly worth calling the flowNodeId field something else, to allow for other uses and not make it 100% pipeline specific. I am not sure.

Sam Van Oort added a comment - 2016-05-02 15:11 To capture discussion with abayer my proposal has been revised some: currently I think the best solution is to create a new Step for JUnit or other test results, but not attach an action to the FlowNode generated by this step exection. Instead, when run, the step will attach a reference to the FlowNodeId in the TestResult. Thus, each test result will be assignable to the exact point within the run where it was generated. Implementation-side, this is what I'm thinking: 1. New step (not just a wrapper) for JUnit result archiving. 2. The step.run method in its execution will invoke a new version of the JUnitResultArchiver.perform method, which takes a (nullable) FlowNodeId parameter or similar. 3. The existing API will be modified to invoke the new API using a null FlowNodeId. 4. The test results will get a (nullable) field for the flowNodeId, probably on the TestAction or TestResult action. This allows us to associate the result with the time of generation. 5. The new JUnitResultArchiver perform method will set this flowNodeId field to store the information. 6. For non-pipeline builds, or archived results before this, we need to handle null values (and allow serializing/deserializing without the field set). 7. If we can find a good way, JUnitResultArchiver may will to look to see if it is inside a pipeline run and attach this field when the old perform API is called, by looking at the current heads for a FlowNode that might be an achiving step. UI representation here TBD. It is possibly worth calling the flowNodeId field something else, to allow for other uses and not make it 100% pipeline specific. I am not sure.

Sam Van Oort added a comment - 2016-05-02 17:02

jglick You may want to see the updated comment (since your comment applies to the old plan); I think with the revised proposal, the only parts that require pipeline dependencies are the step itself, and if we choose to automatically store the flowNodeId when the JUnitResultArchiver is invoked with the original perform API (such as in some current scripts).

I think the PR that rsandell put forth to include an archiveId has a lot in common with this (although the internals seem to differ a bit).

Sam Van Oort added a comment - 2016-05-02 17:02 jglick You may want to see the updated comment (since your comment applies to the old plan); I think with the revised proposal, the only parts that require pipeline dependencies are the step itself, and if we choose to automatically store the flowNodeId when the JUnitResultArchiver is invoked with the original perform API (such as in some current scripts). I think the PR that rsandell put forth to include an archiveId has a lot in common with this (although the internals seem to differ a bit).

Andres Rodriguez added a comment - 2016-05-04 22:39 - edited

For anyone interested in this functionality, here is a small hack I use to achieve something similar:

def setJUnitPackageName(packageName, inFile, outFile) {
    // Prepend the testName as the package name so that we get sorted output in the
    // Jenkins test results page. The packageName is extracted from the classname
    // attribute of the testcase tag.
    // WARNING: the package attribute of the testcase tag is igonred by Jenkins
    sh "sed \"s/\\(classname=['\\\"]\\)/\\1${packageName}./g\" ${inFile} > ${outFile}"
}

With this you can set packageName of a junit file to a string of your choice, we usually use "test-name-platform" for example.

Andres Rodriguez added a comment - 2016-05-04 22:39 - edited For anyone interested in this functionality, here is a small hack I use to achieve something similar: def setJUnitPackageName(packageName, inFile, outFile) { // Prepend the testName as the package name so that we get sorted output in the // Jenkins test results page. The packageName is extracted from the classname // attribute of the testcase tag. // WARNING: the package attribute of the testcase tag is igonred by Jenkins sh "sed \" s/\$classname=['\\\ "]\$/\\1${packageName}./g\" ${inFile} > ${outFile}" } With this you can set packageName of a junit file to a string of your choice, we usually use "test-name-platform" for example.

Reinhold Füreder added a comment - 2017-07-04 12:00

Some thoughts:

Should all test results from potentially (or now with Jenkins2/Pipeline very likely) different test types (unit vs. integration vs. acceptance tests etc) really be merged into one single test result like it seems to be done now?
Other CI Servers allow (i.e. this is optional) giving each test result publication a dedicated title name (e.g. "Unit Tests" vs. "Firefox99 Acceptance Tests"): would it make sense to merge only based on test result publication title and show also stats and graphs per that (test result publication title)?
(I think a test report filename based test result publication title naming approach is not nice)

Reinhold Füreder added a comment - 2017-07-04 12:00 Some thoughts: Should all test results from potentially (or now with Jenkins2/Pipeline very likely) different test types (unit vs. integration vs. acceptance tests etc) really be merged into one single test result like it seems to be done now? Other CI Servers allow (i.e. this is optional) giving each test result publication a dedicated title name (e.g. "Unit Tests" vs. "Firefox99 Acceptance Tests"): would it make sense to merge only based on test result publication title and show also stats and graphs per that (test result publication title)? (I think a test report filename based test result publication title naming approach is not nice)

Jesse Glick added a comment - 2017-07-04 15:39

PR 64 is the most recent proposal, but I think it is more a starting point than a complete fix.

Jesse Glick added a comment - 2017-07-04 15:39 PR 64 is the most recent proposal, but I think it is more a starting point than a complete fix.

Andrew Bayer added a comment - 2017-07-26 17:03

Veeeerrrry preliminary work in progress PR up at https://github.com/jenkinsci/junit-plugin/pull/75 - it subsumes https://github.com/jenkinsci/junit-plugin/pull/64, and isn't anywhere near ready yet, but I wanted to get the initial work up for discussion as I move forward.

Andrew Bayer added a comment - 2017-07-26 17:03 Veeeerrrry preliminary work in progress PR up at https://github.com/jenkinsci/junit-plugin/pull/75 - it subsumes https://github.com/jenkinsci/junit-plugin/pull/64 , and isn't anywhere near ready yet, but I wanted to get the initial work up for discussion as I move forward.

Andrew Bayer added a comment - 2017-07-26 21:53

Ditched the first PR that built on top of #64, which I wasn't happy with, in favor of https://github.com/jenkinsci/junit-plugin/pull/76.

Andrew Bayer added a comment - 2017-07-26 21:53 Ditched the first PR that built on top of #64, which I wasn't happy with, in favor of https://github.com/jenkinsci/junit-plugin/pull/76 .

Cyrille Le Clerc added a comment - 2017-07-26 22:17

abayer will your enhancement support the case where one single step generates multiple test reports that we want to associate with different names?
I have in mind a multi-module Maven project for which I would like to have per module test reports: a single "sh 'mvn test' " step would generate multiple test reports that I want to attach with their maven module name/identifier.

See comment https://github.com/jenkinsci/junit-plugin/pull/64#issuecomment-277625409

Cyrille Le Clerc added a comment - 2017-07-26 22:17 abayer will your enhancement support the case where one single step generates multiple test reports that we want to associate with different names? I have in mind a multi-module Maven project for which I would like to have per module test reports: a single "sh 'mvn test' " step would generate multiple test reports that I want to attach with their maven module name/identifier. See comment https://github.com/jenkinsci/junit-plugin/pull/64#issuecomment-277625409

Andrew Bayer added a comment - 2017-07-26 22:29

cleclerc So sh 'mvn test' wouldn't actually generate any test reports in the job, either now or in the future. It's the junit (or my proposed junitResults step) invocation that does that. So if, with my proposal, you ran sh 'mvn test' and then junitResults "first-module/target/surefire-reports/*/.xml" and junitResults "second-module/target/surefire-reports/*/.xml", that'd be two junitResults invocations, so whatever's displaying the tests could say "Ok, what are the test results for this step or this stage?" and get the individual execution's results. But yeah, overall, I've decided to separate this from ~~JENKINS-41710~~ - they're different matters.

Andrew Bayer added a comment - 2017-07-26 22:29 cleclerc So sh 'mvn test' wouldn't actually generate any test reports in the job, either now or in the future. It's the junit (or my proposed junitResults step) invocation that does that. So if, with my proposal, you ran sh 'mvn test' and then junitResults "first-module/target/surefire-reports/* / .xml" and junitResults "second-module/target/surefire-reports/* / .xml" , that'd be two junitResults invocations, so whatever's displaying the tests could say "Ok, what are the test results for this step or this stage?" and get the individual execution's results. But yeah, overall, I've decided to separate this from JENKINS-41710 - they're different matters.

Cyrille Le Clerc added a comment - 2017-07-26 23:01

abayer I forgot to mention that I publish the test reports with the withMaven plugin, in org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback.TailCall#finished (here). See sample below.

This means that a single step publishes multiple test reports.

node {
    stage ('Build') {
        git "https://github.com/cyrille-leclerc/multi-module-maven-project.git"
        withMaven(maven: 'maven-3.3.9') {
           sh "mvn clean package"
        }
    }
}

> But yeah, overall, I've decided to separate this from ~~JENKINS-41710~~ - they're different matters.

Does it mean that https://github.com/jenkinsci/junit-plugin/pull/64 is no longer blocked and can be merged?

Cyrille Le Clerc added a comment - 2017-07-26 23:01 abayer I forgot to mention that I publish the test reports with the withMaven plugin, in org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback.TailCall#finished ( here ). See sample below. This means that a single step publishes multiple test reports. node { stage ( 'Build' ) { git "https: //github.com/cyrille-leclerc/multi-module-maven-project.git" withMaven(maven: 'maven-3.3.9' ) { sh "mvn clean package " } } } > But yeah, overall, I've decided to separate this from JENKINS-41710 - they're different matters. Does it mean that https://github.com/jenkinsci/junit-plugin/pull/64 is no longer blocked and can be merged?

Andrew Bayer added a comment - 2017-07-26 23:13

This is orthogonal to https://github.com/jenkinsci/junit-plugin/pull/64. And it would have no impact on the withMaven step at all, unless/until you change it to use the static method introduced for parsing and attaching test results rather than https://github.com/jenkinsci/pipeline-maven-plugin/blob/2eba9c8ac50f933567d6b3775ca4b3372e9f1d50/jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/publishers/JunitTestsPublisher.java#L268.

Andrew Bayer added a comment - 2017-07-26 23:13 This is orthogonal to https://github.com/jenkinsci/junit-plugin/pull/64 . And it would have no impact on the withMaven step at all, unless/until you change it to use the static method introduced for parsing and attaching test results rather than https://github.com/jenkinsci/pipeline-maven-plugin/blob/2eba9c8ac50f933567d6b3775ca4b3372e9f1d50/jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/publishers/JunitTestsPublisher.java#L268 .

Andrew Bayer added a comment - 2017-07-27 17:37

XUnit plugin PR up at https://github.com/jenkinsci/xunit-plugin/pull/49 that builds on top of the JUnit one...

Andrew Bayer added a comment - 2017-07-27 17:37 XUnit plugin PR up at https://github.com/jenkinsci/xunit-plugin/pull/49 that builds on top of the JUnit one...

Cyrille Le Clerc added a comment - 2017-07-28 09:47

abayer thanks for the explanation, I'll try my best to contribute to PR-64 and make it happen.

Cyrille Le Clerc added a comment - 2017-07-28 09:47 abayer thanks for the explanation, I'll try my best to contribute to PR-64 and make it happen.

James Dumay added a comment - 2017-08-14 01:15 - edited

Hi all,

We have a design that will differentiate tests across different stages and parallels for Blue Ocean as part of ~~JENKINS-46166~~.

Please take a look and let us know what you think.

Thanks
James

James Dumay added a comment - 2017-08-14 01:15 - edited Hi all, We have a design that will differentiate tests across different stages and parallels for Blue Ocean as part of JENKINS-46166 . Please take a look and let us know what you think. Thanks James

Andrew Bayer added a comment - 2017-08-22 14:47

Added a PR to pipeline-maven (https://github.com/jenkinsci/pipeline-maven-plugin/pull/82) that uses the junit changes as well.

Andrew Bayer added a comment - 2017-08-22 14:47 Added a PR to pipeline-maven ( https://github.com/jenkinsci/pipeline-maven-plugin/pull/82 ) that uses the junit changes as well.

Jesse Glick added a comment - 2017-08-24 16:42

abayer I have not been following details of what you are doing, but you need to verify that parallel-test-executor is able to take advantage of this metadata—it is a critical use case.

Jesse Glick added a comment - 2017-08-24 16:42 abayer I have not been following details of what you are doing, but you need to verify that parallel-test-executor is able to take advantage of this metadata—it is a critical use case.

Andrew Bayer added a comment - 2017-08-24 18:02

Looks like parallel-test-executor will need some work to be able to make any use of the additional data - theoretically, it could be reworked to, say, find just the tests for a specific stage in the previous build, which would work pretty easily.

Andrew Bayer added a comment - 2017-08-24 18:02 Looks like parallel-test-executor will need some work to be able to make any use of the additional data - theoretically, it could be reworked to, say, find just the tests for a specific stage in the previous build, which would work pretty easily.

Jesse Glick added a comment - 2017-08-24 19:49

theoretically, it could be reworked to, say, find just the tests for a specific stage in the previous build

Theory becomes practice when you file the PR.

Jesse Glick added a comment - 2017-08-24 19:49 theoretically, it could be reworked to, say, find just the tests for a specific stage in the previous build Theory becomes practice when you file the PR.

Andrew Bayer added a comment - 2017-08-24 19:56

Indeed. I've started playing around with it. Trying to figure out the best way to specify a parallel branch within a stage (or possibly an arbitrarily deep nest of stages/parallel branches) at the moment. I might wait until svanoort has nailed down more performant querying for enclosing flow nodes (as discussed on the junit PR) and instead have an option that lets you just say "use the tests from this stage/branch in the previous build" without having to specify the full "path" directly.

Andrew Bayer added a comment - 2017-08-24 19:56 Indeed. I've started playing around with it. Trying to figure out the best way to specify a parallel branch within a stage (or possibly an arbitrarily deep nest of stages/parallel branches) at the moment. I might wait until svanoort has nailed down more performant querying for enclosing flow nodes (as discussed on the junit PR) and instead have an option that lets you just say "use the tests from this stage/branch in the previous build" without having to specify the full "path" directly.

SCM/JIRA link daemon added a comment - 2017-09-22 17:49

Code changed in jenkins
User: Andrew Bayer
Path:
src/main/java/org/jenkinsci/plugins/workflow/graph/AtomNode.java
src/main/java/org/jenkinsci/plugins/workflow/graph/BlockEndNode.java
src/main/java/org/jenkinsci/plugins/workflow/graph/BlockStartNode.java
src/main/java/org/jenkinsci/plugins/workflow/graph/FlowNode.java
src/test/java/org/jenkinsci/plugins/workflow/graph/FlowNodeTest.java
http://jenkins-ci.org/commit/workflow-api-plugin/3e8a1b79b07936131098267664f441335318d4e1
Log:
~~JENKINS-27395~~ Start recording enclosing blocks for flow nodes

Specifically for AtomNodes, BlockStartNodes, and BlockEndNodes.

SCM/JIRA link daemon added a comment - 2017-09-22 17:49 Code changed in jenkins User: Andrew Bayer Path: src/main/java/org/jenkinsci/plugins/workflow/graph/AtomNode.java src/main/java/org/jenkinsci/plugins/workflow/graph/BlockEndNode.java src/main/java/org/jenkinsci/plugins/workflow/graph/BlockStartNode.java src/main/java/org/jenkinsci/plugins/workflow/graph/FlowNode.java src/test/java/org/jenkinsci/plugins/workflow/graph/FlowNodeTest.java http://jenkins-ci.org/commit/workflow-api-plugin/3e8a1b79b07936131098267664f441335318d4e1 Log: JENKINS-27395 Start recording enclosing blocks for flow nodes Specifically for AtomNodes, BlockStartNodes, and BlockEndNodes.

Sam Van Oort added a comment - 2017-09-22 21:14

abayer Not sure exactly what you're trying to do?

I had a semi-performant impl for fetching nodes within a block (using something I put into Bismuth ages ago but hadn't fully taken advantage of). Removed it because no downstream consumer yet but I can easily put it back.

Only thing is that in the face of parallels the order of nodes wrt branches is pretty ill defined.

Sam Van Oort added a comment - 2017-09-22 21:14 abayer Not sure exactly what you're trying to do? I had a semi-performant impl for fetching nodes within a block (using something I put into Bismuth ages ago but hadn't fully taken advantage of). Removed it because no downstream consumer yet but I can easily put it back. Only thing is that in the face of parallels the order of nodes wrt branches is pretty ill defined.

Andrew Bayer added a comment - 2017-09-22 21:15

I...totally didn't do anything?

Andrew Bayer added a comment - 2017-09-22 21:15 I...totally didn't do anything?

Sam Van Oort added a comment - 2017-09-26 22:28

abayer I think the upstreams are now unblocked for you when you get back online.

Sam Van Oort added a comment - 2017-09-26 22:28 abayer I think the upstreams are now unblocked for you when you get back online.

SCM/JIRA link daemon added a comment - 2017-10-17 07:34

Code changed in jenkins
User: Andrew Bayer
Path:
pom.xml
src/main/java/hudson/tasks/junit/JUnitParser.java
src/main/java/hudson/tasks/junit/JUnitResultArchiver.java
src/main/java/hudson/tasks/junit/JUnitTask.java
src/main/java/hudson/tasks/junit/SuiteResult.java
src/main/java/hudson/tasks/junit/TestResult.java
src/main/java/hudson/tasks/junit/pipeline/JUnitResultsStep.java
src/main/java/hudson/tasks/junit/pipeline/JUnitResultsStepExecution.java
src/main/java/hudson/tasks/test/TestResultParser.java
src/test/java/hudson/tasks/junit/JUnitParserTest.java
src/test/java/hudson/tasks/junit/JUnitResultArchiverTest.java
src/test/java/hudson/tasks/junit/SuiteResult2Test.java
src/test/java/hudson/tasks/junit/SuiteResultTest.java
src/test/java/hudson/tasks/junit/TestResultTest.java
src/test/java/hudson/tasks/junit/pipeline/JUnitResultsStepTest.java
src/test/resources/hudson/tasks/junit/JUnitResultArchiverTest.zip
src/test/resources/hudson/tasks/junit/JUnitResultArchiverTest/testData.zip
http://jenkins-ci.org/commit/junit-plugin/375f91d78da541fec0f095d996619d5e7cdfc18f
Log:
~~JENKINS-27395~~ Add a junitResults step

This uses the run's externalizable ID and the step's FlowNode.getId()
together as a unique key to track which SuiteResults are associated
with a particular execution, and allows getting a TestResult object
from a run ID and 1 or more node IDs passed to an existing TestResult
object that contains suites for that run/those nodes.

Note that JUnitResultArchiverTest.zip has been moved around and
renamed due to
https://github.com/jenkinsci/jenkins-test-harness/commit/79d9603370d736fa1ede35e78b5f4c421cf2d5be
breaking tests otherwise, since the JUnitResultArchiveTest/ directory
will end up getting picked up as JENKINS_HOME otherwise.

SCM/JIRA link daemon added a comment - 2017-10-17 07:34 Code changed in jenkins User: Andrew Bayer Path: pom.xml src/main/java/hudson/tasks/junit/JUnitParser.java src/main/java/hudson/tasks/junit/JUnitResultArchiver.java src/main/java/hudson/tasks/junit/JUnitTask.java src/main/java/hudson/tasks/junit/SuiteResult.java src/main/java/hudson/tasks/junit/TestResult.java src/main/java/hudson/tasks/junit/pipeline/JUnitResultsStep.java src/main/java/hudson/tasks/junit/pipeline/JUnitResultsStepExecution.java src/main/java/hudson/tasks/test/TestResultParser.java src/test/java/hudson/tasks/junit/JUnitParserTest.java src/test/java/hudson/tasks/junit/JUnitResultArchiverTest.java src/test/java/hudson/tasks/junit/SuiteResult2Test.java src/test/java/hudson/tasks/junit/SuiteResultTest.java src/test/java/hudson/tasks/junit/TestResultTest.java src/test/java/hudson/tasks/junit/pipeline/JUnitResultsStepTest.java src/test/resources/hudson/tasks/junit/JUnitResultArchiverTest.zip src/test/resources/hudson/tasks/junit/JUnitResultArchiverTest/testData.zip http://jenkins-ci.org/commit/junit-plugin/375f91d78da541fec0f095d996619d5e7cdfc18f Log: JENKINS-27395 Add a junitResults step This uses the run's externalizable ID and the step's FlowNode.getId() together as a unique key to track which SuiteResults are associated with a particular execution, and allows getting a TestResult object from a run ID and 1 or more node IDs passed to an existing TestResult object that contains suites for that run/those nodes. Note that JUnitResultArchiverTest.zip has been moved around and renamed due to https://github.com/jenkinsci/jenkins-test-harness/commit/79d9603370d736fa1ede35e78b5f4c421cf2d5be breaking tests otherwise, since the JUnitResultArchiveTest/ directory will end up getting picked up as JENKINS_HOME otherwise.

SCM/JIRA link daemon added a comment - 2017-10-17 07:34

Code changed in jenkins
User: Andrew Bayer
Path:
Jenkinsfile
pom.xml
src/main/java/hudson/tasks/junit/CaseResult.java
src/main/java/hudson/tasks/junit/JUnitParser.java
src/main/java/hudson/tasks/junit/JUnitResultArchiver.java
src/main/java/hudson/tasks/junit/JUnitTask.java
src/main/java/hudson/tasks/junit/SuiteResult.java
src/main/java/hudson/tasks/junit/TestResult.java
src/main/java/hudson/tasks/junit/TestResultAction.java
src/main/java/hudson/tasks/junit/TestResultSummary.java
src/main/java/hudson/tasks/junit/pipeline/JUnitResultsStep.java
src/main/java/hudson/tasks/junit/pipeline/JUnitResultsStepExecution.java
src/main/java/hudson/tasks/test/PipelineBlockWithTests.java
src/main/java/hudson/tasks/test/TabulatedResult.java
src/main/java/hudson/tasks/test/TestResultParser.java
src/main/resources/hudson/tasks/junit/pipeline/JUnitResultsStep/config.jelly
src/test/java/hudson/tasks/junit/JUnitParserTest.java
src/test/java/hudson/tasks/junit/JUnitResultArchiverTest.java
src/test/java/hudson/tasks/junit/SuiteResult2Test.java
src/test/java/hudson/tasks/junit/SuiteResultTest.java
src/test/java/hudson/tasks/junit/TestResultTest.java
src/test/java/hudson/tasks/junit/pipeline/JUnitResultsStepTest.java
src/test/resources/hudson/tasks/junit/pipeline/junit-report-testTrends-first-1.xml
src/test/resources/hudson/tasks/junit/pipeline/junit-report-testTrends-first-2.xml
src/test/resources/hudson/tasks/junit/pipeline/junit-report-testTrends-first-3.xml
src/test/resources/hudson/tasks/junit/pipeline/junit-report-testTrends-second-1.xml
src/test/resources/hudson/tasks/junit/pipeline/junit-report-testTrends-second-2.xml
src/test/resources/hudson/tasks/junit/pipeline/junit-report-testTrends-second-3.xml
http://jenkins-ci.org/commit/junit-plugin/5ae9ec10480d92075f03944462d8a90c8755ed86
Log:
Merge pull request #76 from abayer/jenkins-27395-mk2

~~JENKINS-27395~~ Add a junitResults step

Compare: https://github.com/jenkinsci/junit-plugin/compare/187cb92178f0...5ae9ec10480d

SCM/JIRA link daemon added a comment - 2017-10-17 07:34 Code changed in jenkins User: Andrew Bayer Path: Jenkinsfile pom.xml src/main/java/hudson/tasks/junit/CaseResult.java src/main/java/hudson/tasks/junit/JUnitParser.java src/main/java/hudson/tasks/junit/JUnitResultArchiver.java src/main/java/hudson/tasks/junit/JUnitTask.java src/main/java/hudson/tasks/junit/SuiteResult.java src/main/java/hudson/tasks/junit/TestResult.java src/main/java/hudson/tasks/junit/TestResultAction.java src/main/java/hudson/tasks/junit/TestResultSummary.java src/main/java/hudson/tasks/junit/pipeline/JUnitResultsStep.java src/main/java/hudson/tasks/junit/pipeline/JUnitResultsStepExecution.java src/main/java/hudson/tasks/test/PipelineBlockWithTests.java src/main/java/hudson/tasks/test/TabulatedResult.java src/main/java/hudson/tasks/test/TestResultParser.java src/main/resources/hudson/tasks/junit/pipeline/JUnitResultsStep/config.jelly src/test/java/hudson/tasks/junit/JUnitParserTest.java src/test/java/hudson/tasks/junit/JUnitResultArchiverTest.java src/test/java/hudson/tasks/junit/SuiteResult2Test.java src/test/java/hudson/tasks/junit/SuiteResultTest.java src/test/java/hudson/tasks/junit/TestResultTest.java src/test/java/hudson/tasks/junit/pipeline/JUnitResultsStepTest.java src/test/resources/hudson/tasks/junit/pipeline/junit-report-testTrends-first-1.xml src/test/resources/hudson/tasks/junit/pipeline/junit-report-testTrends-first-2.xml src/test/resources/hudson/tasks/junit/pipeline/junit-report-testTrends-first-3.xml src/test/resources/hudson/tasks/junit/pipeline/junit-report-testTrends-second-1.xml src/test/resources/hudson/tasks/junit/pipeline/junit-report-testTrends-second-2.xml src/test/resources/hudson/tasks/junit/pipeline/junit-report-testTrends-second-3.xml http://jenkins-ci.org/commit/junit-plugin/5ae9ec10480d92075f03944462d8a90c8755ed86 Log: Merge pull request #76 from abayer/jenkins-27395-mk2 JENKINS-27395 Add a junitResults step Compare: https://github.com/jenkinsci/junit-plugin/compare/187cb92178f0...5ae9ec10480d

Andrew Bayer added a comment - 2017-10-17 07:36

Releasing as 1.22-beta-1 - plan is to follow that up with an xunit beta and a parallel-test-executor PR...

Andrew Bayer added a comment - 2017-10-17 07:36 Releasing as 1.22-beta-1 - plan is to follow that up with an xunit beta and a parallel-test-executor PR...

SCM/JIRA link daemon added a comment - 2017-11-02 21:18

Code changed in jenkins
User: Andrew Bayer
Path:
src/main/java/hudson/tasks/junit/CaseResult.java
src/main/java/hudson/tasks/junit/JUnitParser.java
src/main/java/hudson/tasks/junit/JUnitResultArchiver.java
src/main/java/hudson/tasks/junit/SuiteResult.java
src/main/java/hudson/tasks/junit/TestResult.java
src/main/java/hudson/tasks/junit/pipeline/JUnitResultsStepExecution.java
src/main/java/hudson/tasks/test/TabulatedResult.java
src/main/java/hudson/tasks/test/TestResultParser.java
src/test/java/hudson/tasks/junit/JUnitParserTest.java
src/test/java/hudson/tasks/junit/JUnitResultArchiverTest.java
src/test/java/hudson/tasks/junit/pipeline/JUnitResultsStepTest.java
http://jenkins-ci.org/commit/junit-plugin/147692d8717493b17cc8ebf4a3a73256e1029879
Log:
~~JENKINS-27395~~ Removed per-run node tracking

On further inspection, I came to the conclusion that since
multiple-runs-per-TestResult was only possible for
`AggregatedTestResultAction`, which doesn't support Pipeline jobs in
the first place, it made more sense to simplify things and get rid of
the run-and-nodes tracking of suites in favor of just nodes.

This also necessitated gathreing the enclosing block names earlier,
but that's probably a good change anyway.

SCM/JIRA link daemon added a comment - 2017-11-02 21:18 Code changed in jenkins User: Andrew Bayer Path: src/main/java/hudson/tasks/junit/CaseResult.java src/main/java/hudson/tasks/junit/JUnitParser.java src/main/java/hudson/tasks/junit/JUnitResultArchiver.java src/main/java/hudson/tasks/junit/SuiteResult.java src/main/java/hudson/tasks/junit/TestResult.java src/main/java/hudson/tasks/junit/pipeline/JUnitResultsStepExecution.java src/main/java/hudson/tasks/test/TabulatedResult.java src/main/java/hudson/tasks/test/TestResultParser.java src/test/java/hudson/tasks/junit/JUnitParserTest.java src/test/java/hudson/tasks/junit/JUnitResultArchiverTest.java src/test/java/hudson/tasks/junit/pipeline/JUnitResultsStepTest.java http://jenkins-ci.org/commit/junit-plugin/147692d8717493b17cc8ebf4a3a73256e1029879 Log: JENKINS-27395 Removed per-run node tracking On further inspection, I came to the conclusion that since multiple-runs-per-TestResult was only possible for `AggregatedTestResultAction`, which doesn't support Pipeline jobs in the first place, it made more sense to simplify things and get rid of the run-and-nodes tracking of suites in favor of just nodes. This also necessitated gathreing the enclosing block names earlier, but that's probably a good change anyway.

SCM/JIRA link daemon added a comment - 2017-11-02 21:18

Code changed in jenkins
User: Andrew Bayer
Path:
src/main/java/hudson/tasks/junit/CaseResult.java
src/main/java/hudson/tasks/junit/JUnitParser.java
src/main/java/hudson/tasks/junit/JUnitResultArchiver.java
src/main/java/hudson/tasks/junit/SuiteResult.java
src/main/java/hudson/tasks/junit/TestResult.java
src/main/java/hudson/tasks/junit/pipeline/JUnitResultsStepExecution.java
src/main/java/hudson/tasks/test/PipelineTestDetails.java
src/main/java/hudson/tasks/test/TabulatedResult.java
src/main/java/hudson/tasks/test/TestResultParser.java
src/test/java/hudson/tasks/junit/JUnitParserTest.java
src/test/java/hudson/tasks/junit/JUnitResultArchiverTest.java
src/test/java/hudson/tasks/junit/SuiteResult2Test.java
src/test/java/hudson/tasks/junit/SuiteResultTest.java
src/test/java/hudson/tasks/junit/TestResultTest.java
src/test/java/hudson/tasks/junit/pipeline/JUnitResultsStepTest.java
http://jenkins-ci.org/commit/junit-plugin/b98cb6eba3b1207a3eba70a2e0766f3e26953b4e
Log:
Merge pull request #83 from abayer/jenkins-27395-remove-run-id

~~JENKINS-27395~~ Removed per-run node tracking

Compare: https://github.com/jenkinsci/junit-plugin/compare/1c00c38bea6c...b98cb6eba3b1

SCM/JIRA link daemon added a comment - 2017-11-02 21:18 Code changed in jenkins User: Andrew Bayer Path: src/main/java/hudson/tasks/junit/CaseResult.java src/main/java/hudson/tasks/junit/JUnitParser.java src/main/java/hudson/tasks/junit/JUnitResultArchiver.java src/main/java/hudson/tasks/junit/SuiteResult.java src/main/java/hudson/tasks/junit/TestResult.java src/main/java/hudson/tasks/junit/pipeline/JUnitResultsStepExecution.java src/main/java/hudson/tasks/test/PipelineTestDetails.java src/main/java/hudson/tasks/test/TabulatedResult.java src/main/java/hudson/tasks/test/TestResultParser.java src/test/java/hudson/tasks/junit/JUnitParserTest.java src/test/java/hudson/tasks/junit/JUnitResultArchiverTest.java src/test/java/hudson/tasks/junit/SuiteResult2Test.java src/test/java/hudson/tasks/junit/SuiteResultTest.java src/test/java/hudson/tasks/junit/TestResultTest.java src/test/java/hudson/tasks/junit/pipeline/JUnitResultsStepTest.java http://jenkins-ci.org/commit/junit-plugin/b98cb6eba3b1207a3eba70a2e0766f3e26953b4e Log: Merge pull request #83 from abayer/jenkins-27395-remove-run-id JENKINS-27395 Removed per-run node tracking Compare: https://github.com/jenkinsci/junit-plugin/compare/1c00c38bea6c...b98cb6eba3b1

Andrew Bayer added a comment - 2017-11-14 17:47

I've just released 1.22, which includes this. Downstream plugins, such as xunit, parallel-test-executor and Blue Ocean should come soon.

Andrew Bayer added a comment - 2017-11-14 17:47 I've just released 1.22, which includes this. Downstream plugins, such as xunit , parallel-test-executor and Blue Ocean should come soon.

Reinhold Füreder added a comment - 2017-11-15 10:34

abayer Please see ~~JENKINS-48020~~ which IMHO popped up after upgrading to 1.22.1; besides that I am also wondering if any GUI differences should be already visible now...

Reinhold Füreder added a comment - 2017-11-15 10:34 abayer Please see JENKINS-48020 which IMHO popped up after upgrading to 1.22.1; besides that I am also wondering if any GUI differences should be already visible now...

SCM/JIRA link daemon added a comment - 2017-11-27 16:24

Code changed in jenkins
User: Andrew Bayer
Path:
pom.xml
src/main/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutor.java
src/main/java/org/jenkinsci/plugins/parallel_test_executor/PreviousTestResultLookup.java
src/main/java/org/jenkinsci/plugins/parallel_test_executor/SplitStep.java
src/test/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutorTest.java
src/test/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutorUnitTest.java
http://jenkins-ci.org/commit/parallel-test-executor-plugin/7d09e31992b5122200ee4d48c01042eda5d7acb1
Log:
~~JENKINS-27395~~ Add per-stage test splitting option

SCM/JIRA link daemon added a comment - 2017-11-27 16:24 Code changed in jenkins User: Andrew Bayer Path: pom.xml src/main/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutor.java src/main/java/org/jenkinsci/plugins/parallel_test_executor/PreviousTestResultLookup.java src/main/java/org/jenkinsci/plugins/parallel_test_executor/SplitStep.java src/test/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutorTest.java src/test/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutorUnitTest.java http://jenkins-ci.org/commit/parallel-test-executor-plugin/7d09e31992b5122200ee4d48c01042eda5d7acb1 Log: JENKINS-27395 Add per-stage test splitting option

SCM/JIRA link daemon added a comment - 2017-11-27 16:24

Code changed in jenkins
User: Andrew Bayer
Path:
pom.xml
src/main/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutor.java
src/main/java/org/jenkinsci/plugins/parallel_test_executor/SplitStep.java
src/test/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutorTest.java
src/test/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutorUnitTest.java
http://jenkins-ci.org/commit/parallel-test-executor-plugin/89aad65707343dc98339b006fa4fe07ba1fe58b0
Log:
Merge pull request #31 from abayer/jenkins-27395

~~JENKINS-27395~~ Add per-stage test splitting option

Compare: https://github.com/jenkinsci/parallel-test-executor-plugin/compare/fde66c24dda6...89aad6570734

SCM/JIRA link daemon added a comment - 2017-11-27 16:24 Code changed in jenkins User: Andrew Bayer Path: pom.xml src/main/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutor.java src/main/java/org/jenkinsci/plugins/parallel_test_executor/SplitStep.java src/test/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutorTest.java src/test/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutorUnitTest.java http://jenkins-ci.org/commit/parallel-test-executor-plugin/89aad65707343dc98339b006fa4fe07ba1fe58b0 Log: Merge pull request #31 from abayer/jenkins-27395 JENKINS-27395 Add per-stage test splitting option Compare: https://github.com/jenkinsci/parallel-test-executor-plugin/compare/fde66c24dda6...89aad6570734

James Dumay added a comment - 2017-11-27 23:00

The UI component of this work in Blue Ocean was just merged to master.

abayer what is the status of this ticket?

James Dumay added a comment - 2017-11-27 23:00 The UI component of this work in Blue Ocean was just merged to master. abayer what is the status of this ticket?

Jonathan Rogers added a comment - 2017-12-04 19:41

How does one disable this "feature"? I've been running different test suites in parallel branches and getting aggregated results. Now, it's harder to follow the history of the results for a particular test because the URL varies across runs.

Jonathan Rogers added a comment - 2017-12-04 19:41 How does one disable this "feature"? I've been running different test suites in parallel branches and getting aggregated results. Now, it's harder to follow the history of the results for a particular test because the URL varies across runs.

Andrew Bayer added a comment - 2017-12-04 19:44

jrogers I was thinking about that and have opened JENKINS-48372 for this.

Andrew Bayer added a comment - 2017-12-04 19:44 jrogers I was thinking about that and have opened JENKINS-48372 for this.

Andrew Bayer added a comment - 2017-12-04 19:45

jamesdumay I think it's basically done. Marking resolved.

Andrew Bayer added a comment - 2017-12-04 19:45 jamesdumay I think it's basically done. Marking resolved.

Jonathan Rogers added a comment - 2017-12-04 19:48

OK, I'll follow JENKINS-48372

Jonathan Rogers added a comment - 2017-12-04 19:48 OK, I'll follow JENKINS-48372

SCM/JIRA link daemon added a comment - 2018-02-18 21:13

Code changed in jenkins
User: Andrew Bayer
Path:
jenkins-plugin/pom.xml
jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/publishers/JunitTestsPublisher.java
jenkins-plugin/src/test/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepOnMasterTest.java
http://jenkins-ci.org/commit/pipeline-maven-plugin/aeb7d81775305282afcaa0464c4bf72c1faf3ca9
Log:
~~JENKINS-27395~~ Record flow node and enclosing stage/branch for tests

Downstream of https://github.com/jenkinsci/junit-plugin/pull/76. This
records the `FlowNode#id` for the step that's actually recording the
test results and the enclosing stage(s)/branch(es) of that `FlowNode`
with each suite, along with the names of the enclosing
stage(s)/branch(es) on each `CaseResult` for display purposes.

This is accomplished by getting the `FlowNode` via
`context.get(FlowNode.class)`, and then passing the configured
`JUnitResultArchiver`, the node's ID, the enclosing stage/branch IDs,
and the run (which is used to get the `Run#getExternalizableId()` for
internal purposes) to the new `JUnitResultArchiver#parseAndAttach`
static method. That method does the rest of the work.

Once this is all released, the classic test results pages will display
the test name with the enclosing stage/branch names prepended (if
there are test results from multiple stages/branches in the full test
result - i.e., only if relevant). Blue Ocean has a PR to do the same
that's pending.

SCM/JIRA link daemon added a comment - 2018-02-18 21:13 Code changed in jenkins User: Andrew Bayer Path: jenkins-plugin/pom.xml jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/publishers/JunitTestsPublisher.java jenkins-plugin/src/test/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepOnMasterTest.java http://jenkins-ci.org/commit/pipeline-maven-plugin/aeb7d81775305282afcaa0464c4bf72c1faf3ca9 Log: JENKINS-27395 Record flow node and enclosing stage/branch for tests Downstream of https://github.com/jenkinsci/junit-plugin/pull/76 . This records the `FlowNode#id` for the step that's actually recording the test results and the enclosing stage(s)/branch(es) of that `FlowNode` with each suite, along with the names of the enclosing stage(s)/branch(es) on each `CaseResult` for display purposes. This is accomplished by getting the `FlowNode` via `context.get(FlowNode.class)`, and then passing the configured `JUnitResultArchiver`, the node's ID, the enclosing stage/branch IDs, and the run (which is used to get the `Run#getExternalizableId()` for internal purposes) to the new `JUnitResultArchiver#parseAndAttach` static method. That method does the rest of the work. Once this is all released, the classic test results pages will display the test name with the enclosing stage/branch names prepended (if there are test results from multiple stages/branches in the full test result - i.e., only if relevant). Blue Ocean has a PR to do the same that's pending.

SCM/JIRA link daemon added a comment - 2018-02-18 21:13

Code changed in jenkins
User: Cyrille Le Clerc
Path:
jenkins-plugin/pom.xml
jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/publishers/JunitTestsPublisher.java
jenkins-plugin/src/test/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepOnMasterTest.java
http://jenkins-ci.org/commit/pipeline-maven-plugin/7224db1606420368db3f54b02a79f784c9202665
Log:
~~JENKINS-27395~~ Record flow node and enclosing stage/branch for tests #82

Merge pull request #82 from abayer/jenkins-27395

Compare: https://github.com/jenkinsci/pipeline-maven-plugin/compare/15990d246db2...7224db160642

SCM/JIRA link daemon added a comment - 2018-02-18 21:13 Code changed in jenkins User: Cyrille Le Clerc Path: jenkins-plugin/pom.xml jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/publishers/JunitTestsPublisher.java jenkins-plugin/src/test/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepOnMasterTest.java http://jenkins-ci.org/commit/pipeline-maven-plugin/7224db1606420368db3f54b02a79f784c9202665 Log: JENKINS-27395 Record flow node and enclosing stage/branch for tests #82 Merge pull request #82 from abayer/jenkins-27395 Compare: https://github.com/jenkinsci/pipeline-maven-plugin/compare/15990d246db2...7224db160642

SCM/JIRA link daemon added a comment - 2018-02-20 13:59

Code changed in jenkins
User: Cyrille Le Clerc
Path:
jenkins-plugin/pom.xml
jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/publishers/JunitTestsPublisher.java
jenkins-plugin/src/test/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepOnMasterTest.java
http://jenkins-ci.org/commit/pipeline-maven-plugin/31c1156f199515f907154938ed8e3bf5a14431c8
Log:
~~JENKINS-27395~~ temporarily disable bump to the jenkins-junit-pipeline 1.23 due to additional dependencies.

SCM/JIRA link daemon added a comment - 2018-02-20 13:59 Code changed in jenkins User: Cyrille Le Clerc Path: jenkins-plugin/pom.xml jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/publishers/JunitTestsPublisher.java jenkins-plugin/src/test/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepOnMasterTest.java http://jenkins-ci.org/commit/pipeline-maven-plugin/31c1156f199515f907154938ed8e3bf5a14431c8 Log: JENKINS-27395 temporarily disable bump to the jenkins-junit-pipeline 1.23 due to additional dependencies.

Sid S added a comment - 2018-05-23 19:12 - edited

How do you actually consume this in a Jenkinsfile? Our tests have the same name/namespaces across different variants and each variant gets it's own build stage (it's serial for unrelated reasons below). Right now tests across multiple runs/stages are indistinguishable as identical duplicates. Currently we're using the NUnitPublisher which converts nunit3 test XMLs to junit format. We could switch from NUnitPublisher to mstest but it's unclear if we need the test transforming plugins to also be updated OR if the junit changes are sufficient (since it can pull stage names directly).

Jenkins 2.124
Blue Ocean 1.5.0
JUnit 1.24
Nunit 0.23

This is the pipeline being used and it doesn't work i.e.

pipeline {
  agent {
    node {
      label 'master'
    }
  }
  stages {
    stage('Build variant1') {
      // build steps here
    }
    stage('Publish variant 1 tests') {
      steps {
        step([$class: 'NUnitPublisher', testResultsPattern: 'results/variant1/*.xml', debug: false, keepJUnitReports: true, skipJUnitArchiver:false, failIfNoResults: true])
      }
    }
    stage('Build variant2') {
      // build steps here
    }
    stage('Publish variant 2 tests') {
      steps {
        step([$class: 'NUnitPublisher', testResultsPattern: 'results/variant2/*.xml', debug: false, keepJUnitReports: true, skipJUnitArchiver:false, failIfNoResults: true])
      }
    }
  }
}

Sid S added a comment - 2018-05-23 19:12 - edited How do you actually consume this in a Jenkinsfile? Our tests have the same name/namespaces across different variants and each variant gets it's own build stage (it's serial for unrelated reasons below). Right now tests across multiple runs/stages are indistinguishable as identical duplicates. Currently we're using the NUnitPublisher which converts nunit3 test XMLs to junit format. We could switch from NUnitPublisher to mstest but it's unclear if we need the test transforming plugins to also be updated OR if the junit changes are sufficient (since it can pull stage names directly). Jenkins 2.124 Blue Ocean 1.5.0 JUnit 1.24 Nunit 0.23 This is the pipeline being used and it doesn't work i.e. pipeline { agent { node { label 'master' } } stages { stage( 'Build variant1' ) { // build steps here } stage( 'Publish variant 1 tests' ) { steps { step([$class: 'NUnitPublisher' , testResultsPattern: 'results/variant1/*.xml' , debug: false , keepJUnitReports: true , skipJUnitArchiver: false , failIfNoResults: true ]) } } stage( 'Build variant2' ) { // build steps here } stage( 'Publish variant 2 tests' ) { steps { step([$class: 'NUnitPublisher' , testResultsPattern: 'results/variant2/*.xml' , debug: false , keepJUnitReports: true , skipJUnitArchiver: false , failIfNoResults: true ]) } } } }

Sid S added a comment - 2018-05-24 02:03 - edited

Documenting for future users

1. Switched from nunit3.x to mstest's .trx (xml) test output. In dotnet this simply meant swapping the logger to the mstest one even though we're still using nunit as the actual test framework i.e.

dotnet test --logger "trx;LogFileName=resultfile.trx"

2. Directly used the xsl file published by the mstest folks (here) to transform .trx to junit compatible .xml files. In powershell is this a simple

$Xslt = New-Object System.Xml.Xsl.XslCompiledTransform
$Xslt.Load($xsltFilePath)    
$Xslt.Transform($xmlInputFilePath, $xmlOutputFilePath)

3. Use a Jenkinsfile like the one below. In the traditional Jenkins UI, the stage names get prefixed, so tests with same namespace are now seen differently. BlueOcean UI still shows them as indistinguishable / identical duplicates

pipeline {
  agent {
    node {
      label 'master'
    }
  }
  stages {    
    stage('variant-1') {
      steps {
        // build steps
      }
      post {
        always {
            junit 'results/*.xml'
        }
      }
    }
    stage('variant-2') {
      steps {
        // build steps
      }
      post {
        always {
            junit 'results/*.xml'
        }
      }
    }
  }
}

Sid S added a comment - 2018-05-24 02:03 - edited Documenting for future users 1. Switched from nunit3.x to mstest's .trx (xml) test output. In dotnet this simply meant swapping the logger to the mstest one even though we're still using nunit as the actual test framework i.e. dotnet test --logger "trx;LogFileName=resultfile.trx" 2. Directly used the xsl file published by the mstest folks ( here ) to transform .trx to junit compatible .xml files. In powershell is this a simple $Xslt = New- Object System .Xml.Xsl.XslCompiledTransform $Xslt.Load($xsltFilePath) $Xslt.Transform($xmlInputFilePath, $xmlOutputFilePath) 3. Use a Jenkinsfile like the one below. In the traditional Jenkins UI, the stage names get prefixed, so tests with same namespace are now seen differently. BlueOcean UI still shows them as indistinguishable / identical duplicates pipeline { agent { node { label 'master' } } stages { stage( 'variant-1' ) { steps { // build steps } post { always { junit 'results/*.xml' } } } stage( 'variant-2' ) { steps { // build steps } post { always { junit 'results/*.xml' } } } } }

trejkaz added a comment - 2018-05-24 02:05

Something doesn't quite add up here. Our build already has stage names, but we still don't get the tests distinguished between the different stages. The only workaround idea I had so far was to use XSLT to add the prefix into the actual XML files.

trejkaz added a comment - 2018-05-24 02:05 Something doesn't quite add up here. Our build already has stage names, but we still don't get the tests distinguished between the different stages. The only workaround idea I had so far was to use XSLT to add the prefix into the actual XML files.

Sid S added a comment - 2018-05-24 02:16

trejkaz, could you share your XSLT file or link to it?

Sid S added a comment - 2018-05-24 02:16 trejkaz , could you share your XSLT file or link to it?

trejkaz added a comment - 2018-05-24 02:18

We don't use an XSLT yet because we're using Gradle's reports, which are already in the right format.

trejkaz added a comment - 2018-05-24 02:18 We don't use an XSLT yet because we're using Gradle's reports, which are already in the right format.

Nicholas Klopfer-Webber added a comment - 2018-08-07 05:33

We had the same problem, but managed to work it out.

When we started using a Jenkinsfile we copied this example, however this bypasses the code that keeps each run of tests independent.

step([$class : 'JUnitResultArchiver', testResults: '**/build/test-results/**/*.xml'])

What we now use and works is:

junit testResults: '**/build/test-results/**/*.xml'

Nicholas Klopfer-Webber added a comment - 2018-08-07 05:33 We had the same problem, but managed to work it out. When we started using a Jenkinsfile we copied this example, however this bypasses the code that keeps each run of tests independent. step([$class : 'JUnitResultArchiver' , testResults: '**/build/test-results /**/ *.xml' ]) What we now use and works is: junit testResults: '**/build/test-results /**/ *.xml'

Mark Han added a comment - 2019-09-06 03:43

I think there's still a problem where running a simple `./gradlew test` merges all tests into one section, even though there can be multiple flavors of tests run. Not sure if this is the case if we explicitly break up the flavors e.g. `./gradlew testDebug` Do we need to break up `./gradlew` commands between separate stages?

Mark Han added a comment - 2019-09-06 03:43 I think there's still a problem where running a simple `./gradlew test` merges all tests into one section, even though there can be multiple flavors of tests run. Not sure if this is the case if we explicitly break up the flavors e.g. `./gradlew testDebug` Do we need to break up `./gradlew` commands between separate stages?

Jesse Glick added a comment - 2019-09-06 14:12

Do we need to break up ./gradlew commands between separate stages?

Currently I am afraid you do.

Jesse Glick added a comment - 2019-09-06 14:12 Do we need to break up ./gradlew commands between separate stages? Currently I am afraid you do.

Jenkins

Details

Description

Attachments

Issue Links

Activity

Collapse comment: Jesse Glick added a comment - 2016-04-08 19:48

Expand comment: Jesse Glick added a comment - 2016-04-08 19:48

Collapse comment: Sam Van Oort added a comment - 2016-04-27 18:32, Edited by Sam Van Oort - 2016-04-27 18:32

Expand comment: Sam Van Oort added a comment - 2016-04-27 18:32, Edited by Sam Van Oort - 2016-04-27 18:32

Collapse comment: Andrew Bayer added a comment - 2016-04-27 18:33

Expand comment: Andrew Bayer added a comment - 2016-04-27 18:33

Collapse comment: Jesse Glick added a comment - 2016-05-02 14:59

Expand comment: Jesse Glick added a comment - 2016-05-02 14:59

Collapse comment: Andrew Bayer added a comment - 2016-05-02 15:09

Expand comment: Andrew Bayer added a comment - 2016-05-02 15:09

Collapse comment: Sam Van Oort added a comment - 2016-05-02 15:11

Expand comment: Sam Van Oort added a comment - 2016-05-02 15:11

Collapse comment: Sam Van Oort added a comment - 2016-05-02 17:02

Expand comment: Sam Van Oort added a comment - 2016-05-02 17:02

Collapse comment: Andres Rodriguez added a comment - 2016-05-04 22:39, Edited by Andres Rodriguez - 2016-05-04 22:39

Expand comment: Andres Rodriguez added a comment - 2016-05-04 22:39, Edited by Andres Rodriguez - 2016-05-04 22:39

Collapse comment: Reinhold Füreder added a comment - 2017-07-04 12:00

Expand comment: Reinhold Füreder added a comment - 2017-07-04 12:00

Collapse comment: Jesse Glick added a comment - 2017-07-04 15:39

Expand comment: Jesse Glick added a comment - 2017-07-04 15:39

Collapse comment: Andrew Bayer added a comment - 2017-07-26 17:03

Expand comment: Andrew Bayer added a comment - 2017-07-26 17:03

Collapse comment: Andrew Bayer added a comment - 2017-07-26 21:53

Expand comment: Andrew Bayer added a comment - 2017-07-26 21:53

Collapse comment: Cyrille Le Clerc added a comment - 2017-07-26 22:17

Expand comment: Cyrille Le Clerc added a comment - 2017-07-26 22:17

Collapse comment: Andrew Bayer added a comment - 2017-07-26 22:29

Expand comment: Andrew Bayer added a comment - 2017-07-26 22:29

Collapse comment: Cyrille Le Clerc added a comment - 2017-07-26 23:01

Expand comment: Cyrille Le Clerc added a comment - 2017-07-26 23:01

Collapse comment: Andrew Bayer added a comment - 2017-07-26 23:13

Expand comment: Andrew Bayer added a comment - 2017-07-26 23:13

Collapse comment: Andrew Bayer added a comment - 2017-07-27 17:37

Expand comment: Andrew Bayer added a comment - 2017-07-27 17:37

Collapse comment: Cyrille Le Clerc added a comment - 2017-07-28 09:47

Expand comment: Cyrille Le Clerc added a comment - 2017-07-28 09:47

Collapse comment: James Dumay added a comment - 2017-08-14 01:15, Edited by James Dumay - 2017-08-14 01:39

Expand comment: James Dumay added a comment - 2017-08-14 01:15, Edited by James Dumay - 2017-08-14 01:39

Collapse comment: Andrew Bayer added a comment - 2017-08-22 14:47

Expand comment: Andrew Bayer added a comment - 2017-08-22 14:47

Collapse comment: Jesse Glick added a comment - 2017-08-24 16:42

Expand comment: Jesse Glick added a comment - 2017-08-24 16:42

Collapse comment: Andrew Bayer added a comment - 2017-08-24 18:02

Expand comment: Andrew Bayer added a comment - 2017-08-24 18:02

Collapse comment: Jesse Glick added a comment - 2017-08-24 19:49

Expand comment: Jesse Glick added a comment - 2017-08-24 19:49

Collapse comment: Andrew Bayer added a comment - 2017-08-24 19:56

Expand comment: Andrew Bayer added a comment - 2017-08-24 19:56

Collapse comment: SCM/JIRA link daemon added a comment - 2017-09-22 17:49

Expand comment: SCM/JIRA link daemon added a comment - 2017-09-22 17:49

Collapse comment: Sam Van Oort added a comment - 2017-09-22 21:14

Expand comment: Sam Van Oort added a comment - 2017-09-22 21:14

Collapse comment: Andrew Bayer added a comment - 2017-09-22 21:15

Expand comment: Andrew Bayer added a comment - 2017-09-22 21:15

Collapse comment: Sam Van Oort added a comment - 2017-09-26 22:28

Expand comment: Sam Van Oort added a comment - 2017-09-26 22:28

Collapse comment: SCM/JIRA link daemon added a comment - 2017-10-17 07:34

Expand comment: SCM/JIRA link daemon added a comment - 2017-10-17 07:34

Collapse comment: SCM/JIRA link daemon added a comment - 2017-10-17 07:34

Expand comment: SCM/JIRA link daemon added a comment - 2017-10-17 07:34

Collapse comment: Andrew Bayer added a comment - 2017-10-17 07:36

Expand comment: Andrew Bayer added a comment - 2017-10-17 07:36

Collapse comment: SCM/JIRA link daemon added a comment - 2017-11-02 21:18

Expand comment: SCM/JIRA link daemon added a comment - 2017-11-02 21:18

Collapse comment: SCM/JIRA link daemon added a comment - 2017-11-02 21:18

Expand comment: SCM/JIRA link daemon added a comment - 2017-11-02 21:18

Collapse comment: Andrew Bayer added a comment - 2017-11-14 17:47

Expand comment: Andrew Bayer added a comment - 2017-11-14 17:47

Collapse comment: Reinhold Füreder added a comment - 2017-11-15 10:34

Expand comment: Reinhold Füreder added a comment - 2017-11-15 10:34

Collapse comment: SCM/JIRA link daemon added a comment - 2017-11-27 16:24

Expand comment: SCM/JIRA link daemon added a comment - 2017-11-27 16:24

Collapse comment: SCM/JIRA link daemon added a comment - 2017-11-27 16:24