Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-27395

distinguish test reports filed from different parts of workflow

    • Pipeline - July/August, Pipeline - December

      JUnit test result archiving can happen multiple times inside workflow, and currently there's no way to meaningfully distinguish multiple invocations.

          [JENKINS-27395] distinguish test reports filed from different parts of workflow

          Code changed in jenkins
          User: Cyrille Le Clerc
          Path:
          jenkins-plugin/pom.xml
          jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/publishers/JunitTestsPublisher.java
          jenkins-plugin/src/test/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepOnMasterTest.java
          http://jenkins-ci.org/commit/pipeline-maven-plugin/7224db1606420368db3f54b02a79f784c9202665
          Log:
          JENKINS-27395 Record flow node and enclosing stage/branch for tests #82

          Merge pull request #82 from abayer/jenkins-27395

          Compare: https://github.com/jenkinsci/pipeline-maven-plugin/compare/15990d246db2...7224db160642

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Cyrille Le Clerc Path: jenkins-plugin/pom.xml jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/publishers/JunitTestsPublisher.java jenkins-plugin/src/test/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepOnMasterTest.java http://jenkins-ci.org/commit/pipeline-maven-plugin/7224db1606420368db3f54b02a79f784c9202665 Log: JENKINS-27395 Record flow node and enclosing stage/branch for tests #82 Merge pull request #82 from abayer/jenkins-27395 Compare: https://github.com/jenkinsci/pipeline-maven-plugin/compare/15990d246db2...7224db160642

          Code changed in jenkins
          User: Cyrille Le Clerc
          Path:
          jenkins-plugin/pom.xml
          jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/publishers/JunitTestsPublisher.java
          jenkins-plugin/src/test/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepOnMasterTest.java
          http://jenkins-ci.org/commit/pipeline-maven-plugin/31c1156f199515f907154938ed8e3bf5a14431c8
          Log:
          JENKINS-27395 temporarily disable bump to the jenkins-junit-pipeline 1.23 due to additional dependencies.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Cyrille Le Clerc Path: jenkins-plugin/pom.xml jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/publishers/JunitTestsPublisher.java jenkins-plugin/src/test/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepOnMasterTest.java http://jenkins-ci.org/commit/pipeline-maven-plugin/31c1156f199515f907154938ed8e3bf5a14431c8 Log: JENKINS-27395 temporarily disable bump to the jenkins-junit-pipeline 1.23 due to additional dependencies.

          Sid S added a comment - - edited

          How do you actually consume this in a Jenkinsfile? Our tests have the same name/namespaces across different variants and each variant gets it's own build stage (it's serial for unrelated reasons below). Right now tests across multiple runs/stages are indistinguishable as identical duplicates. Currently we're using the NUnitPublisher which converts nunit3 test XMLs to junit format. We could switch from NUnitPublisher to mstest but it's unclear if we need the test transforming plugins to also be updated OR if the junit changes are sufficient (since it can pull stage names directly).

          Jenkins 2.124
          Blue Ocean 1.5.0
          JUnit 1.24
          Nunit 0.23

          This is the pipeline being used and it doesn't work i.e.

          pipeline {
            agent {
              node {
                label 'master'
              }
            }
            stages {
              stage('Build variant1') {
                // build steps here
              }
              stage('Publish variant 1 tests') {
                steps {
                  step([$class: 'NUnitPublisher', testResultsPattern: 'results/variant1/*.xml', debug: false, keepJUnitReports: true, skipJUnitArchiver:false, failIfNoResults: true])
                }
              }
              stage('Build variant2') {
                // build steps here
              }
              stage('Publish variant 2 tests') {
                steps {
                  step([$class: 'NUnitPublisher', testResultsPattern: 'results/variant2/*.xml', debug: false, keepJUnitReports: true, skipJUnitArchiver:false, failIfNoResults: true])
                }
              }
            }
          }
          

          Sid S added a comment - - edited How do you actually consume this in a Jenkinsfile? Our tests have the same name/namespaces across different variants and each variant gets it's own build stage (it's serial for unrelated reasons below). Right now tests across multiple runs/stages are indistinguishable as identical duplicates. Currently we're using the NUnitPublisher which converts nunit3 test XMLs to junit format. We could switch from NUnitPublisher to mstest but it's unclear if we need the test transforming plugins to also be updated OR if the junit changes are sufficient (since it can pull stage names directly). Jenkins 2.124 Blue Ocean 1.5.0 JUnit 1.24 Nunit 0.23 This is the pipeline being used and it doesn't work i.e. pipeline { agent { node { label 'master' } } stages { stage( 'Build variant1' ) { // build steps here } stage( 'Publish variant 1 tests' ) { steps { step([$class: 'NUnitPublisher' , testResultsPattern: 'results/variant1/*.xml' , debug: false , keepJUnitReports: true , skipJUnitArchiver: false , failIfNoResults: true ]) } } stage( 'Build variant2' ) { // build steps here } stage( 'Publish variant 2 tests' ) { steps { step([$class: 'NUnitPublisher' , testResultsPattern: 'results/variant2/*.xml' , debug: false , keepJUnitReports: true , skipJUnitArchiver: false , failIfNoResults: true ]) } } } }

          Sid S added a comment - - edited

          Documenting for future users

          1. Switched from nunit3.x to mstest's .trx (xml) test output. In dotnet this simply meant swapping the logger to the mstest one even though we're still using nunit as the actual test framework i.e.

          dotnet test --logger "trx;LogFileName=resultfile.trx"

          2. Directly used the xsl file published by the mstest folks (here) to transform .trx to junit compatible .xml files. In powershell is this a simple

          $Xslt = New-Object System.Xml.Xsl.XslCompiledTransform
          $Xslt.Load($xsltFilePath)    
          $Xslt.Transform($xmlInputFilePath, $xmlOutputFilePath)
          

          3. Use a Jenkinsfile like the one below. In the traditional Jenkins UI, the stage names get prefixed, so tests with same namespace are now seen differently. BlueOcean UI still shows them as indistinguishable / identical duplicates

          pipeline {
            agent {
              node {
                label 'master'
              }
            }
            stages {    
              stage('variant-1') {
                steps {
                  // build steps
                }
                post {
                  always {
                      junit 'results/*.xml'
                  }
                }
              }
              stage('variant-2') {
                steps {
                  // build steps
                }
                post {
                  always {
                      junit 'results/*.xml'
                  }
                }
              }
            }
          }
          

          Sid S added a comment - - edited Documenting for future users 1. Switched from nunit3.x to mstest's .trx (xml) test output. In dotnet this simply meant swapping the logger to the mstest one even though we're still using nunit as the actual test framework i.e. dotnet test --logger "trx;LogFileName=resultfile.trx" 2. Directly used the xsl file published by the mstest folks ( here ) to transform .trx to junit compatible .xml files. In powershell is this a simple $Xslt = New- Object System .Xml.Xsl.XslCompiledTransform $Xslt.Load($xsltFilePath) $Xslt.Transform($xmlInputFilePath, $xmlOutputFilePath) 3. Use a Jenkinsfile like the one below. In the traditional Jenkins UI, the stage names get prefixed, so tests with same namespace are now seen differently. BlueOcean UI still shows them as indistinguishable / identical duplicates pipeline { agent { node { label 'master' } } stages { stage( 'variant-1' ) { steps { // build steps } post { always { junit 'results/*.xml' } } } stage( 'variant-2' ) { steps { // build steps } post { always { junit 'results/*.xml' } } } } }

          trejkaz added a comment -

          Something doesn't quite add up here. Our build already has stage names, but we still don't get the tests distinguished between the different stages. The only workaround idea I had so far was to use XSLT to add the prefix into the actual XML files.

           

          trejkaz added a comment - Something doesn't quite add up here. Our build already has stage names, but we still don't get the tests distinguished between the different stages. The only workaround idea I had so far was to use XSLT to add the prefix into the actual XML files.  

          Sid S added a comment -

          trejkaz, could you share your XSLT file or link to it?

          Sid S added a comment - trejkaz , could you share your XSLT file or link to it?

          trejkaz added a comment -

          We don't use an XSLT yet because we're using Gradle's reports, which are already in the right format.

           

          trejkaz added a comment - We don't use an XSLT yet because we're using Gradle's reports, which are already in the right format.  

          We had the same problem, but managed to work it out.

          When we started using a Jenkinsfile we copied this example, however this bypasses the code that keeps each run of tests independent.

          step([$class : 'JUnitResultArchiver', testResults: '**/build/test-results/**/*.xml'])
          

          What we now use and works is:

          junit testResults: '**/build/test-results/**/*.xml'
          

          Nicholas Klopfer-Webber added a comment - We had the same problem, but managed to work it out. When we started using a Jenkinsfile we copied this example, however this bypasses the code that keeps each run of tests independent. step([$class : 'JUnitResultArchiver' , testResults: '**/build/test-results /**/ *.xml' ]) What we now use and works is: junit testResults: '**/build/test-results /**/ *.xml'

          Mark Han added a comment -

          I think there's still a problem where running a simple `./gradlew test` merges all tests into one section, even though there can be multiple flavors of tests run. Not sure if this is the case if we explicitly break up the flavors e.g. `./gradlew testDebug` Do we need to break up `./gradlew` commands between separate stages? 

          Mark Han added a comment - I think there's still a problem where running a simple `./gradlew test` merges all tests into one section, even though there can be multiple flavors of tests run. Not sure if this is the case if we explicitly break up the flavors e.g. `./gradlew testDebug` Do we need to break up `./gradlew` commands between separate stages? 

          Jesse Glick added a comment -

          Do we need to break up ./gradlew commands between separate stages?

          Currently I am afraid you do.

          Jesse Glick added a comment - Do we need to break up ./gradlew commands between separate stages? Currently I am afraid you do.

            abayer Andrew Bayer
            kohsuke Kohsuke Kawaguchi
            Votes:
            35 Vote for this issue
            Watchers:
            66 Start watching this issue

              Created:
              Updated:
              Resolved: