Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-14089

JUnit report - Quarantine intermittent tests

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      I see other CI systems with a Quarantine feature for their JUnit testing.
      How about Jenkins keeps pace?

      Quarantine is a reporting feature. The idea of the feature is that intermittent or misbehaving tests can be quarantined. This means that they wouldn't be reported in the totals of passing/failing tests and would not affect the pass/fail status of the Jenkins job. The tests can later be removed from quarantine when they are consistently passing or otherwise behaving well.

      The quarantine feature wouldn't change whether a test executes, but rather how it is reported.

        Attachments

          Issue Links

            Activity

            Hide
            stephanebruckert Stéphane Bruckert added a comment -

            This issue/feature should deserve some more attention. This is a great plugin which needs some support.

            Show
            stephanebruckert Stéphane Bruckert added a comment - This issue/feature should deserve some more attention. This is a great plugin which needs some support.
            Hide
            davidlumpkin David Lumpkin added a comment -

            Any update on this? It looks like the plugin has fallen out of date but it'd be a really helpful feature to have

            Show
            davidlumpkin David Lumpkin added a comment - Any update on this? It looks like the plugin has fallen out of date but it'd be a really helpful feature to have
            Hide
            samsta samuel nobs added a comment -

            No update. We're using the plugin as is and it works fine for what we need it. Just build it from source: https://github.com/samsta/quarantine

            Show
            samsta samuel nobs added a comment - No update. We're using the plugin as is and it works fine for what we need it. Just build it from source: https://github.com/samsta/quarantine
            Hide
            mireksz Mirek Sz added a comment -

            samuel nobs - do you know of any way to make the Quarantine Plugin work in pipelines? Currently it freezes. Easy way to reproduce:

            node { 
              def mvnHome 
              stage('Preparation') { // for display purposes 
                 // Get some code from a GitHub repository 
                 git 'https://github.com/jglick/simple-maven-project-with-tests.git' 
                 // Get the Maven tool. 
                 // ** NOTE: This 'M3' Maven tool must be configured 
                 // **       in the global configuration.            
                 mvnHome = tool 'M3' 
              } 
              stage('Build') { 
                 // Run the maven build 
                 if (isUnix()) { 
                    sh "'${mvnHome}/bin/mvn' -Dmaven.test.failure.ignore clean package" 
                 } else { 
                    bat(/"${mvnHome}\bin\mvn" -Dmaven.test.failure.ignore clean package/) 
                 } 
              } 
              stage('Results') { 
                 step([$class: 'QuarantinableJUnitResultArchiver', testDataPublishers: [[$class: 'QuarantineTestDataPublisher']], testResults: '**/target/surefire-reports/TEST-*.xml']
            ) 
                // junit '**/target/surefire-reports/TEST-*.xml' 
            
              } 
            }
            
            
            
            Show
            mireksz Mirek Sz added a comment - samuel nobs - do you know of any way to make the Quarantine Plugin work in pipelines? Currently it freezes. Easy way to reproduce: node {   def mvnHome   stage( 'Preparation' ) { // for display purposes       // Get some code from a GitHub repository      git 'https: //github.com/jglick/simple-maven-project-with-tests.git'       // Get the Maven tool.       // ** NOTE: This 'M3' Maven tool must be configured       // **       in the global configuration.                  mvnHome = tool 'M3'   }   stage( 'Build' ) {       // Run the maven build       if (isUnix()) {         sh " '${mvnHome}/bin/mvn' -Dmaven.test.failure.ignore clean package "      } else {         bat(/ "${mvnHome}\bin\mvn" -Dmaven.test.failure.ignore clean package /)      }   }   stage( 'Results' ) {      step([$class: 'QuarantinableJUnitResultArchiver' , testDataPublishers: [[$class: 'QuarantineTestDataPublisher' ]], testResults: '**/target/surefire-reports/TEST-*.xml' ] )      // junit '**/target/surefire-reports/TEST-*.xml'   } }
            Hide
            samsta samuel nobs added a comment -

            Mirek Sz, sorry, no idea. I have never used pipelines before

            Show
            samsta samuel nobs added a comment - Mirek Sz , sorry, no idea. I have never used pipelines before

              People

              Assignee:
              samsta samuel nobs
              Reporter:
              johnstosh John Muczynski
              Votes:
              14 Vote for this issue
              Watchers:
              13 Start watching this issue

                Dates

                Created:
                Updated: