Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-30413

Failed test age is incorrect if same test name exist in different package

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • junit-plugin
    • None
    • Jenkins-1.609.3 & 2.198
      Junit-plugin 1.9 & 1.28
      gtest to run test & produce xml results
    • junit-1.30

      If a test case name is present in several packages, and a test is failed in one of the packages, the age of the failure always will be '1'.

      Moreover the ext email plugin will send a mail at every run if the regression trigger is configured.

      See this simple example of test results:

      <testsuites name="AllTests"><testsuite name="foo"><testcase name="ThisTestNameIsDuplicated" classname="package2.test"><failure message="FAILED"/></testcase></testsuite></testsuites>

      <testsuites name="AllTests2"><testsuite name="foo"><testcase name="ThisTestNameIsDuplicated" classname="package1.test"/></testsuite></testsuites>

      These two test cases :

      • package1.test.ThisTestNameIsDuplicated
      • package2.test.ThisTestNameIsDuplicated
        are displayed properly in Jenkins,
        but the age of failure of package2.test.ThisTestNameIsDuplicated will always be 1
        after several run of the "tests".

      It seems that the classname is not taken into account when computing the age.

          [JENKINS-30413] Failed test age is incorrect if same test name exist in different package

          Krip added a comment -

          Hi,

          I have this issue with Jenkins 2.60.1 and Junit-plugin 1.20 (latest version as of this day)

          This issue is quite impacting for my workflow as the email alerts I send in case of regression are not accurate. To prevent this I made a groovy script that I execute in email-ext as a pre-send script.

          It is not perfect, especially I did not find how to know which email-ext trigger is configured. This script should be executed only when email-ext is configured for regression events.

          It is not necessary to use an array "regressing_tests = []" but anyway it does the job for me and it might help others so here it is:

          import hudson.tasks.junit.CaseResult;
          import hudson.tasks.test.AbstractTestResultAction
          import hudson.tasks.test.TestResult
          
          regressing_tests = []
          
          AbstractTestResultAction currentTestResultAction = build.getAction(AbstractTestResultAction.class)
          if (currentTestResultAction != null) {
          
            //Workaround to jenkins bug https://issues.jenkins-ci.org/browse/JENKINS-30413  "Failed test age is incorrect if same test name exist in different package"
            AbstractTestResultAction previousTestResultAction = currentTestResultAction.getPreviousResult()
            List<CaseResult> previous_failedTests = previousTestResultAction.getFailedTests();
          
            List<CaseResult> failedTests = currentTestResultAction.getFailedTests();
            failedTests.each() { failedT ->
              boolean bRegression = (failedT.getAge() == 1);
              if ( bRegression )
              {
                //Workaround to jenkins bug https://issues.jenkins-ci.org/browse/JENKINS-30413  "Failed test age is incorrect if same test name exist in different package"
                //
                boolean bPreviouslyFailing = false
                previous_failedTests.each() { prev_failedT ->
                  if ( failedT.getFullDisplayName() == prev_failedT.getFullDisplayName() )
                  {
                    //out.println("The test " + failedT.getFullDisplayName() + "has a failing age of 1 but was also failing previously so skippping it." )
                    bPreviouslyFailing=true
                    return true
                  }
                }
          
                if ( ! bPreviouslyFailing )
                {
                  regressing_tests.add(failedT)
                }
              }
            }
          }
          //If no regressions were found, tells email-ext jenkins extension to not send the email:
          if ( regressing_tests.size() == 0 ) {
            cancel=true
          }
          

           

          Krip added a comment - Hi, I have this issue with Jenkins 2.60.1 and Junit-plugin 1.20 (latest version as of this day) This issue is quite impacting for my workflow as the email alerts I send in case of regression are not accurate. To prevent this I made a groovy script that I execute in email-ext as a pre-send script. It is not perfect, especially I did not find how to know which email-ext trigger is configured. This script should be executed only when email-ext is configured for regression events. It is not necessary to use an array "regressing_tests = []" but anyway it does the job for me and it might help others so here it is: import hudson.tasks.junit.CaseResult; import hudson.tasks.test.AbstractTestResultAction import hudson.tasks.test.TestResult regressing_tests = [] AbstractTestResultAction currentTestResultAction = build.getAction(AbstractTestResultAction.class) if (currentTestResultAction != null ) {   //Workaround to jenkins bug https://issues.jenkins-ci.org/browse/JENKINS-30413  "Failed test age is incorrect if same test name exist in different package "   AbstractTestResultAction previousTestResultAction = currentTestResultAction.getPreviousResult()   List<CaseResult> previous_failedTests = previousTestResultAction.getFailedTests();   List<CaseResult> failedTests = currentTestResultAction.getFailedTests();   failedTests.each() { failedT ->     boolean bRegression = (failedT.getAge() == 1);     if ( bRegression )     {       //Workaround to jenkins bug https://issues.jenkins-ci.org/browse/JENKINS-30413  "Failed test age is incorrect if same test name exist in different package "       //       boolean bPreviouslyFailing = false       previous_failedTests.each() { prev_failedT ->         if ( failedT.getFullDisplayName() == prev_failedT.getFullDisplayName() )         {           //out.println( "The test " + failedT.getFullDisplayName() + "has a failing age of 1 but was also failing previously so skippping it." )           bPreviouslyFailing= true           return true         }       }       if ( ! bPreviouslyFailing )       {         regressing_tests.add(failedT)       }     }   } } //If no regressions were found, tells email-ext jenkins extension to not send the email: if ( regressing_tests.size() == 0 ) {   cancel= true }  

          Krip added a comment -

          Looking at this plugin code I found something interesting in file:
          junit-plugin-master/src/main/java/hudson/tasks/junit/SuiteResult.java

          In function:
          {{public CaseResult getCase(String name) { }}

          Its description says:
          Note that test name needs not be unique.

          I think our issue is here.
          The Junit plugin takes for assumption that test names are unique !

          So in fact the fix may be just to replace "<CaseResult>.getName" calls by "<CaseResult>.getFullDisplayName" !
          Easy !
          Can someone try to implement this please?

          Krip added a comment - Looking at this plugin code I found something interesting in file: junit-plugin-master/src/main/java/hudson/tasks/junit/SuiteResult.java In function: {{public CaseResult getCase(String name) { }} Its description says: Note that test name needs not be unique. I think our issue is here. The Junit plugin takes for assumption that test names are unique ! So in fact the fix may be just to replace "<CaseResult>.getName" calls by "<CaseResult>.getFullDisplayName" ! Easy ! Can someone try to implement this please?

          Krip added a comment - - edited

          Add a simpler xml to reproduce the issue:

          <testsuites>
              <testsuite name="MyTestSuitename" >
                  <testcase classname="test.classname1" name="DuplicateTestName">
                      <failure message="FAILED"/>
                  </testcase>
                  <testcase classname="test.classname2" name="DuplicateTestName"/>
              </testsuite>
          </testsuites>
          

          With this junit result, the test "test.classname1.DuplicateTestName" will always appear with an age of 1.

          Krip added a comment - - edited Add a simpler xml to reproduce the issue: <testsuites> <testsuite name= "MyTestSuitename" > <testcase classname= "test.classname1" name= "DuplicateTestName" > <failure message= "FAILED" /> </testcase> <testcase classname= "test.classname2" name= "DuplicateTestName" /> </testsuite> </testsuites> With this junit result, the test "test.classname1.DuplicateTestName" will always appear with an age of 1.

          Krip added a comment -

          Krip added a comment - I proposed a fix here: https://github.com/jenkinsci/junit-plugin/pull/97

          Stephen McCants added a comment - - edited

          I'm seeing this problem on Jenkins 2.198 with JUnit plugin 1.28.  Since there is a test case demonstrating the problem and a proposed fix (along with 6 votes) could this ticket get a little attention to fix the problem?  Thanks.

           

          Stephen McCants added a comment - - edited I'm seeing this problem on Jenkins 2.198 with JUnit plugin 1.28.  Since there is a test case demonstrating the problem and a proposed fix (along with 6 votes) could this ticket get a little attention to fix the problem?  Thanks.  

          Lars added a comment -

          Still no attention for this issue nor the working fix...

          Lars added a comment - Still no attention for this issue nor the working fix...

            krissap Krip
            blatinville Bertrand Latinville
            Votes:
            8 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated:
              Resolved: