Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54255

xUnit sets build status incorrectly

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: xunit-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.89.3, xUnit 1.102
    • Similar Issues:

      Description

      We are counting on the xUnit "Publish xUnit test report" post-build step to properly set the build status. The Failed Tests section has 4 inputs that let you set thresholds to set the build to either UNSTABLE or FAILED based on whether the Total number of Failed tests, or total number of New failures, exceeds the specified thresholds.

      There is something broken in the test of the number of failures against the specified threshold. A recent build has the number of actual failures < the specified threshold for Total, and the number of New failures (1) < the specified threshold for New (5). It set the build to failed anyway.

      It appears that whenever the number of New failed tests is positive, that is reported as a build failure even when that number is less than the specified threshold.

      00:17:17 [xUnit] [INFO] - Starting to record.
      00:17:17 [xUnit] [INFO] - Processing CppUnit-1.12.1 (default)
      00:17:18 [xUnit] [INFO] - [CppUnit-1.12.1 (default)] - 1 test report file(s) were found with the pattern '**/testResult.xml' relative to '/home/bnet/jenkins/workspace/Linux-integration-test' for the testing framework 'CppUnit-1.12.1 (default)'.
      00:17:18 [xUnit] [INFO] - Check 'Failed Tests' threshold.
      00:17:18 [xUnit] [INFO] - Check 'Skipped Tests' threshold.
      00:17:18 [xUnit] [INFO] - Setting the build status to FAILURE
      00:17:18 [xUnit] [INFO] - Stopping recording.
      

        Attachments

          Activity

          Hide
          nfalco Nikolas Falco added a comment -

          If I change unstable Total to 1 I got

          INFO: Starting to record.
          INFO: Processing PHPUnit-4.x (default)
          INFO: [PHPUnit-4.x (default)] - 1 test report file(s) were found with the pattern 'results.xml' relative to 'D:\git\xunit-plugin\work\workspace\test' for the testing framework 'PHPUnit-4.x (default)'.
          INFO: Check 'Failed Tests' threshold.
          INFO: The total number of tests for the threshold 'Failed Tests' exceeds the specified "unstable threshold" value.
          INFO: Setting the build status to UNSTABLE
          INFO: Stopping recording.
          Build step 'Publish xUnit test result report' changed build result to UNSTABLE
          Finished: UNSTABLE

          because 2 failures exceed 1 I set and this with failed Total to 1

          INFO: Starting to record.
          INFO: Processing PHPUnit-4.x (default)
          INFO: [PHPUnit-4.x (default)] - 1 test report file(s) were found with the pattern 'results.xml' relative to 'D:\git\xunit-plugin\work\workspace\test' for the testing framework 'PHPUnit-4.x (default)'.
          INFO: Check 'Failed Tests' threshold.
          INFO: The total number of tests for the threshold 'Failed Tests' exceeds the specified "failure threshold" value.
          INFO: Setting the build status to FAILURE
          INFO: Stopping recording.
          Build step 'Publish xUnit test result report' changed build result to FAILURE
          Finished: FAILURE
          

          because 2 failures exceed 1 I set

          Show
          nfalco Nikolas Falco added a comment - If I change unstable Total to 1 I got INFO: Starting to record. INFO: Processing PHPUnit-4.x (default) INFO: [PHPUnit-4.x (default)] - 1 test report file(s) were found with the pattern 'results.xml' relative to 'D:\git\xunit-plugin\work\workspace\test' for the testing framework 'PHPUnit-4.x (default)'. INFO: Check 'Failed Tests' threshold. INFO: The total number of tests for the threshold 'Failed Tests' exceeds the specified "unstable threshold" value. INFO: Setting the build status to UNSTABLE INFO: Stopping recording. Build step 'Publish xUnit test result report' changed build result to UNSTABLE Finished: UNSTABLE because 2 failures exceed 1 I set and this with failed Total to 1 INFO: Starting to record. INFO: Processing PHPUnit-4.x (default) INFO: [PHPUnit-4.x (default)] - 1 test report file(s) were found with the pattern 'results.xml' relative to 'D:\git\xunit-plugin\work\workspace\test' for the testing framework 'PHPUnit-4.x (default)'. INFO: Check 'Failed Tests' threshold. INFO: The total number of tests for the threshold 'Failed Tests' exceeds the specified "failure threshold" value. INFO: Setting the build status to FAILURE INFO: Stopping recording. Build step 'Publish xUnit test result report' changed build result to FAILURE Finished: FAILURE because 2 failures exceed 1 I set
          Hide
          xholubov Andrii Holubov added a comment - - edited

          Thank you for answer. 

          Any ideas what can cause such a result on my machine or what can I do to resolve this problem ? 

          Show
          xholubov Andrii Holubov added a comment - - edited Thank you for answer.  Any ideas what can cause such a result on my machine or what can I do to resolve this problem ? 
          Hide
          nfalco Nikolas Falco added a comment -

          Which version of plugin do you have?

          Show
          nfalco Nikolas Falco added a comment - Which version of plugin do you have?
          Hide
          xholubov Andrii Holubov added a comment -

          I am using xunit - 2.3.9

          Show
          xholubov Andrii Holubov added a comment - I am using xunit - 2.3.9
          Hide
          nfalco Nikolas Falco added a comment -

          Test 1: wipeout the job workspace.

          Test 2: could you download a jenkins.war and start it with "java -jar jenkins.war" install xunit plugin, create a new freestyle project with only xunit configured as above and copy paste you result in the job folder?

          Show
          nfalco Nikolas Falco added a comment - Test 1: wipeout the job workspace. Test 2: could you download a jenkins.war and start it with "java -jar jenkins.war" install xunit plugin, create a new freestyle project with only xunit configured as above and copy paste you result in the job folder?

            People

            Assignee:
            nfalco Nikolas Falco
            Reporter:
            zmi_johnschmitz John Schmitz
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: