Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54255

xUnit sets build status incorrectly

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: xunit-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.89.3, xUnit 1.102
    • Similar Issues:

      Description

      We are counting on the xUnit "Publish xUnit test report" post-build step to properly set the build status. The Failed Tests section has 4 inputs that let you set thresholds to set the build to either UNSTABLE or FAILED based on whether the Total number of Failed tests, or total number of New failures, exceeds the specified thresholds.

      There is something broken in the test of the number of failures against the specified threshold. A recent build has the number of actual failures < the specified threshold for Total, and the number of New failures (1) < the specified threshold for New (5). It set the build to failed anyway.

      It appears that whenever the number of New failed tests is positive, that is reported as a build failure even when that number is less than the specified threshold.

      00:17:17 [xUnit] [INFO] - Starting to record.
      00:17:17 [xUnit] [INFO] - Processing CppUnit-1.12.1 (default)
      00:17:18 [xUnit] [INFO] - [CppUnit-1.12.1 (default)] - 1 test report file(s) were found with the pattern '**/testResult.xml' relative to '/home/bnet/jenkins/workspace/Linux-integration-test' for the testing framework 'CppUnit-1.12.1 (default)'.
      00:17:18 [xUnit] [INFO] - Check 'Failed Tests' threshold.
      00:17:18 [xUnit] [INFO] - Check 'Skipped Tests' threshold.
      00:17:18 [xUnit] [INFO] - Setting the build status to FAILURE
      00:17:18 [xUnit] [INFO] - Stopping recording.
      

        Attachments

          Activity

          zmi_johnschmitz John Schmitz created issue -
          nfalco Nikolas Falco made changes -
          Field Original Value New Value
          Description We are counting on the xUnit "Publish xUnit test report" post-build step to properly set the build status. The Failed Tests section has 4 inputs that let you set thresholds to set the build to either UNSTABLE or FAILED based on whether the Total number of Failed tests, or total number of New failures, exceeds the specified thresholds.

          There is something broken in the test of the number of failures against the specified threshold. A recent build has the number of actual failures < the specified threshold for Total, and the number of New failures (1) < the specified threshold for New (5). It set the build to failed anyway.

          It appears that whenever the number of New failed tests is positive, that is reported as a build failure even when that number is less than the specified threshold.
          *00:17:17* [xUnit] [INFO] - Starting to record.*00:17:17* [xUnit] [INFO] - Processing CppUnit-1.12.1 (default)*00:17:18* [xUnit] [INFO] - [CppUnit-1.12.1 (default)] - 1 test report file(s) were found with the pattern '**/testResult.xml' relative to '/home/bnet/jenkins/workspace/Linux-integration-test' for the testing framework 'CppUnit-1.12.1 (default)'.*00:17:18* [xUnit] [INFO] - Check 'Failed Tests' threshold.*00:17:18* [xUnit] [INFO] - Check 'Skipped Tests' threshold.*00:17:18* [xUnit] [INFO] - Setting the build status to FAILURE*00:17:18* [xUnit] [INFO] - Stopping recording.
          We are counting on the xUnit "Publish xUnit test report" post-build step to properly set the build status. The Failed Tests section has 4 inputs that let you set thresholds to set the build to either UNSTABLE or FAILED based on whether the Total number of Failed tests, or total number of New failures, exceeds the specified thresholds.

          There is something broken in the test of the number of failures against the specified threshold. A recent build has the number of actual failures < the specified threshold for Total, and the number of New failures (1) < the specified threshold for New (5). It set the build to failed anyway.

          It appears that whenever the number of New failed tests is positive, that is reported as a build failure even when that number is less than the specified threshold.
          {noformat}
          00:17:17 [xUnit] [INFO] - Starting to record.
          00:17:17 [xUnit] [INFO] - Processing CppUnit-1.12.1 (default)
          00:17:18 [xUnit] [INFO] - [CppUnit-1.12.1 (default)] - 1 test report file(s) were found with the pattern '**/testResult.xml' relative to '/home/bnet/jenkins/workspace/Linux-integration-test' for the testing framework 'CppUnit-1.12.1 (default)'.
          00:17:18 [xUnit] [INFO] - Check 'Failed Tests' threshold.
          00:17:18 [xUnit] [INFO] - Check 'Skipped Tests' threshold.
          00:17:18 [xUnit] [INFO] - Setting the build status to FAILURE
          00:17:18 [xUnit] [INFO] - Stopping recording.
          {noformat}
          nfalco Nikolas Falco made changes -
          Resolution Cannot Reproduce [ 5 ]
          Status Open [ 1 ] Closed [ 6 ]
          vladichko Vlad Aginsky made changes -
          Resolution Cannot Reproduce [ 5 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          nfalco Nikolas Falco made changes -
          Resolution Cannot Reproduce [ 5 ]
          Status Reopened [ 4 ] Closed [ 6 ]
          vladichko Vlad Aginsky made changes -
          Attachment image-2019-03-04-09-13-18-589.png [ 46250 ]
          xholubov Andrii Holubov made changes -
          Resolution Cannot Reproduce [ 5 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          xholubov Andrii Holubov made changes -
          Attachment Screenshot from 2020-10-15 17-43-30.png [ 52957 ]
          Attachment Screenshot from 2020-10-15 17-40-23.png [ 52958 ]
          xholubov Andrii Holubov made changes -
          Attachment results.xml [ 52959 ]
          xholubov Andrii Holubov made changes -
          Attachment Screenshot from 2020-10-15 17-47-35.png [ 52960 ]
          xholubov Andrii Holubov made changes -
          Attachment Screenshot from 2020-10-15 17-47-35.png [ 52960 ]
          xholubov Andrii Holubov made changes -
          Attachment Screenshot from 2020-10-15 17-43-30.png [ 52957 ]
          xholubov Andrii Holubov made changes -
          xholubov Andrii Holubov made changes -
          xholubov Andrii Holubov made changes -
          Attachment results.xml [ 52963 ]
          nfalco Nikolas Falco made changes -
          Resolution Fixed [ 1 ]
          Status Reopened [ 4 ] Fixed but Unreleased [ 10203 ]
          nfalco Nikolas Falco made changes -
          Status Fixed but Unreleased [ 10203 ] Closed [ 6 ]

            People

            Assignee:
            nfalco Nikolas Falco
            Reporter:
            zmi_johnschmitz John Schmitz
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: