Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54255

xUnit sets build status incorrectly

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: xunit-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.89.3, xUnit 1.102
    • Similar Issues:

      Description

      We are counting on the xUnit "Publish xUnit test report" post-build step to properly set the build status. The Failed Tests section has 4 inputs that let you set thresholds to set the build to either UNSTABLE or FAILED based on whether the Total number of Failed tests, or total number of New failures, exceeds the specified thresholds.

      There is something broken in the test of the number of failures against the specified threshold. A recent build has the number of actual failures < the specified threshold for Total, and the number of New failures (1) < the specified threshold for New (5). It set the build to failed anyway.

      It appears that whenever the number of New failed tests is positive, that is reported as a build failure even when that number is less than the specified threshold.

      00:17:17 [xUnit] [INFO] - Starting to record.
      00:17:17 [xUnit] [INFO] - Processing CppUnit-1.12.1 (default)
      00:17:18 [xUnit] [INFO] - [CppUnit-1.12.1 (default)] - 1 test report file(s) were found with the pattern '**/testResult.xml' relative to '/home/bnet/jenkins/workspace/Linux-integration-test' for the testing framework 'CppUnit-1.12.1 (default)'.
      00:17:18 [xUnit] [INFO] - Check 'Failed Tests' threshold.
      00:17:18 [xUnit] [INFO] - Check 'Skipped Tests' threshold.
      00:17:18 [xUnit] [INFO] - Setting the build status to FAILURE
      00:17:18 [xUnit] [INFO] - Stopping recording.
      

        Attachments

          Activity

          Hide
          nfalco Nikolas Falco added a comment - - edited

          Update xunit plugin to latest version. If happens again please post the cppunit report and the xunit configuration.

          Show
          nfalco Nikolas Falco added a comment - - edited Update xunit plugin to latest version. If happens again please post the cppunit report and the xunit configuration.
          Hide
          nfalco Nikolas Falco added a comment -

          Any news? Any report attachment? Otherwise I have to close as can not reproduce

          Show
          nfalco Nikolas Falco added a comment - Any news? Any report attachment? Otherwise I have to close as can not reproduce
          Hide
          nfalco Nikolas Falco added a comment -

          With my local test it works. With Failed Threashold setup to total 5 and new to 2

          run 1 test count 5 all pass -> BUILD OK
          run 2 test count 8, 3 fails (3 new fails) -> BUILD FAILED (new threshold)
          run 2 test count 9, 4 fails (1 new fails) -> BUILD OK
          run 2 test count 11, 6 fails (2 new fails) -> BUILD FAILED (total threshold)

          Show
          nfalco Nikolas Falco added a comment - With my local test it works. With Failed Threashold setup to total 5 and new to 2 run 1 test count 5 all pass -> BUILD OK run 2 test count 8, 3 fails (3 new fails) -> BUILD FAILED (new threshold) run 2 test count 9, 4 fails (1 new fails) -> BUILD OK run 2 test count 11, 6 fails (2 new fails) -> BUILD FAILED (total threshold)
          Hide
          vladichko Vlad Aginsky added a comment - - edited

          Hi i am facing very similar issue:

          this is how i define xunit trasholds:

           

          // xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')],
                          tools: [Custom(customXSL: "${JENKINS_URL}/userContent/pytest-to-junit.xsl",
                                  deleteOutputFiles: true,
                                  failIfNotNew: true,
                                  pattern: "${env.DOWNLOADED_ARTIFACTS_PATH}/${env.TEST_RESULTS_PATH}/*.xml",
                                  skipNoTestFiles: true,
                                  stopProcessingIfError: true)]
          

           

           

          I have some build new failed tests:

          Test Result (40 failures / +17)

            i expect to get failed status, but xunit still sets the build to be unstable:

          01:57:59.736  INFO: Starting to record.
          01:57:59.736  INFO: Processing Custom Tool
          01:58:00.146  INFO: Stopping recording.
          01:58:00.147  INFO: Check 'Failed Tests' threshold.
          01:58:00.147  INFO: The new number of tests for the threshold 'Failed Tests' exceeds the specified "new failure threshold" value.
          

           I use:

          Show
          vladichko Vlad Aginsky added a comment - - edited Hi i am facing very similar issue: this is how i define xunit trasholds:   // xunit thresholds: [failed(unstableThreshold: '0' ,failureNewThreshold: '0' )], tools: [Custom(customXSL: "${JENKINS_URL}/userContent/pytest-to-junit.xsl" , deleteOutputFiles: true , failIfNotNew: true , pattern: "${env.DOWNLOADED_ARTIFACTS_PATH}/${env.TEST_RESULTS_PATH}/*.xml" , skipNoTestFiles: true , stopProcessingIfError: true )]     I have some build new failed tests: Test Result (40 failures / +17)   i expect to get failed status, but xunit still sets the build to be unstable: 01:57:59.736 INFO: Starting to record. 01:57:59.736 INFO: Processing Custom Tool 01:58:00.146 INFO: Stopping recording. 01:58:00.147 INFO: Check 'Failed Tests' threshold. 01:58:00.147 INFO: The new number of tests for the threshold 'Failed Tests' exceeds the specified " new failure threshold" value.  I use:   Jenkins ver. 2.160  xUnit plugin 2.3.2
          Hide
          nfalco Nikolas Falco added a comment -

          Vlad Aginsky your issue is not similar to this one. This one is related to new tests counting.

          Anyway if this is your configuration (it's not clear from your post, seems commented)

          xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')],
          

          the unstable build status is correct and is in accordance with what the tooltips document.

          Maybe you wanted to set up the threshold in this manner

          xunit thresholds: [failed(failureNewThreshold: '0', failureThreshold: '0')], tools: [Cus...
          

          with failureThreshold instead of unstableThreshold

          Keep in mind also what failureNewThreshold means.

          Show
          nfalco Nikolas Falco added a comment - Vlad Aginsky your issue is not similar to this one. This one is related to new tests counting. Anyway if this is your configuration (it's not clear from your post, seems commented) xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')], the unstable build status is correct and is in accordance with what the tooltips document. Maybe you wanted to set up the threshold in this manner xunit thresholds: [failed(failureNewThreshold: '0', failureThreshold: '0')], tools: [Cus... with failureThreshold instead of unstableThreshold Keep in mind also what failureNewThreshold means.
          Hide
          vladichko Vlad Aginsky added a comment - - edited

          Nicolas, thanks for prompt response, since i am not sure how to proceed with this issue, i will make another try here before opening another one, please.

          I want to define following behavior:

          {{Any new failed test to fail the build, if no new failures, any old failure to cause to unstable build status. }}

          This is how I use Snippet Generator  

          this is what it produces:

           

          xunit thresholds: [failed(failureNewThreshold: '0', unstableThreshold: '0')], tools: []
          

           

          and it is very similar to my original post. I just thought that may be precedence makes difference here (does it?) so i have then in reverse order:

          xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')], ...
          

          Important note: i see that in some cases it behaves just right, but sometimes it produces yellow status on new failed tests, just as described above.

          And, IMHO(really humble) what you wrote

          thresholds: [failed(failureNewThreshold: '0', failureThreshold: '0')]
          

           will cause the build to fail on any failed test, new and old. please correct me.

           Bottom line, do you still say that i dont use thresholds in a right manner?

          Show
          vladichko Vlad Aginsky added a comment - - edited Nicolas, thanks for prompt response, since i am not sure how to proceed with this issue, i will make another try here before opening another one, please. I want to define following behavior: {{ Any new failed test to fail the build, if no new failures, any old failure to cause to unstable build status. }} This is how I use Snippet Generator   this is what it produces:   xunit thresholds: [failed(failureNewThreshold: '0' , unstableThreshold: '0' )], tools: []   and it is very similar to my original post. I just thought that may be precedence makes difference here (does it?) so i have then in reverse order: xunit thresholds: [failed(unstableThreshold: '0' ,failureNewThreshold: '0' )], ... Important note: i see that in some cases it behaves just right, but sometimes it produces yellow status on new failed tests, just as described above. And, IMHO(really humble) what you wrote thresholds: [failed(failureNewThreshold: '0' , failureThreshold: '0' )]  will cause the build to fail on any failed test, new and old. please correct me.  Bottom line, do you still say that i dont use thresholds in a right manner?
          Hide
          nfalco Nikolas Falco added a comment -

          The order of parameters are not meaningful. Are used to contruct the object by reflection.

          > Any new failed test to fail the build, if no new failures, any old failure to cause to unstable build status.
          Failed thresholds are analysed before unstable thresholds. Total count of test failure are taken into account before of New test failures. This logic is applied to all thresholds (skipped, failed and passed). This means your scenario is not possible.

          Show
          nfalco Nikolas Falco added a comment - The order of parameters are not meaningful. Are used to contruct the object by reflection. > Any new failed test to fail the build, if no new failures, any old failure to cause to unstable build status. Failed thresholds are analysed before unstable thresholds. Total count of test failure are taken into account before of New test failures. This logic is applied to all thresholds (skipped, failed and passed). This means your scenario is not possible.
          Hide
          xholubov Andrii Holubov added a comment - - edited

          Hello, 

          I am sorry if I'm not reporting my problem in the "right way" and reopening it, but I am facing the similar issue.

          When I am running tests threshold values are ignored and test always sets its status to failed.

          I've tried setting Passing test threshold and Failed test threshold as well.

          I am using PHPUnit  9.1.2 and paratest for running my tests. Also I am using the newest version of xunit.

          results.xml

           

           

           
           

          Show
          xholubov Andrii Holubov added a comment - - edited Hello,  I am sorry if I'm not reporting my problem in the "right way" and reopening it, but I am facing the similar issue. When I am running tests threshold values are ignored and test always sets its status to failed. I've tried setting Passing test threshold and Failed test threshold as well. I am using PHPUnit  9.1.2 and paratest for running my tests. Also I am using the newest version of xunit. results.xml        
          Hide
          nfalco Nikolas Falco added a comment -

          images are not coherent console show 'Passed Tests' threashold but the job configuration shows Failed Tests

          Show
          nfalco Nikolas Falco added a comment - images are not coherent console show 'Passed Tests' threashold but the job configuration shows Failed Tests
          Hide
          xholubov Andrii Holubov added a comment -

          I am sorry, here is the correct image.

          Show
          xholubov Andrii Holubov added a comment - I am sorry, here is the correct image.
          Hide
          nfalco Nikolas Falco added a comment -

          report is not coherent with console log that show 4 test, 6 assertions and 2 failure.

          Could you simply attach the report where you have the issue and the configuration (write it do not post the image)

          Show
          nfalco Nikolas Falco added a comment - report is not coherent with console log that show 4 test, 6 assertions and 2 failure. Could you simply attach the report where you have the issue and the configuration (write it do not post the image)
          Hide
          xholubov Andrii Holubov added a comment -

          Here is the report results.xml

          Report type: PHPUnit-4.x

          I am checking failed tests:

          Total 15, New <empty field> - for unstable

          Total 30, New <empty field> - for failed

          The last test results were: 
          Tests: 4, Assertions: 6, Failures: 2, Errors: 0.

          And build was still marked as "Failed"

          Show
          xholubov Andrii Holubov added a comment - Here is the report  results.xml Report type: PHPUnit-4.x I am checking failed tests: Total 15, New <empty field> - for unstable Total 30, New <empty field> - for failed The last test results were:  Tests: 4, Assertions: 6, Failures: 2, Errors: 0. And build was still marked as "Failed"
          Hide
          nfalco Nikolas Falco added a comment -

          This is my output with results.xml (testsuite name="/home/tester/selenium-php-test/bootstrap/..//tests/Dedoles" tests="4" assertions="6" failures="2" skipped="0" errors="0" time="287.791277") :

          Avviato dall'utente unknown or anonymous
          Running as SYSTEM
          Compilazione in corso nello spazio di lavoro D:\git\xunit-plugin\work\workspace\test
          INFO: Starting to record.
          INFO: Processing PHPUnit-4.x (default)
          INFO: [PHPUnit-4.x (default)] - 1 test report file(s) were found with the pattern 'results.xml' relative to 'D:\git\xunit-plugin\work\workspace\test' for the testing framework 'PHPUnit-4.x (default)'.
          INFO: Check 'Failed Tests' threshold.
          INFO: Setting the build status to SUCCESS
          INFO: Stopping recording.
          Finished: SUCCESS
          Show
          nfalco Nikolas Falco added a comment - This is my output with results.xml (testsuite name="/home/tester/selenium-php-test/bootstrap/..//tests/Dedoles" tests="4" assertions="6" failures="2" skipped="0" errors="0" time="287.791277") : Avviato dall'utente unknown or anonymous Running as SYSTEM Compilazione in corso nello spazio di lavoro D:\git\xunit-plugin\work\workspace\test INFO: Starting to record. INFO: Processing PHPUnit-4.x (default) INFO: [PHPUnit-4.x (default)] - 1 test report file(s) were found with the pattern 'results.xml' relative to 'D:\git\xunit-plugin\work\workspace\test' for the testing framework 'PHPUnit-4.x (default)'. INFO: Check 'Failed Tests' threshold. INFO: Setting the build status to SUCCESS INFO: Stopping recording. Finished: SUCCESS
          Hide
          nfalco Nikolas Falco added a comment -

          If I change unstable Total to 1 I got

          INFO: Starting to record.
          INFO: Processing PHPUnit-4.x (default)
          INFO: [PHPUnit-4.x (default)] - 1 test report file(s) were found with the pattern 'results.xml' relative to 'D:\git\xunit-plugin\work\workspace\test' for the testing framework 'PHPUnit-4.x (default)'.
          INFO: Check 'Failed Tests' threshold.
          INFO: The total number of tests for the threshold 'Failed Tests' exceeds the specified "unstable threshold" value.
          INFO: Setting the build status to UNSTABLE
          INFO: Stopping recording.
          Build step 'Publish xUnit test result report' changed build result to UNSTABLE
          Finished: UNSTABLE

          because 2 failures exceed 1 I set and this with failed Total to 1

          INFO: Starting to record.
          INFO: Processing PHPUnit-4.x (default)
          INFO: [PHPUnit-4.x (default)] - 1 test report file(s) were found with the pattern 'results.xml' relative to 'D:\git\xunit-plugin\work\workspace\test' for the testing framework 'PHPUnit-4.x (default)'.
          INFO: Check 'Failed Tests' threshold.
          INFO: The total number of tests for the threshold 'Failed Tests' exceeds the specified "failure threshold" value.
          INFO: Setting the build status to FAILURE
          INFO: Stopping recording.
          Build step 'Publish xUnit test result report' changed build result to FAILURE
          Finished: FAILURE
          

          because 2 failures exceed 1 I set

          Show
          nfalco Nikolas Falco added a comment - If I change unstable Total to 1 I got INFO: Starting to record. INFO: Processing PHPUnit-4.x (default) INFO: [PHPUnit-4.x (default)] - 1 test report file(s) were found with the pattern 'results.xml' relative to 'D:\git\xunit-plugin\work\workspace\test' for the testing framework 'PHPUnit-4.x (default)'. INFO: Check 'Failed Tests' threshold. INFO: The total number of tests for the threshold 'Failed Tests' exceeds the specified "unstable threshold" value. INFO: Setting the build status to UNSTABLE INFO: Stopping recording. Build step 'Publish xUnit test result report' changed build result to UNSTABLE Finished: UNSTABLE because 2 failures exceed 1 I set and this with failed Total to 1 INFO: Starting to record. INFO: Processing PHPUnit-4.x (default) INFO: [PHPUnit-4.x (default)] - 1 test report file(s) were found with the pattern 'results.xml' relative to 'D:\git\xunit-plugin\work\workspace\test' for the testing framework 'PHPUnit-4.x (default)'. INFO: Check 'Failed Tests' threshold. INFO: The total number of tests for the threshold 'Failed Tests' exceeds the specified "failure threshold" value. INFO: Setting the build status to FAILURE INFO: Stopping recording. Build step 'Publish xUnit test result report' changed build result to FAILURE Finished: FAILURE because 2 failures exceed 1 I set
          Hide
          xholubov Andrii Holubov added a comment - - edited

          Thank you for answer. 

          Any ideas what can cause such a result on my machine or what can I do to resolve this problem ? 

          Show
          xholubov Andrii Holubov added a comment - - edited Thank you for answer.  Any ideas what can cause such a result on my machine or what can I do to resolve this problem ? 
          Hide
          nfalco Nikolas Falco added a comment -

          Which version of plugin do you have?

          Show
          nfalco Nikolas Falco added a comment - Which version of plugin do you have?
          Hide
          xholubov Andrii Holubov added a comment -

          I am using xunit - 2.3.9

          Show
          xholubov Andrii Holubov added a comment - I am using xunit - 2.3.9
          Hide
          nfalco Nikolas Falco added a comment -

          Test 1: wipeout the job workspace.

          Test 2: could you download a jenkins.war and start it with "java -jar jenkins.war" install xunit plugin, create a new freestyle project with only xunit configured as above and copy paste you result in the job folder?

          Show
          nfalco Nikolas Falco added a comment - Test 1: wipeout the job workspace. Test 2: could you download a jenkins.war and start it with "java -jar jenkins.war" install xunit plugin, create a new freestyle project with only xunit configured as above and copy paste you result in the job folder?

            People

            Assignee:
            nfalco Nikolas Falco
            Reporter:
            zmi_johnschmitz John Schmitz
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: