Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65355

xunit fails job when files are not found, even when skip is checked and threshold is 100

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Component/s: xunit-plugin
    • Labels:
      None
    • Environment:
      Jenkins LTS 2.263.4
      xUnit plugin 3.0.1
    • Similar Issues:

      Description

      We have jobs that produce custom tests result XMLs, but not on every run. We use the custom tool in xunit, along with a custom stylesheet to read those test files, and set skip-if-no-files to true.

      Since the upgrade to plugin version 3.0.1, our jobs are failing when test result files are not found, even though "skip test if files not found" is checked.

      We also noticed this issue with the JUnit tool in the xUnit plugin.

       

      Steps to reproduce:

       

      1> create a job that has the xunit plugin set to use junit or custom tool, and add path to some test files, doesn't matter where

      2> set both failure thresholds to 100

      3> ensure the "Skip if there are no test files" is checked

      4> ensure the build does not produce any xunit files

       

      Expected:

      job passes, because threshold is 100 but skip-if-no-files is checked

       

      Actual:

      job fails, with the following in the log

       

      INFO: [Custom Tool] - No test report file(s) were found with the pattern 'dev/bin/vnv/mod/tests/**/utf*.xml' relative to 'T:\' for the testing framework 'Custom Tool'.
      Did you enter a pattern relative to (and within) the workspace directory?
      Did you generate the result report(s) for 'Custom Tool'?"
      WARNING: All test reports are empty.
      INFO: Check 'Failed Tests' threshold.
      INFO: The total number of tests for the threshold 'Failed Tests' exceeds the specified "failure threshold" value.
      INFO: Setting the build status to FAILURE
      Build step 'Publish xUnit test result report' changed build result to FAILURE 
      
      
      

       

      FYI: We downgraded to 2.4.0, and this started working again.

        Attachments

          Activity

          Hide
          cesarjorgemartinez Cesar Jorge Martínez added a comment -

          Hi,

          Same fail here using Jenkins 2.277.3 and xunit 3.0.2 report junit/xunit

          The xml result are generated correcty but show:

          INFO: Processing JUnit
          WARNING: All test reports are empty.
          INFO: Check 'Failed Tests' threshold.
          INFO: The total number of tests for the threshold 'Failed Tests' exceeds the specified "failure threshold" value.
          INFO: Setting the build status to FAILURE
          Build step 'Publish xUnit test result report' changed build result to FAILURE
          Started calculate disk usage of build
          Finished Calculation of disk usage of build in 0 seconds
          Started calculate disk usage of workspace
          Finished Calculation of disk usage of workspace in 0 seconds
          Finished: FAILURE

           

          Rollback to 2.4.0 works correctly.

          What is happening to fail with xunit 3.0.2?

           

          Regards,

          Cesar Jorge

           

          Show
          cesarjorgemartinez Cesar Jorge Martínez added a comment - Hi, Same fail here using Jenkins 2.277.3 and xunit 3.0.2 report junit/xunit The xml result are generated correcty but show: INFO: Processing JUnit WARNING: All test reports are empty. INFO: Check 'Failed Tests' threshold. INFO: The total number of tests for the threshold 'Failed Tests' exceeds the specified "failure threshold" value. INFO: Setting the build status to FAILURE Build step 'Publish xUnit test result report' changed build result to FAILURE Started calculate disk usage of build Finished Calculation of disk usage of build in 0 seconds Started calculate disk usage of workspace Finished Calculation of disk usage of workspace in 0 seconds Finished: FAILURE   Rollback to 2.4.0 works correctly. What is happening to fail with xunit 3.0.2?   Regards, Cesar Jorge  
          Hide
          cesarjorgemartinez Cesar Jorge Martínez added a comment -

          You can delete the xunit 3.x from the available plugins until this critical bug is fixed?

          Regards,

          Cesar Jorge

           

          Show
          cesarjorgemartinez Cesar Jorge Martínez added a comment - You can delete the xunit 3.x from the available plugins until this critical bug is fixed? Regards, Cesar Jorge  
          Hide
          nfalco Nikolas Falco added a comment - - edited

          no, I never do that.

          The bug is critical only if you produce reports sometimes and do not update is an acceptable workaround since there is not automatic update.

          Other workaround is put an empty test case result

          Show
          nfalco Nikolas Falco added a comment - - edited no, I never do that. The bug is critical only if you produce reports sometimes and do not update is an acceptable workaround since there is not automatic update. Other workaround is put an empty test case result
          Hide
          cesarjorgemartinez Cesar Jorge Martínez added a comment -

          Uhm, the workaround is that always put one file empty?

          But all my tests always write the results all found:

          "Ficheros XML con los informes de tests" => tests/results/features.TS04_flow.002.xml

          workspace]$ ls -l tests/results/features.TS04_flow*
          rw-r---- 1 jenkins jenkins 75839 abr 21 16:09 tests/results/features.TS04_flow.002.xml

           

          Show
          cesarjorgemartinez Cesar Jorge Martínez added a comment - Uhm, the workaround is that always put one file empty? But all my tests always write the results all found: "Ficheros XML con los informes de tests" => tests/results/features.TS04_flow.002.xml workspace]$ ls -l tests/results/features.TS04_flow* rw-r ---- 1 jenkins jenkins 75839 abr 21 16:09 tests/results/features.TS04_flow.002.xml  
          Hide
          nfalco Nikolas Falco added a comment -

          Sorry my bad.

          The "Skip if there are no test files" means do not throw error is there are not test files at all. If there are test files but empty (suite should have at least 1 test) than threashold is considered.

          Show
          nfalco Nikolas Falco added a comment - Sorry my bad. The "Skip if there are no test files" means do not throw error is there are not test files at all. If there are test files but empty (suite should have at least 1 test) than threashold is considered.
          Hide
          nfalco Nikolas Falco added a comment - - edited

          I followed your steps using JUnit and it works. This is the output of XUnit (3.0.2):

          Running as SYSTEM
          Compilazione in corso nello spazio di lavoro D:\git\xunit-plugin\work\workspace\test
          INFO: Processing JUnit
          INFO: [JUnit] - No test report file(s) were found with the pattern '**/*' relative to 'D:\git\xunit-plugin\work\workspace\test' for the testing framework 'JUnit'.
          Did you enter a pattern relative to (and within) the workspace directory?
          Did you generate the result report(s) for 'JUnit'?"
          WARNING: All test reports are empty.
          INFO: Check 'Failed Tests' threshold.
          INFO: Setting the build status to SUCCESS
          Finished: SUCCESS
          Show
          nfalco Nikolas Falco added a comment - - edited I followed your steps using JUnit and it works. This is the output of XUnit (3.0.2): Running as SYSTEM Compilazione in corso nello spazio di lavoro D:\git\xunit-plugin\work\workspace\test INFO: Processing JUnit INFO: [JUnit] - No test report file(s) were found with the pattern '**/*' relative to 'D:\git\xunit-plugin\work\workspace\test' for the testing framework 'JUnit'. Did you enter a pattern relative to (and within) the workspace directory? Did you generate the result report(s) for 'JUnit'?" WARNING: All test reports are empty. INFO: Check 'Failed Tests' threshold. INFO: Setting the build status to SUCCESS Finished: SUCCESS
          Hide
          skhushalani Sagar Khushalani added a comment - - edited

          [ Nikolas Falco

          We had to downgrade and use 2.4.0. I am in the process of upgrading Jenkins, and thought I'd give 3.0.2 a chance. Having the same issue. Please note: there are NO files found (they are not just empty files). Why does it complain about no files being found and then files are empty? Can you check it with custom-tool as well, please? We will have to stay on 2.4.0 but I'm not sure how long that is feasable.

          INFO: Processing Custom Tool
          INFO: [Custom Tool] - No test report file(s) were found with the pattern 'dev/bin/vnv/mod/tests/**/utf*.xml' relative to 'T:\' for the testing framework 'Custom Tool'.
          Did you enter a pattern relative to (and within) the workspace directory?
          Did you generate the result report(s) for 'Custom Tool'?"
          WARNING: All test reports are empty.
          INFO: Check 'Failed Tests' threshold.
          INFO: The total number of tests for the threshold 'Failed Tests' exceeds the specified "failure threshold" value.
          INFO: Setting the build status to FAILURE
          Show
          skhushalani Sagar Khushalani added a comment - - edited [ Nikolas Falco We had to downgrade and use 2.4.0. I am in the process of upgrading Jenkins, and thought I'd give 3.0.2 a chance. Having the same issue. Please note: there are NO files found (they are not just empty files). Why does it complain about no files being found and then files are empty? Can you check it with custom-tool as well, please? We will have to stay on 2.4.0 but I'm not sure how long that is feasable. INFO: Processing Custom Tool INFO: [Custom Tool] - No test report file(s) were found with the pattern 'dev/bin/vnv/mod/tests/**/utf*.xml' relative to 'T:\' for the testing framework 'Custom Tool'. Did you enter a pattern relative to (and within) the workspace directory? Did you generate the result report(s) for 'Custom Tool'?" WARNING: All test reports are empty. INFO: Check 'Failed Tests' threshold. INFO: The total number of tests for the threshold 'Failed Tests' exceeds the specified "failure threshold" value. INFO: Setting the build status to FAILURE
          Hide
          nfalco Nikolas Falco added a comment -

          Sagar Khushalani I need the job configuration

          Show
          nfalco Nikolas Falco added a comment - Sagar Khushalani I need the job configuration
          Hide
          skhushalani Sagar Khushalani added a comment -

          Show
          skhushalani Sagar Khushalani added a comment -
          Hide
          nfalco Nikolas Falco added a comment -

          You can remove skipped threashold it is empty.

          I have copy you exact configuration and locally it does not fails. the failed threashold return SUCCESS

          Show
          nfalco Nikolas Falco added a comment - You can remove skipped threashold it is empty. I have copy you exact configuration and locally it does not fails. the failed threashold return SUCCESS

            People

            Assignee:
            nfalco Nikolas Falco
            Reporter:
            skhushalani Sagar Khushalani
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: