-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
None
-
Jenkins LTS 2.263.4
xUnit plugin 3.0.1
We have jobs that produce custom tests result XMLs, but not on every run. We use the custom tool in xunit, along with a custom stylesheet to read those test files, and set skip-if-no-files to true.
Since the upgrade to plugin version 3.0.1, our jobs are failing when test result files are not found, even though "skip test if files not found" is checked.
We also noticed this issue with the JUnit tool in the xUnit plugin.
Steps to reproduce:
1> create a job that has the xunit plugin set to use junit or custom tool, and add path to some test files, doesn't matter where
2> set both failure thresholds to 100
3> ensure the "Skip if there are no test files" is checked
4> ensure the build does not produce any xunit files
Expected:
job passes, because threshold is 100 but skip-if-no-files is checked
Actual:
job fails, with the following in the log
INFO: [Custom Tool] - No test report file(s) were found with the pattern 'dev/bin/vnv/mod/tests/**/utf*.xml' relative to 'T:\' for the testing framework 'Custom Tool'. Did you enter a pattern relative to (and within) the workspace directory? Did you generate the result report(s) for 'Custom Tool'?" WARNING: All test reports are empty. INFO: Check 'Failed Tests' threshold. INFO: The total number of tests for the threshold 'Failed Tests' exceeds the specified "failure threshold" value. INFO: Setting the build status to FAILURE Build step 'Publish xUnit test result report' changed build result to FAILURE
FYI: We downgraded to 2.4.0, and this started working again.
[JENKINS-65355] xunit fails job when files are not found, even when skip is checked and threshold is 100
Sagar Khushalani
created issue -
Sagar Khushalani
made changes -
| Environment | Original: Jenkins LTS 2.263.4 |
New:
Jenkins LTS 2.263.4
xUnit plugin 3.0.1 |
Sagar Khushalani
made changes -
| Description |
Original:
We have jobs that produce custom tests result XMLs, but not on every run. We use the custom tool in xunit, along with a custom stylesheet to read those test files, and set skip-if-no-files to true.
Since the upgrade to plugin version 3.0.1, our jobs are failing when test result files are not found, even though "skip test if files not found" is checked. We also noticed this issue with the JUnit tool in the xUnit plugin. Steps to reproduce: 1> create a job that has the xunit plugin set to use junit or custom tool, and add path to some test files, doesn't matter where 2> set both failure thresholds to 100 3> ensure the "Skip if there are no test files" is checked 4> ensure the build does not produce any xunit files Expected: job passes, because threshold is 100 but skip-if-no-files is checked Actual: job fails, with the following in the log {noformat} INFO: [Custom Tool] - No test report file(s) were found with the pattern 'dev/bin/vnv/mod/tests/**/utf*.xml' relative to 'T:\' for the testing framework 'Custom Tool'. Did you enter a pattern relative to (and within) the workspace directory? Did you generate the result report(s) for 'Custom Tool'?" WARNING: All test reports are empty. INFO: Check 'Failed Tests' threshold. INFO: The total number of tests for the threshold 'Failed Tests' exceeds the specified "failure threshold" value. INFO: Setting the build status to FAILURE Build step 'Publish xUnit test result report' changed build result to FAILURE {noformat} |
New:
We have jobs that produce custom tests result XMLs, but not on every run. We use the custom tool in xunit, along with a custom stylesheet to read those test files, and set skip-if-no-files to true.
Since the upgrade to plugin version 3.0.1, our jobs are failing when test result files are not found, even though "skip test if files not found" is checked. We also noticed this issue with the JUnit tool in the xUnit plugin. Steps to reproduce: 1> create a job that has the xunit plugin set to use junit or custom tool, and add path to some test files, doesn't matter where 2> set both failure thresholds to 100 3> ensure the "Skip if there are no test files" is checked 4> ensure the build does not produce any xunit files Expected: job passes, because threshold is 100 but skip-if-no-files is checked Actual: job fails, with the following in the log {noformat} INFO: [Custom Tool] - No test report file(s) were found with the pattern 'dev/bin/vnv/mod/tests/**/utf*.xml' relative to 'T:\' for the testing framework 'Custom Tool'. Did you enter a pattern relative to (and within) the workspace directory? Did you generate the result report(s) for 'Custom Tool'?" WARNING: All test reports are empty. INFO: Check 'Failed Tests' threshold. INFO: The total number of tests for the threshold 'Failed Tests' exceeds the specified "failure threshold" value. INFO: Setting the build status to FAILURE Build step 'Publish xUnit test result report' changed build result to FAILURE {noformat} FYI: We downgraded to 2.4.0, and this started working again. |
Sagar Khushalani
made changes -
| Priority | Original: Blocker [ 1 ] | New: Critical [ 2 ] |
| Priority | Original: Critical [ 2 ] | New: Major [ 3 ] |
| Resolution | New: Cannot Reproduce [ 5 ] | |
| Status | Original: Open [ 1 ] | New: Closed [ 6 ] |
Sagar Khushalani
made changes -
| Attachment | New: screenshot-1.png [ 55413 ] |
Sagar Khushalani
made changes -
| Comment |
[ [~nfalco]
We had to downgrade and use 2.4.0. I am in the process of upgrading Jenkins, and thought I'd give 3.0.2 a chance. Having the same issue. Please note: there are NO files found (they are not just empty files). Why does it complain about no files being found and then files are empty? Can you double check this, please? We will have to stay on 2.4.0 but I'm not sure how long that is feasable. {noformat} INFO: Processing Custom Tool INFO: [Custom Tool] - No test report file(s) were found with the pattern 'dev/bin/vnv/mod/tests/**/utf*.xml' relative to 'T:\' for the testing framework 'Custom Tool'. Did you enter a pattern relative to (and within) the workspace directory? Did you generate the result report(s) for 'Custom Tool'?" WARNING: All test reports are empty. INFO: Check 'Failed Tests' threshold. INFO: The total number of tests for the threshold 'Failed Tests' exceeds the specified "failure threshold" value. INFO: Setting the build status to FAILURE{noformat} !screenshot-1.png|thumbnail! ] |
Sagar Khushalani
made changes -
| Attachment | Original: screenshot-1.png [ 55413 ] |
Sagar Khushalani
made changes -
| Attachment | New: screenshot-1.png [ 55414 ] |