-
Bug
-
Resolution: Fixed
-
Major
-
Jenkins 2.289.3
xunit-plugin 3.0.2
In the case where no test files are found and a failure threshold is set, the build status is always set to failure, regardless of the skipNoTestFiles setting.
Xunit pipeline configuration
step([$class : 'XUnitPublisher', testTimeMargin: '3000', thresholdMode: 2, thresholds: [ [ $class : 'FailedThreshold', failureNewThreshold : '100', failureThreshold : '100', unstableNewThreshold: '100', unstableThreshold : '100' ], [ $class : 'SkippedThreshold', failureNewThreshold : '100', failureThreshold : '100', unstableNewThreshold: '100', unstableThreshold : '100' ] ], tools : [ [ $class : 'GoogleTestType', deleteOutputFiles : true, failIfNotNew : false, pattern : '*.xml', skipNoTestFiles : true, stopProcessingIfError: true ] ], sleepTime: 20 ])
Expected behavior
If no test files are found, the build status should not be changed. This behavior was present in version 2.4.0.
Actual behavior
The build status is always set to failure when no test files are found, even if all build stages are passing.
[JENKINS-66650] Build status set to failure when no test files exist if failure thresholds are set
Kent Barnett
added a comment - This is happening with a pipeline job with no other steps included. Here is the full pipeline script:
pipeline {
agent any
stages {
stage('Hello') {
steps {
echo 'Hello World'
}
}
}
post {
always {
step([$class : 'XUnitPublisher', testTimeMargin: '3000', thresholdMode: 2,
thresholds: [
[
$class : 'FailedThreshold',
failureNewThreshold : '100',
failureThreshold : '100',
unstableNewThreshold: '100',
unstableThreshold : '100'
],
[
$class : 'SkippedThreshold',
failureNewThreshold : '100',
failureThreshold : '100',
unstableNewThreshold: '100',
unstableThreshold : '100'
]
],
tools : [
[
$class : 'GoogleTestType',
deleteOutputFiles : true,
failIfNotNew : false,
pattern : '*.xml',
skipNoTestFiles : true,
stopProcessingIfError: true
]
],
sleepTime: 20
])
}
}
}
The full job output is:
Started by user unknown or anonymous Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Running on Jenkins in /var/jenkins_home/workspace/test-xunit [Pipeline] { [Pipeline] stage [Pipeline] { (Hello) [Pipeline] echo Hello World [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Declarative: Post Actions) [Pipeline] step INFO: Processing GoogleTest-1.8 INFO: [GoogleTest-1.8] - No test report file(s) were found with the pattern '*.xml' relative to '/var/jenkins_home/workspace/test-xunit' for the testing framework 'GoogleTest-1.8'. Did you enter a pattern relative to (and within) the workspace directory? Did you generate the result report(s) for 'GoogleTest-1.8'?" WARNING: All test reports are empty. INFO: Check 'Failed Tests' threshold. INFO: The total number of tests for the threshold 'Failed Tests' exceeds the specified "failure threshold" value. INFO: Setting the build status to FAILURE [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: FAILURE
Kent Barnett
added a comment - This is happening with a pipeline job with no other steps included. Here is the full pipeline script:
pipeline {
agent any
stages {
stage( 'Hello' ) {
steps {
echo 'Hello World'
}
}
}
post {
always {
step([$class : 'XUnitPublisher' , testTimeMargin: '3000' , thresholdMode: 2,
thresholds: [
[
$class : 'FailedThreshold' ,
failureNewThreshold : '100' ,
failureThreshold : '100' ,
unstableNewThreshold: '100' ,
unstableThreshold : '100'
],
[
$class : 'SkippedThreshold' ,
failureNewThreshold : '100' ,
failureThreshold : '100' ,
unstableNewThreshold: '100' ,
unstableThreshold : '100'
]
],
tools : [
[
$class : 'GoogleTestType' ,
deleteOutputFiles : true ,
failIfNotNew : false ,
pattern : '*.xml' ,
skipNoTestFiles : true ,
stopProcessingIfError: true
]
],
sleepTime: 20
])
}
}
}
The full job output is:
Started by user unknown or anonymous
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in / var /jenkins_home/workspace/test-xunit
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Hello)
[Pipeline] echo
Hello World
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] step
INFO: Processing GoogleTest-1.8
INFO: [GoogleTest-1.8] - No test report file(s) were found with the pattern '*.xml' relative to '/ var /jenkins_home/workspace/test-xunit' for the testing framework 'GoogleTest-1.8' .
Did you enter a pattern relative to (and within) the workspace directory?
Did you generate the result report(s) for 'GoogleTest-1.8' ?"
WARNING: All test reports are empty.
INFO: Check 'Failed Tests' threshold.
INFO: The total number of tests for the threshold 'Failed Tests' exceeds the specified "failure threshold" value.
INFO: Setting the build status to FAILURE
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: FAILURE
WARNING: All test reports are empty.
This is the difference. There are test files but empty. The option refers to no files at all. In case there are test files but the suites are empty that apply thresholds.
Anyway is strange becuase if wrote "No test report file(s) were found with the pattern '*.xml' relative...."
Found the issue, it happens in percent mode, by default are in count so it did happend on my tries. There was a 0 by 0 division -> NaN that fails the build result
neither with freestyle job neither pipeline job I able to reproduce this issue. The build result is SUCCESS.
node { stage('Results') { xunit thresholds: [ failed(failureNewThreshold: '100', failureThreshold: '100', unstableNewThreshold: '100', unstableThreshold: '100'), skipped(failureNewThreshold: '100', failureThreshold: '100', unstableNewThreshold: '100', unstableThreshold: '100')], tools: [GoogleTest(excludesPattern: '', failIfNotNew: false, pattern: '*.xml', skipNoTestFiles: true, stopProcessingIfError: true)] } }Please try with an empty project if you have the issue. Also in case previous build was failed and there are test failure the build does not fail, it is success
[Pipeline] { [Pipeline] stage [Pipeline] { (Results) [Pipeline] xunit INFO: Processing GoogleTest-1.8 INFO: [GoogleTest-1.8] - No test report file(s) were found with the pattern '*.xml' relative to 'C:\Users\NIKOLAS-PC\Documents\git\xunit-plugin\work\workspace\pipeline' for the testing framework 'GoogleTest-1.8'. Did you enter a pattern relative to (and within) the workspace directory? Did you generate the result report(s) for 'GoogleTest-1.8'?" INFO: Check 'Failed Tests' threshold. INFO: Check 'Skipped Tests' threshold. [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS