-
Bug
-
Resolution: Fixed
-
Major
-
Jenkins 2.19
junit 1.18
I'm using JUnitResultsArchive to record test results, and I'm seeing a weird bug where there are failures reported but the build is not marked unstable.
This is via Jenkins pipeline, and immediately after the step, I'm echoing the build result:
step([$class: 'JUnitResultArchiver', testResults: 'build/xunit.xml']) echo ">>> BUILD RESULT ${currentBuild.result}"
I can confirm that a test failed in that step, and the result printed is ">>> BUILD RESULT null"
This doesn't always happen - in some other runs, the result is UNSTABLE as expected.
- links to
[JENKINS-37663] Build is not marked as UNSTABLE even though failures are found
dantran Thanks for bringing this to my attention - will dig into it tomorrow.
So this is due to this line here - we're calling getContext().setResult(Result.UNSTABLE), which is setting the result on the CpsFlowExecution but not on the run directly. The run still ends up getting set to UNSTABLE at completion, but not until then. That is, in theory, the right behavior overall - once we eventually get better per-step/stage status granularity, we want to avoid setting the Run result directly. But Declarative's post doesn't currently look at the CpsFlowExecution's result, just the Run result.
What I'm thinking right now is updating Declarative to look at both the CpsFlowExecution result (which doesn't get set if you just set the Run result directly) and the Run result (which doesn't get set if you just set the CpsFlowExecution result directly) - that'd let us leave junit doing the forward-looking Right Thing while fixing this regression.
Ok, PRs up for Declarative (actually fixing the post behavior) and junit (verifying the Declarative fix).
Code changed in jenkins
User: Andrew Bayer
Path:
pipeline-model-definition/pom.xml
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/conditions/Aborted.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/conditions/Changed.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/conditions/Failure.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/conditions/NotBuilt.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/conditions/Success.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/conditions/Unstable.groovy
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/BuildConditionResponderTest.java
pipeline-model-definition/src/test/resources/contextResultOverridesRunResult.groovy
pipeline-model-definition/src/test/resources/junitResult.xml
pipeline-model-extensions/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/model/BuildCondition.java
pom.xml
http://jenkins-ci.org/commit/pipeline-model-definition-plugin/d533dd551dadca3d278f3c877aead9016d514b8c
Log:
[FIXED JENKINS-37663] Check execution result in post as well
The new junit step doesn't set the Run to UNSTABLE directly, it sets
the context (and therefore the execution) result to UNSTABLE. We
should be checking the execution result as well as the Run result.
Some time back, declarative pipelines were marking a stage as unstable, and I was able to have something like:
pipeline {
stage {
steps {
sh 'runuittests'
}
post {
always {
junit 'junit.xml'
}
}
}
stage {
when {
expression {
currentBuild.resultIsBetterOrEqualTo('SUCCESS')
}
}
steps {
// Only runs when tests are stable
}
}
}
However, this stopped working at some point in the recent past. I'm not sure what the cause was, but was hoping this ticket would resolve the issue.
I have applied the latest update, but this is still not working.
abayer can you suggest how one might prevent, in a declarative manner or with an expression, a build stage from executing if the build is marked unstable in a previous stage? Or is there still a bug present?
Will Freeman
added a comment - - edited Some time back, declarative pipelines were marking a stage as unstable, and I was able to have something like:
pipeline {
stage {
steps {
sh 'runuittests'
}
post {
always {
junit 'junit.xml'
}
}
}
stage {
when {
expression {
currentBuild.resultIsBetterOrEqualTo( 'SUCCESS' )
}
}
steps {
// Only runs when tests are stable
}
}
}
However, this stopped working at some point in the recent past. I'm not sure what the cause was, but was hoping this ticket would resolve the issue.
I have applied the latest update, but this is still not working.
abayer can you suggest how one might prevent, in a declarative manner or with an expression, a build stage from executing if the build is marked unstable in a previous stage? Or is there still a bug present? therealwaldo Aaaaah. I thought of that and then forgot it because argh. Could you open another ticket against the workflow-cps-plugin component for currentBuild.result not picking up the junit status? Thanks.
Will Freeman
added a comment - abayer filed as JENKINS-48178
I started to see this issue after upgrading to 1.22.x at my declarative pipeline. Ended up to revert to 1.21
our pipeline maven surefire test stages with test failure set to ignore, and use JUnit plugin at post always block to detect build failure state.
abayer , I think this is reproducible regression
post {
always {
//detect test failure
junit allowEmptyResults: true, testResults: '**/junitreports/.xml
}
success{
//this always gets called even with test failure in always block
sendEmail(config, 'SUCCESS', 'ecdm-mail')}
}
....
}