-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
None
This is spin-off issue from https://issues.jenkins-ci.org/browse/JENKINS-54255
i think that it is better not to abuse old closed issue with this one.
here is the summary of discussion i had with Nicholas.:
======================================
i see that xunit thresholds produce non deterministic results in parallel pipe runs
This is how i define xunit thresholds:
xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')], tools: [Custom(customXSL: "${JENKINS_URL}/userContent/pytest-to-junit.xsl", deleteOutputFiles: true, failIfNotNew: true, pattern: "${env.DOWNLOADED_ARTIFACTS_PATH}/${env.TEST_RESULTS_PATH}/*.xml", skipNoTestFiles: true, stopProcessingIfError: true)]
I run few parallel branches of jenkins pipe, each in its own workspace and each pipe branch uploads it test results independently.
I have some build new failed tests:
Test Result (40 failures / +17) |
i expect to get failed status, but xunit still sets the build to be unstable:
01:57:59.736 INFO: Starting to record. 01:57:59.736 INFO: Processing Custom Tool 01:58:00.146 INFO: Stopping recording. 01:58:00.147 INFO: Check 'Failed Tests' threshold. 01:58:00.147 INFO: The new number of tests for the threshold 'Failed Tests' exceeds the specified "new failure threshold" value.
I use:
- Jenkins ver. 2.160
- xUnit plugin 2.3.2
some more details:
I want to define following behavior:
Any new failed test to fail the build, if no new failures, any old failure to cause to unstable build status.
This is how I use Snippet Generator
this is what it produces:
xunit thresholds: [failed(failureNewThreshold: '0', unstableThreshold: '0')], tools: []
Important note: i see that in some cases it behaves just right, but sometimes it produces yellow status on new failed tests, just as described above.
Button line of previous discussion with Nicolas was:Failed thresholds are analysed before unstable thresholds. Total count of test failure are taken into account before of New test failures. This logic is applied to all thresholds (skipped, failed and passed). This means your scenario is not possible.
That doesn't make sense to me:
Failed thresholds are analysed before unstable thresholds: that's ok, in example above only was supposed to kick-in, i hope the logic says that if one threshold declared failure, later threshold cannot cancel it.
Total count of test failure are taken into account before of New test failures: I hope that means no matter how many total failure there are, in my case new failures will define the outcome.
So, after all, it seems that there is a problem with my results, and again, as i wrote above, in most cases it behaves ok, may be related to some race between parallel parts of pipe