-
Improvement
-
Resolution: Unresolved
-
Major
Feature request:
If tests in a test suite often flick between failure and pass, it would be nice to report on this right in the test results screen in blue ocean.
https://wiki.jenkins-ci.org/display/JENKINS/Flaky+Test+Handler+Plugin shows some of the functionality, but this is not about rerunning bad things, but just identifying that "often this test fails, but then passes in a subsequent run when there were no code changes.
A fairly simple heuristic could be used - doens't necessarily need to use stats over many builds. If any test failed, but then passed due a subsequent run when there were NO CODE changes, mark it as flaky. This could also be applicable across branches (ie use the default branch mainly).
This would mean when you see failures, you know that you can run it again, but also that you really should get your shit together and fix that test somehow.
- relates to
-
JENKINS-45118 Per-test history in Test tab
- Open