Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45014

Developer is able to see which tests are intermittent

      Feature request: 

      If tests in a test suite often flick between failure and pass, it would be nice to report on this right in the test results screen in blue ocean. 

       

      https://wiki.jenkins-ci.org/display/JENKINS/Flaky+Test+Handler+Plugin shows some of the functionality, but this is not about rerunning bad things, but just identifying that "often this test fails, but then passes in a subsequent run when there were no code changes. 

      A fairly simple heuristic could be used - doens't necessarily need to use stats over many builds. If any test failed, but then passed due a subsequent run when there were NO CODE changes, mark it as flaky. This could also be applicable across branches (ie use the default branch mainly). 

       

      This would mean when you see failures, you know that you can run it again, but also that you really should get your shit together and fix that test somehow. 

          [JENKINS-45014] Developer is able to see which tests are intermittent

          Michael Neale added a comment -

          jamesdumay I think this is something that could just be baked in - any time the tests tab is used. 

          Michael Neale added a comment - jamesdumay I think this is something that could just be baked in - any time the tests tab is used. 

          James Dumay added a comment -

          Yes, this would be most helpful. Not sure how this would work with the 10,000+ tests that some of our users have. Another case where this problem could be easily solved with a DB query.

          James Dumay added a comment - Yes, this would be most helpful. Not sure how this would work with the 10,000+ tests that some of our users have. Another case where this problem could be easily solved with a DB query.

            Unassigned Unassigned
            michaelneale Michael Neale
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: