Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45014

Developer is able to see which tests are intermittent

      Feature request: 

      If tests in a test suite often flick between failure and pass, it would be nice to report on this right in the test results screen in blue ocean. 

       

      https://wiki.jenkins-ci.org/display/JENKINS/Flaky+Test+Handler+Plugin shows some of the functionality, but this is not about rerunning bad things, but just identifying that "often this test fails, but then passes in a subsequent run when there were no code changes. 

      A fairly simple heuristic could be used - doens't necessarily need to use stats over many builds. If any test failed, but then passed due a subsequent run when there were NO CODE changes, mark it as flaky. This could also be applicable across branches (ie use the default branch mainly). 

       

      This would mean when you see failures, you know that you can run it again, but also that you really should get your shit together and fix that test somehow. 

          [JENKINS-45014] Developer is able to see which tests are intermittent

          Michael Neale created issue -
          Michael Neale made changes -
          Epic Link New: JENKINS-36296 [ 172305 ]
          James Dumay made changes -
          Epic Link Original: JENKINS-36296 [ 172305 ] New: JENKINS-43952 [ 181484 ]
          Michael Neale made changes -
          Labels New: feature-request

          Michael Neale added a comment -

          jamesdumay I think this is something that could just be baked in - any time the tests tab is used. 

          Michael Neale added a comment - jamesdumay I think this is something that could just be baked in - any time the tests tab is used. 

          James Dumay added a comment -

          Yes, this would be most helpful. Not sure how this would work with the 10,000+ tests that some of our users have. Another case where this problem could be easily solved with a DB query.

          James Dumay added a comment - Yes, this would be most helpful. Not sure how this would work with the 10,000+ tests that some of our users have. Another case where this problem could be easily solved with a DB query.
          James Dumay made changes -
          Labels Original: feature-request New: db feature-request
          James Dumay made changes -
          Summary Original: Flaky test snitch New: Developer is able to see which tests are intermittent
          James Dumay made changes -
          Link New: This issue relates to JENKINS-45118 [ JENKINS-45118 ]

            Unassigned Unassigned
            michaelneale Michael Neale
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: