Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61263

JUnit support flaky test - for multiple records of the same test in xml report

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      We started to use retry policy while running our tests, because some tests are flaky. We use https://github.com/gradle/test-retry-gradle-plugin for that. The thing is, that a test might appear N times in junit XML report, where N is number of retries. For example 2 times as a "failed" test, and 1 time as "passes". Such test ideally should be marked as flaky and depending on a junit configuration it should not mark the build as unstable/failed. 

      Current behaviour:

      It's enough if 1 run of a given test fails and that will make junit plugin to mark build as unstable. Even though there are also positive passes for the same test (separate records in xml).

      Expected behaviour:

      Group tests by name, if at least one run is "passes", mark the test as:

      a) ideal solution: flaky one

      b) good enough: "passes" one.

       

      There is some another junit jenkins plugin that seems to support flaky tests, but is no longer supported (4 years now) and does not work anymore. It would be nice to add such feature in junit plugin, as discovering flaky tests seems to become a standard.

        Attachments

          Activity

          There are no comments yet on this issue.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            mwos Michał Woś
            Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated: