Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61263

JUnit support flaky test - for multiple records of the same test in xml report

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      We started to use retry policy while running our tests, because some tests are flaky. We use https://github.com/gradle/test-retry-gradle-plugin for that. The thing is, that a test might appear N times in junit XML report, where N is number of retries. For example 2 times as a "failed" test, and 1 time as "passes". Such test ideally should be marked as flaky and depending on a junit configuration it should not mark the build as unstable/failed. 

      Current behaviour:

      It's enough if 1 run of a given test fails and that will make junit plugin to mark build as unstable. Even though there are also positive passes for the same test (separate records in xml).

      Expected behaviour:

      Group tests by name, if at least one run is "passes", mark the test as:

      a) ideal solution: flaky one

      b) good enough: "passes" one.

       

      There is some another junit jenkins plugin that seems to support flaky tests, but is no longer supported (4 years now) and does not work anymore. It would be nice to add such feature in junit plugin, as discovering flaky tests seems to become a standard.

        Attachments

          Activity

          Hide
          vtintillier Vivien Tintillier added a comment -

          Hi Michał Woś

          I quite agree this should likely be part of the main junit plugin, but in the meantime I became maintainer of https://plugins.jenkins.io/flaky-test-handler/ and released a new version today.

          Maybe you can try again this new 1.1.0 version?

          Regards

          Show
          vtintillier Vivien Tintillier added a comment - Hi Michał Woś I quite agree this should likely be part of the main junit plugin, but in the meantime I became maintainer of https://plugins.jenkins.io/flaky-test-handler/  and released a new version today. Maybe you can try again this new 1.1.0 version? Regards
          Hide
          mwos Michał Woś added a comment -

          Vivien Tintillier Thank you for letting me know! I added to our backlog testing this plugin. It won't happen soon due to the other priorities and workaround we applied (we deduplicate records in xml before providing to junit plugin). But looks promising and surely we'll try at some point, as it would be way better to have "flakiness" information in the report.

          Show
          mwos Michał Woś added a comment - Vivien Tintillier  Thank you for letting me know! I added to our backlog testing this plugin. It won't happen soon due to the other priorities and workaround we applied (we deduplicate records in xml before providing to junit plugin). But looks promising and surely we'll try at some point, as it would be way better to have "flakiness" information in the report.
          Hide
          gfjalar Piotr Galar added a comment -

          Hi Vivien Tintillier,

          We've tried Flaky Test Handler plugin (v1.1.0). Some feedback after initial testing -> we couldn't find any indication about flakiness if all the tests eventually passed during a run. In particular, we have tested the plugin using the test XML from the repo https://github.com/jenkinsci/flaky-test-handler-plugin/blob/master/src/test/resources/com/google/jenkins/flakyTestHandler/junit/flaky-reports/flaky-report-1.xml. I'm attaching screenshots of how test reports looks like with that exact report vs. the same report but with <testcase classname="test.foo.bar.BundleResolverIntegrationTest" name="testGetBundle" time="0.0" > removed.

          Jenkins version: 2.284
          Junit plugin version: 1.49

          Flaky + failing:

          Just flaky:

          Show
          gfjalar Piotr Galar added a comment - Hi Vivien Tintillier , We've tried Flaky Test Handler plugin (v1.1.0). Some feedback after initial testing -> we couldn't find any indication about flakiness if all the tests eventually passed during a run. In particular, we have tested the plugin using the test XML from the repo https://github.com/jenkinsci/flaky-test-handler-plugin/blob/master/src/test/resources/com/google/jenkins/flakyTestHandler/junit/flaky-reports/flaky-report-1.xml.  I'm attaching screenshots of how test reports looks like with that exact report vs. the same report but with  <testcase classname="test.foo.bar.BundleResolverIntegrationTest" name="testGetBundle" time="0.0" >  removed. Jenkins version: 2.284 Junit plugin version: 1.49 Flaky + failing: Just flaky:
          Hide
          vtintillier Vivien Tintillier added a comment -

          Hi Piotr Galar

          thank you for the report, I will look into it.

          Would you mind creating a separate bug report specific to the flaky test handler plugin?

          Show
          vtintillier Vivien Tintillier added a comment - Hi Piotr Galar thank you for the report, I will look into it. Would you mind creating a separate bug report specific to the flaky test handler plugin?
          Hide
          vtintillier Vivien Tintillier added a comment -

          I reproduce the issue with Jenkins 2.262.2 and 2.222.4. The JUnitFlakyTestDataAction seems to be here just fine when I debug with 2.222.4 locally, but it is like the Jelly is not called, or is not working anymore.

          Unfortunately, I have no idea how to debug that Jelly code further...

          Sven Jost does it still work for you? With which Jenkins version?

          Show
          vtintillier Vivien Tintillier added a comment - I reproduce the issue with Jenkins 2.262.2 and 2.222.4. The JUnitFlakyTestDataAction seems to be here just fine when I debug with 2.222.4 locally, but it is like the Jelly is not called, or is not working anymore. Unfortunately, I have no idea how to debug that Jelly code further... Sven Jost does it still work for you? With which Jenkins version?
          Hide
          l_r Louis-Rémi Paquet added a comment -

          With recent versions of Gradle, you can use mergeReRuns

          https://docs.gradle.org/current/userguide/java_testing.html#junit_xml_configuration

          Show
          l_r Louis-Rémi Paquet added a comment - With recent versions of Gradle, you can use mergeReRuns https://docs.gradle.org/current/userguide/java_testing.html#junit_xml_configuration
          Hide
          svenjost Sven Jost added a comment -

          Vivien Tintillier I finally have some time to look into it. I opened a new issue to not spam this one here: JENKINS-66126

          Show
          svenjost Sven Jost added a comment - Vivien Tintillier I finally have some time to look into it. I opened a new issue to not spam this one here: JENKINS-66126

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            mwos Michał Woś
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated: