Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61198

Thresholds not reliable

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      We have a few jobs that use the xUnit plugin with multiple sources to collect results from multiple test frameworks. The build should be marked as unstable if there is a single test failure.

      In several builds, the build is marked as successful with multiple test failures. My suspicion is that thresholds are checked only for one of the sources, not all of them cumulatively.

      Example build

        Attachments

          Activity

          simonrichter Simon Richter created issue -
          simonrichter Simon Richter made changes -
          Field Original Value New Value
          Description We have a few jobs that use the xUnit plugin with multiple sources to collect results from multiple test frameworks. The build should be marked as unstable if there is a single test failure.

          In several builds, the build is marked as successful with multiple test failures. My suspicion is that thresholds are checked only for one of the sources, not all of them cumulatively.

          [Example build](https://jenkins.simonrichter.eu/job/linux-kicad-head/4211/)
          We have a few jobs that use the xUnit plugin with multiple sources to collect results from multiple test frameworks. The build should be marked as unstable if there is a single test failure.

          In several builds, the build is marked as successful with multiple test failures. My suspicion is that thresholds are checked only for one of the sources, not all of them cumulatively.

          [[Example build]|[https://jenkins.simonrichter.eu/job/linux-kicad-head/4211/]|https://jenkins.simonrichter.eu/job/linux-kicad-head/4211/]
          simonrichter Simon Richter made changes -
          Description We have a few jobs that use the xUnit plugin with multiple sources to collect results from multiple test frameworks. The build should be marked as unstable if there is a single test failure.

          In several builds, the build is marked as successful with multiple test failures. My suspicion is that thresholds are checked only for one of the sources, not all of them cumulatively.

          [[Example build]|[https://jenkins.simonrichter.eu/job/linux-kicad-head/4211/]|https://jenkins.simonrichter.eu/job/linux-kicad-head/4211/]
          We have a few jobs that use the xUnit plugin with multiple sources to collect results from multiple test frameworks. The build should be marked as unstable if there is a single test failure.

          In several builds, the build is marked as successful with multiple test failures. My suspicion is that thresholds are checked only for one of the sources, not all of them cumulatively.

          [Example build|[https://jenkins.simonrichter.eu/job/linux-kicad-head/4211/]
          simonrichter Simon Richter made changes -
          Description We have a few jobs that use the xUnit plugin with multiple sources to collect results from multiple test frameworks. The build should be marked as unstable if there is a single test failure.

          In several builds, the build is marked as successful with multiple test failures. My suspicion is that thresholds are checked only for one of the sources, not all of them cumulatively.

          [Example build|[https://jenkins.simonrichter.eu/job/linux-kicad-head/4211/]
          We have a few jobs that use the xUnit plugin with multiple sources to collect results from multiple test frameworks. The build should be marked as unstable if there is a single test failure.

          In several builds, the build is marked as successful with multiple test failures. My suspicion is that thresholds are checked only for one of the sources, not all of them cumulatively.

          [Example build|https://jenkins.simonrichter.eu/job/linux-kicad-head/4211/]
          simonrichter Simon Richter made changes -
          Attachment warn.png [ 50398 ]
          simonrichter Simon Richter made changes -
          Resolution Not A Defect [ 7 ]
          Status Open [ 1 ] Fixed but Unreleased [ 10203 ]
          simonrichter Simon Richter made changes -
          Status Fixed but Unreleased [ 10203 ] Closed [ 6 ]

            People

            Assignee:
            nfalco Nikolas Falco
            Reporter:
            simonrichter Simon Richter
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: