Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61198

Thresholds not reliable

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      We have a few jobs that use the xUnit plugin with multiple sources to collect results from multiple test frameworks. The build should be marked as unstable if there is a single test failure.

      In several builds, the build is marked as successful with multiple test failures. My suspicion is that thresholds are checked only for one of the sources, not all of them cumulatively.

      Example build

        Attachments

          Activity

          Hide
          nfalco Nikolas Falco added a comment -

          As you can see from the logs the threadshold check is performed at the end.

          INFO: Starting to record.
          INFO: Processing BoostTest-1.x (default)
          INFO: [BoostTest-1.x (default)] - 7 test report file(s) were found with the pattern 'build/qa/**/*.boost-results.xml' relative to '/var/lib/jenkins-agent/workspace/linux-kicad-head' for the testing framework 'BoostTest-1.x (default)'.
          INFO: Processing JUnit
          INFO: [JUnit] - 6 test report file(s) were found with the pattern 'build/qa/*.xunit-results/*.xml' relative to '/var/lib/jenkins-agent/workspace/linux-kicad-head' for the testing framework 'JUnit'.
          INFO: Processing CTest-Version 3.x (default)
          INFO: [CTest-Version 3.x (default)] - 2 test report file(s) were found with the pattern 'build/Testing/**/*.xml' relative to '/var/lib/jenkins-agent/workspace/linux-kicad-head' for the testing framework 'CTest-Version 3.x (default)'.
          INFO: Check 'Failed Tests' threshold.
          INFO: Setting the build status to SUCCESS
          INFO: Stopping recording.

          How did you configure the threshold?

          Show
          nfalco Nikolas Falco added a comment - As you can see from the logs the threadshold check is performed at the end. INFO: Starting to record. INFO: Processing BoostTest-1.x (default) INFO: [BoostTest-1.x (default)] - 7 test report file(s) were found with the pattern 'build/qa/**/*.boost-results.xml' relative to '/var/lib/jenkins-agent/workspace/linux-kicad-head' for the testing framework 'BoostTest-1.x (default)'. INFO: Processing JUnit INFO: [JUnit] - 6 test report file(s) were found with the pattern 'build/qa/*.xunit-results/*.xml' relative to '/var/lib/jenkins-agent/workspace/linux-kicad-head' for the testing framework 'JUnit'. INFO: Processing CTest-Version 3.x (default) INFO: [CTest-Version 3.x (default)] - 2 test report file(s) were found with the pattern 'build/Testing/**/*.xml' relative to '/var/lib/jenkins-agent/workspace/linux-kicad-head' for the testing framework 'CTest-Version 3.x (default)'. INFO: Check 'Failed Tests' threshold. INFO: Setting the build status to SUCCESS INFO: Stopping recording. How did you configure the threshold?
          Hide
          simonrichter Simon Richter added a comment - - edited

          Thresholds are "1", "1", empty, empty:

          With 1 failing test, I'd expect this to give me "unstable"

          Show
          simonrichter Simon Richter added a comment - - edited Thresholds are "1", "1", empty, empty: With 1 failing test, I'd expect this to give me "unstable"
          Hide
          simonrichter Simon Richter added a comment -

          Wait, am I being stupid?

          Show
          simonrichter Simon Richter added a comment - Wait, am I being stupid?
          Hide
          simonrichter Simon Richter added a comment -

          Okay, that was me being stupid and interpreting "greater than" as "greater or equal".

          Show
          simonrichter Simon Richter added a comment - Okay, that was me being stupid and interpreting "greater than" as "greater or equal".

            People

            Assignee:
            nfalco Nikolas Falco
            Reporter:
            simonrichter Simon Richter
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: