Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61198

Thresholds not reliable

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not A Defect
    • Icon: Minor Minor
    • xunit-plugin
    • None

      We have a few jobs that use the xUnit plugin with multiple sources to collect results from multiple test frameworks. The build should be marked as unstable if there is a single test failure.

      In several builds, the build is marked as successful with multiple test failures. My suspicion is that thresholds are checked only for one of the sources, not all of them cumulatively.

      Example build

        1. warn.png
          warn.png
          114 kB

          [JENKINS-61198] Thresholds not reliable

          Nikolas Falco added a comment -

          As you can see from the logs the threadshold check is performed at the end.

          INFO: Starting to record.
INFO: Processing BoostTest-1.x (default)
INFO: [BoostTest-1.x (default)] - 7 test report file(s) were found with the pattern 'build/qa/**/*.boost-results.xml' relative to '/var/lib/jenkins-agent/workspace/linux-kicad-head' for the testing framework 'BoostTest-1.x (default)'.
INFO: Processing JUnit
INFO: [JUnit] - 6 test report file(s) were found with the pattern 'build/qa/*.xunit-results/*.xml' relative to '/var/lib/jenkins-agent/workspace/linux-kicad-head' for the testing framework 'JUnit'.
INFO: Processing CTest-Version 3.x (default)
INFO: [CTest-Version 3.x (default)] - 2 test report file(s) were found with the pattern 'build/Testing/**/*.xml' relative to '/var/lib/jenkins-agent/workspace/linux-kicad-head' for the testing framework 'CTest-Version 3.x (default)'.
INFO: Check 'Failed Tests' threshold.
INFO: Setting the build status to SUCCESS
INFO: Stopping recording.

          How did you configure the threshold?

          Nikolas Falco added a comment - As you can see from the logs the threadshold check is performed at the end. INFO: Starting to record. INFO: Processing BoostTest-1.x (default) INFO: [BoostTest-1.x (default)] - 7 test report file(s) were found with the pattern 'build/qa/**/*.boost-results.xml' relative to '/var/lib/jenkins-agent/workspace/linux-kicad-head' for the testing framework 'BoostTest-1.x (default)'. INFO: Processing JUnit INFO: [JUnit] - 6 test report file(s) were found with the pattern 'build/qa/*.xunit-results/*.xml' relative to '/var/lib/jenkins-agent/workspace/linux-kicad-head' for the testing framework 'JUnit'. INFO: Processing CTest-Version 3.x (default) INFO: [CTest-Version 3.x (default)] - 2 test report file(s) were found with the pattern 'build/Testing/**/*.xml' relative to '/var/lib/jenkins-agent/workspace/linux-kicad-head' for the testing framework 'CTest-Version 3.x (default)'. INFO: Check 'Failed Tests' threshold. INFO: Setting the build status to SUCCESS INFO: Stopping recording. How did you configure the threshold?

          Simon Richter added a comment - - edited

          Thresholds are "1", "1", empty, empty:

          With 1 failing test, I'd expect this to give me "unstable"

          Simon Richter added a comment - - edited Thresholds are "1", "1", empty, empty: With 1 failing test, I'd expect this to give me "unstable"

          Simon Richter added a comment -

          Wait, am I being stupid?

          Simon Richter added a comment - Wait, am I being stupid?

          Simon Richter added a comment -

          Okay, that was me being stupid and interpreting "greater than" as "greater or equal".

          Simon Richter added a comment - Okay, that was me being stupid and interpreting "greater than" as "greater or equal".

            nfalco Nikolas Falco
            simonrichter Simon Richter
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: