Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12616

Tests failes but the job is still OK

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Fix
    • Icon: Minor Minor
    • xunit-plugin
    • None
    • jenkins 1.445
      xunit 1.34

      Ubuntu server 11.11

      just use the attached result file with BOOST

      the error is listed correct in the test report, but the job is still OK but need to be at least UNSTABLE.
      As you can see in the screenshot I do not have any thresholds set.

      I do not have any example at the moment, but I think we have this problem also with NUnit tests.

        1. boost-config.PNG
          25 kB
          Thomas Fürer
        2. failedButOK.PNG
          23 kB
          Thomas Fürer
        3. TEST-failed.xml
          0.4 kB
          Thomas Fürer

          [JENKINS-12616] Tests failes but the job is still OK

          Vladimir Zak added a comment - - edited

          yes, the same setting and the same behavior - this is problem with cppunit too.

          Vladimir Zak added a comment - - edited yes, the same setting and the same behavior - this is problem with cppunit too.

          Gregory Boissinot added a comment -

          If you don't have any threshold set, the expected build result is always set to SUCCESS.

          Gregory Boissinot added a comment - If you don't have any threshold set, the expected build result is always set to SUCCESS.

          Thomas Fürer added a comment -

          it is the same behavior if I set a threshold of
          Build Status Total New Total New
          Thresholds: 1 1 10 10

          I'm using the file attached which has only one failing test.

          Thomas Fürer added a comment - it is the same behavior if I set a threshold of Build Status Total New Total New Thresholds: 1 1 10 10 I'm using the file attached which has only one failing test.

          Gregory Boissinot added a comment -

          If no values are set in the threshold section, threshold process is not activated.

          In your case, you have one failed test and if you set the value '0' in 'Total' column, the build should be set to UNSTABLE (the number of test must exceed strict the value).
          It's OK in my environment, could you confirm?

          Gregory Boissinot added a comment - If no values are set in the threshold section, threshold process is not activated. In your case, you have one failed test and if you set the value '0' in 'Total' column, the build should be set to UNSTABLE (the number of test must exceed strict the value). It's OK in my environment, could you confirm?

          Thomas Fürer added a comment -

          setting threshold values solves the problem

          Thomas Fürer added a comment - setting threshold values solves the problem

          Thomas Fürer added a comment -

          because this is a change of the default behaivor I suggest to set default thresholds so set the default behavior is set.

          Thomas Fürer added a comment - because this is a change of the default behaivor I suggest to set default thresholds so set the default behavior is set.

          Gregory Boissinot added a comment -

          Giving default values for threshold is difficult: failed test or skipped test; unstable or fail.

          The response depends on the context.

          My choice: if no values are set, no threshold is processed.

          Gregory Boissinot added a comment - Giving default values for threshold is difficult: failed test or skipped test; unstable or fail. The response depends on the context. My choice: if no values are set, no threshold is processed.

            gbois Gregory Boissinot
            tofuatjava Thomas Fürer
            Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: