Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-35169

Jenkins "Job status" shows success even if it has many / all test cases failed

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not A Defect
    • Icon: Critical Critical
    • testng-plugin
    • None
    • Windows 7, Chrome 50.0.2661.102, Jenkins 2.5,2.6

      Hi All,

      We recently upgraded Jenkins from 2.1 to 2.6. After which Job Status is always success once the execution is completed.

      For example : if a job as 24 test cases and 24 of them also failed, in dashboard the status is still shown as success.

      Please refer to attached screen shot.

      in which test result clearly says that there are 9 failures out of 30 test cases, but still the job status is showing as success. which was not the case with Previous version of Jenkins 2.1.

      Thanks,
      pavan

        1. TestNGConfigaration.PNG
          TestNGConfigaration.PNG
          37 kB
        2. TestngResult.PNG
          TestngResult.PNG
          5 kB

          [JENKINS-35169] Jenkins "Job status" shows success even if it has many / all test cases failed

          pavan pothnis added a comment -

          we have downgraded Jenkins version to 2.5 still no help.

          pavan pothnis added a comment - we have downgraded Jenkins version to 2.5 still no help.

          Tomas Westling added a comment -

          In which way is this problem related to the build failure analyzer plugin?

          Tomas Westling added a comment - In which way is this problem related to the build failure analyzer plugin?

          pavan pothnis added a comment -

          I was under impression that build failure analyzer is supposed analyze the result and show the status.

          If not, could you please let me know what could be the issue for which Jenkins Shows Job Status as Success, even when many test cases have failed.

          Thanks,
          Pavan

          pavan pothnis added a comment - I was under impression that build failure analyzer is supposed analyze the result and show the status. If not, could you please let me know what could be the issue for which Jenkins Shows Job Status as Success, even when many test cases have failed. Thanks, Pavan

          Tomas Westling added a comment -

          The build failure analyzer plugin is used to quickly tell a user what went wrong with a build when it failed, by matching regular expressions
          with the build log. It doesn't have any power over which builds fail or not.

          Since TestNG tells you that tests failed, I would start there, with the testng-plugin.

          Br Tomas

          Tomas Westling added a comment - The build failure analyzer plugin is used to quickly tell a user what went wrong with a build when it failed, by matching regular expressions with the build log. It doesn't have any power over which builds fail or not. Since TestNG tells you that tests failed, I would start there, with the testng-plugin. Br Tomas

          pavan pothnis added a comment -

          As per Tomas Suggestion, I looked into Publish TestNG plugin configuration (See attached Screenshot).

          The threshold is set to mark build unstable / failure when there is 100 % of test cases fails or gets skipped.

          However, even when we have 11/11 Test cases failed. build status is still shown as Success.

          Thanks,
          Pavan

          pavan pothnis added a comment - As per Tomas Suggestion, I looked into Publish TestNG plugin configuration (See attached Screenshot). The threshold is set to mark build unstable / failure when there is 100 % of test cases fails or gets skipped. However, even when we have 11/11 Test cases failed. build status is still shown as Success. Thanks, Pavan

          pavan pothnis added a comment -

          Trying to solve this issue with JENKINS-35254.

          Thanks Tom.

          pavan pothnis added a comment - Trying to solve this issue with JENKINS-35254 . Thanks Tom.

            t_westling Tomas Westling
            ppothnis pavan pothnis
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: