Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12648

TestNG 0.31 does not show test results when tests failed

      Whenever I use the TestNG-Plugin 0.31 it only shows reports if the tests passed. But if the failey there are no test results. If I downgrade to 0.26 for example everthing works fine. Both the failed and passed test results are displayed. Please fix this, would be very important. Thank you!

          [JENKINS-12648] TestNG 0.31 does not show test results when tests failed

          Peter Lustig added a comment -

          Okay, here are two pics of the case where it does not show the results:
          http://img191.imageshack.us/img191/8543/pic1ap.png
          http://img163.imageshack.us/img163/7302/pic2yb.png

          Here are two pics where it shows the results:
          http://img705.imageshack.us/img705/3614/pic3g.png
          http://img195.imageshack.us/img195/9258/pic4ns.png

          See how it says "Not looking for any testNG results" at the bottom of the console where it fails? I don't get it why. There is no option in preferences or something it just fails to show the reports.

          Peter Lustig added a comment - Okay, here are two pics of the case where it does not show the results: http://img191.imageshack.us/img191/8543/pic1ap.png http://img163.imageshack.us/img163/7302/pic2yb.png Here are two pics where it shows the results: http://img705.imageshack.us/img705/3614/pic3g.png http://img195.imageshack.us/img195/9258/pic4ns.png See how it says "Not looking for any testNG results" at the bottom of the console where it fails? I don't get it why. There is no option in preferences or something it just fails to show the reports.

          Jeff Weiss added a comment - - edited

          I installed 0.31 this morning and I don't see this issue either. Peter Lustig, I have seen the message "Not looking for any testNG results" when the previous build step failed (in my case, a shell script step returned nonzero). So the build fails at that step and it makes sense for testNG plugin to not bother looking for results. Maybe that is what you are seeing. I don't know what your config looks like, but you may want to verify that all the build steps complete without error.

          Jeff Weiss added a comment - - edited I installed 0.31 this morning and I don't see this issue either. Peter Lustig, I have seen the message "Not looking for any testNG results" when the previous build step failed (in my case, a shell script step returned nonzero). So the build fails at that step and it makes sense for testNG plugin to not bother looking for results. Maybe that is what you are seeing. I don't know what your config looks like, but you may want to verify that all the build steps complete without error.

          nalin_makar added a comment -

          I figured the same, but I am not sure now that it's the right behavior.

          Sometimes testng ant ask itself can be configured to to fail the build if tests fail, but that should not stop us from reporting the tests that were executed. I made a similar change for aborted builds recently based on bug that kutzi filed and now am wondering if I should revert that behavior as well or not.

          nalin_makar added a comment - I figured the same, but I am not sure now that it's the right behavior. Sometimes testng ant ask itself can be configured to to fail the build if tests fail, but that should not stop us from reporting the tests that were executed. I made a similar change for aborted builds recently based on bug that kutzi filed and now am wondering if I should revert that behavior as well or not.

          Jeff Weiss added a comment -

          nalin, I would think that testng plugin should fail a job that would have otherwise passed, if there is no result file (which it does). And it should mark a job as unstable that otherwise would be success, if the result file is there and contains failed tests (which it also does). But seems like it should always look - even when the job is failed or aborted, the user might want to see the test results, if there are any.

          Jeff Weiss added a comment - nalin, I would think that testng plugin should fail a job that would have otherwise passed, if there is no result file (which it does). And it should mark a job as unstable that otherwise would be success, if the result file is there and contains failed tests (which it also does). But seems like it should always look - even when the job is failed or aborted, the user might want to see the test results, if there are any.

          kutzi added a comment -

          Jeff, I agree that the plugin should look for results when a build failed.
          But not when it was aborted, because - IMHO - cancelling/aborting a build means: 'stop as fast as possible'

          kutzi added a comment - Jeff, I agree that the plugin should look for results when a build failed. But not when it was aborted, because - IMHO - cancelling/aborting a build means: 'stop as fast as possible'

          nalin_makar added a comment -

          kutzi,

          When user aborts a bit, the current executing step should stop asap, but because testng plugin only reports results and doesnt really execute any build steps or modify the workspace, it might be beneficial to run it. Simple use case would be where there are multiple calls to running tests and even though we want to abort the build, we want to make sure that results that were already published are parsed and displayed.

          nalin_makar added a comment - kutzi, When user aborts a bit, the current executing step should stop asap, but because testng plugin only reports results and doesnt really execute any build steps or modify the workspace, it might be beneficial to run it. Simple use case would be where there are multiple calls to running tests and even though we want to abort the build, we want to make sure that results that were already published are parsed and displayed.

          Peter Lustig added a comment -

          Okay, so my testNG-Tests fail and this makes the whole build to fail. And that is the reason why no results are parsed and displayed? I definitely think this is not the right behaviour. I mean, it makes sense, to mark a build as fail if the tests failed. And it definitely makes sense to display test results if the tests failed. But nothing of this behaviour is implemented at the current moment. I would really appreciate it if one can see the results if tests fail.

          Peter Lustig added a comment - Okay, so my testNG-Tests fail and this makes the whole build to fail. And that is the reason why no results are parsed and displayed? I definitely think this is not the right behaviour. I mean, it makes sense, to mark a build as fail if the tests failed. And it definitely makes sense to display test results if the tests failed. But nothing of this behaviour is implemented at the current moment. I would really appreciate it if one can see the results if tests fail.

          nalin_makar added a comment -

          Fix is to revert the change made for fixing JENKINS-12046.

          nalin_makar added a comment - Fix is to revert the change made for fixing JENKINS-12046 .

          Code changed in jenkins
          User: Nalin Makar
          Path:
          README
          src/main/java/hudson/plugins/testng/Publisher.java
          http://jenkins-ci.org/commit/testng-plugin-plugin/523dfea36f91a1425d1f377101e9207d02a48afc
          Log:
          Fixed: JENKINS-12648 - Results should be reported for failed/aborted builds as well (Reverted fix for JENKINS-12046 in v0.30)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Nalin Makar Path: README src/main/java/hudson/plugins/testng/Publisher.java http://jenkins-ci.org/commit/testng-plugin-plugin/523dfea36f91a1425d1f377101e9207d02a48afc Log: Fixed: JENKINS-12648 - Results should be reported for failed/aborted builds as well (Reverted fix for JENKINS-12046 in v0.30)

          nalin_makar added a comment -

          Fix will be available as part of v0.32

          nalin_makar added a comment - Fix will be available as part of v0.32

            nalin_makar nalin_makar
            gurkensalat Peter Lustig
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: