-
New Feature
-
Resolution: Fixed
-
Major
-
None
Current version of TestNG plugin does not have a possibility to use threshold values for failed and skipped tests. If some build has at least one failed test, then this build will be marked as unstable.
Maybe it would be better to have some mechanism to mark build as successfull instead of unstable with using a threshold value for failed tests.
- is related to
-
JENKINS-34822 TestNG plugin has incorrect default threshold values
-
- Open
-
- relates to
-
-
- Open
-
-
-
- Closed
-
[JENKINS-20985] TestNG plugin should has a possibility to use thresholds for failed tests
Chris Everling
added a comment - During a company hackathon, a team of co-workers and myself worked on this ticket (as well as others). Here is the PR link: https://github.com/jenkinsci/testng-plugin-plugin/pull/20/files
Chris Everling
added a comment - During a company hackathon, a team of co-workers and myself worked on this ticket (as well as others). Here is the PR link: https://github.com/jenkinsci/testng-plugin-plugin/pull/20/files 
rakesh singh
added a comment - Is there any update is available to mark build as Stable if some configuration test cases failed ?
rakesh singh
added a comment - Is there any update is available to mark build as Stable if some configuration test cases failed ? 
Nalin Makar
added a comment - https://github.com/jenkinsci/testng-plugin-plugin/commit/52a4637968ea1407ae09be257992e4c0ef4d3c41
merged pull #20.
Nalin Makar
added a comment - https://github.com/jenkinsci/testng-plugin-plugin/commit/52a4637968ea1407ae09be257992e4c0ef4d3c41
merged pull #20. 
Cam Spencer
added a comment - Hey nalin_makar, I don't think the default threshold value should be 100%. We updated the plugin without noticing this feature, and this was causing tests that failed to show as stable.
Cam Spencer
added a comment - Hey nalin_makar , I don't think the default threshold value should be 100%. We updated the plugin without noticing this feature, and this was causing tests that failed to show as stable. Nalin Makar
added a comment - cwcam, can you please file a separate bug? And, it'll be great if you can give specific information about the properties you are referring to. Thanks!
Nalin Makar
added a comment - cwcam , can you please file a separate bug? And, it'll be great if you can give specific information about the properties you are referring to. Thanks! Chris Everling
added a comment - cwcam, nalin_makar, I don't think that is a bug. The default behavior before the update was to mark the build as UNSTABLE whenever there were test failures. The default failure threshold to FAIL the build is 100%, but the default failure threshold to mark the build as UNSTABLE is 0. The default behavior should not have changed with the update.
Chris Everling
added a comment - cwcam , nalin_makar , I don't think that is a bug. The default behavior before the update was to mark the build as UNSTABLE whenever there were test failures. The default failure threshold to FAIL the build is 100%, but the default failure threshold to mark the build as UNSTABLE is 0. The default behavior should not have changed with the update. 
James Theisen
added a comment - The default is fixed, but now I am stuck in a situation where all my jobs are set to 100 for failed tests to be considered unstable. Ungrading to 1.13 fixes the default for new jobs. But how about the rest of us that upgraded to 1.12 and ALL our jobs are stuck with 100% for this value? I guess I am stuck with having the server admin help me change all the job xmls directly (since I dont have access). Or maybe you want to come up with a config slicer or something for this. Any other suggestions on how I can correct this now since your initial release was very poorly configured and tested?
James Theisen
added a comment - The default is fixed, but now I am stuck in a situation where all my jobs are set to 100 for failed tests to be considered unstable. Ungrading to 1.13 fixes the default for new jobs. But how about the rest of us that upgraded to 1.12 and ALL our jobs are stuck with 100% for this value? I guess I am stuck with having the server admin help me change all the job xmls directly (since I dont have access). Or maybe you want to come up with a config slicer or something for this. Any other suggestions on how I can correct this now since your initial release was very poorly configured and tested? Nalin Makar
added a comment - original issue reported here is fixed. Editing config xml is the way to go (sorry for the trouble).
Nalin Makar
added a comment - original issue reported here is fixed. Editing config xml is the way to go (sorry for the trouble). 
I lost some steam after the back and forth in January. I'll pick it up again next week and send out a pull request.