Details
-
Type:
New Feature
-
Status: Resolved (View Workflow)
-
Priority:
Major
-
Resolution: Fixed
-
Component/s: testng-plugin
-
Labels:None
-
Similar Issues:
Description
Current version of TestNG plugin does not have a possibility to use threshold values for failed and skipped tests. If some build has at least one failed test, then this build will be marked as unstable.
Maybe it would be better to have some mechanism to mark build as successfull instead of unstable with using a threshold value for failed tests.
Attachments
Issue Links
- is related to
-
JENKINS-34822 TestNG plugin has incorrect default threshold values
-
- Reopened
-
- relates to
-
-
- Open
-
-
-
- Resolved
-
Activity
Cam Spencer, can you please file a separate bug? And, it'll be great if you can give specific information about the properties you are referring to. Thanks!
Nalin Makar
added a comment - Cam Spencer , can you please file a separate bug? And, it'll be great if you can give specific information about the properties you are referring to. Thanks! Cam Spencer, nalin_makar, I don't think that is a bug. The default behavior before the update was to mark the build as UNSTABLE whenever there were test failures. The default failure threshold to FAIL the build is 100%, but the default failure threshold to mark the build as UNSTABLE is 0. The default behavior should not have changed with the update.
Chris Everling
added a comment - Cam Spencer , nalin_makar , I don't think that is a bug. The default behavior before the update was to mark the build as UNSTABLE whenever there were test failures. The default failure threshold to FAIL the build is 100%, but the default failure threshold to mark the build as UNSTABLE is 0. The default behavior should not have changed with the update. The default is fixed, but now I am stuck in a situation where all my jobs are set to 100 for failed tests to be considered unstable. Ungrading to 1.13 fixes the default for new jobs. But how about the rest of us that upgraded to 1.12 and ALL our jobs are stuck with 100% for this value? I guess I am stuck with having the server admin help me change all the job xmls directly (since I dont have access). Or maybe you want to come up with a config slicer or something for this. Any other suggestions on how I can correct this now since your initial release was very poorly configured and tested?
James Theisen
added a comment - The default is fixed, but now I am stuck in a situation where all my jobs are set to 100 for failed tests to be considered unstable. Ungrading to 1.13 fixes the default for new jobs. But how about the rest of us that upgraded to 1.12 and ALL our jobs are stuck with 100% for this value? I guess I am stuck with having the server admin help me change all the job xmls directly (since I dont have access). Or maybe you want to come up with a config slicer or something for this. Any other suggestions on how I can correct this now since your initial release was very poorly configured and tested? original issue reported here is fixed. Editing config xml is the way to go (sorry for the trouble).
Nalin Makar
added a comment - original issue reported here is fixed. Editing config xml is the way to go (sorry for the trouble). 
Hey nalin_makar, I don't think the default threshold value should be 100%. We updated the plugin without noticing this feature, and this was causing tests that failed to show as stable.