-
New Feature
-
Resolution: Fixed
-
Major
-
None
Current version of TestNG plugin does not have a possibility to use threshold values for failed and skipped tests. If some build has at least one failed test, then this build will be marked as unstable.
Maybe it would be better to have some mechanism to mark build as successfull instead of unstable with using a threshold value for failed tests.
- is related to
-
JENKINS-34822 TestNG plugin has incorrect default threshold values
-
- Open
-
- relates to
-
-
- Open
-
-
-
- Closed
-
[JENKINS-20985] TestNG plugin should has a possibility to use thresholds for failed tests
Nikolay Driuchatyi
created issue -
Nikolay Driuchatyi
made changes -
| Description |
Original:
Current version of TestNG plugin does not have a possibility to use thresholds values for failed and skipped tests. If some build has at least one skipped or failed test then this build will be marked as unstable. I suppose that would be good to have some mechanism to leave build as successfully instead of unstable with using of threshold values for skipped/failed tests. |
New:
Current version of TestNG plugin does not have a possibility to use threshold values for failed and skipped tests. If some build has at least one failed test, then this build will be marked as unstable. Maybe it would be better to have some mechanism to mark build as successfull instead of unstable with using a threshold value for failed tests. |
Nikolay Driuchatyi
made changes -
| Summary | Original: TestNG plugin should has a possibility to use thresholds for failed/skipped tests | New: TestNG plugin should has a possibility to use thresholds for failed tests |
Nikolay Driuchatyi
added a comment - Yep, it would be good 
Gavin Gray
made changes -
| Assignee | Original: Nalin Makar [ nullin ] | New: Gavin Gray [ gavingray ] |
Gavin Gray
made changes -
| Status | Original: Open [ 1 ] | New: In Progress [ 3 ] |
Gavin Gray
added a comment - I took a look at this. If you're using Maven (as I am), then Maven will call the build UNSTABLE before the TestNG plugin is invoked. If we try to set the build back to successful after this happens, we cannot. This is by design in Jenkins. The build can only go from successful->unstable->failed and not in the other direction, as far as I can tell.
We can still add a field in the plugin that fails the build if the pass/fail ratio dips below a certain percentage, but there are some cases where we can't go back to the success state if it has already been set to something worse than success.
Does anyone have any thoughts on this? I can continue with the pull request, but I don't want to move forward if I'm going to wind up discarding it because this isn't the ideal behavior.
Gavin Gray
added a comment - I took a look at this. If you're using Maven (as I am), then Maven will call the build UNSTABLE before the TestNG plugin is invoked. If we try to set the build back to successful after this happens, we cannot. This is by design in Jenkins. The build can only go from successful->unstable->failed and not in the other direction, as far as I can tell.
We can still add a field in the plugin that fails the build if the pass/fail ratio dips below a certain percentage, but there are some cases where we can't go back to the success state if it has already been set to something worse than success.
Does anyone have any thoughts on this? I can continue with the pull request, but I don't want to move forward if I'm going to wind up discarding it because this isn't the ideal behavior. Nikolay Driuchatyi
added a comment - Unfortunatelly, I'm using Ant and I don't have such problem and can't reproduce it. In my case, after running of the tests, build marked as successfully. I receive "Unstable" or "Failed" result after publishing of the TestNG results.
Nikolay Driuchatyi
added a comment - Unfortunatelly, I'm using Ant and I don't have such problem and can't reproduce it. In my case, after running of the tests, build marked as successfully. I receive "Unstable" or "Failed" result after publishing of the TestNG results. Gavin Gray
added a comment - Depending on how you configure your ant build, you may or may not run into the problem I'm describing above. Any change you can copy/paste the snippet of your ant build.xml that invokes your TestNG tests?
In your case, if I add a threshold field, you'll likely see a SUCCESS so long as your ant run does not terminate abnormally when it runs (i.e. - the ant exit code is non-zero). If it does, then you'll see a FAILURE. If you configure your ant build to fail when tests fails, then you'll always see a FAILURE (red) regardless of the new field that I'm adding. This is probably not the case for you since you said you get UNSTABLE (yellow) sometimes.
All that said, I'm happy to add the field, knowing that whether or not it functions as advertised is very specific to the build tool you're using to execute TestNG tests.
I'd like to know if Nalin has any opinion on this as the maintainer of the repo. I don't want to spend the time if the pull request won't be merged back into the mainline.
Gavin Gray
added a comment - Depending on how you configure your ant build, you may or may not run into the problem I'm describing above. Any change you can copy/paste the snippet of your ant build.xml that invokes your TestNG tests?
In your case, if I add a threshold field, you'll likely see a SUCCESS so long as your ant run does not terminate abnormally when it runs (i.e. - the ant exit code is non-zero). If it does, then you'll see a FAILURE. If you configure your ant build to fail when tests fails, then you'll always see a FAILURE (red) regardless of the new field that I'm adding. This is probably not the case for you since you said you get UNSTABLE (yellow) sometimes.
All that said, I'm happy to add the field, knowing that whether or not it functions as advertised is very specific to the build tool you're using to execute TestNG tests.
I'd like to know if Nalin has any opinion on this as the maintainer of the repo. I don't want to spend the time if the pull request won't be merged back into the mainline. 
This would be pretty useful for our company. If nobody picks it up, would it help if we took a stab?