This would be pretty useful for our company. If nobody picks it up, would it help if we took a stab?

Yep, it would be good

I took a look at this. If you're using Maven (as I am), then Maven will call the build UNSTABLE before the TestNG plugin is invoked. If we try to set the build back to successful after this happens, we cannot. This is by design in Jenkins. The build can only go from successful->unstable->failed and not in the other direction, as far as I can tell.

We can still add a field in the plugin that fails the build if the pass/fail ratio dips below a certain percentage, but there are some cases where we can't go back to the success state if it has already been set to something worse than success.

Does anyone have any thoughts on this? I can continue with the pull request, but I don't want to move forward if I'm going to wind up discarding it because this isn't the ideal behavior.

Unfortunatelly, I'm using Ant and I don't have such problem and can't reproduce it. In my case, after running of the tests, build marked as successfully. I receive "Unstable" or "Failed" result after publishing of the TestNG results.

Depending on how you configure your ant build, you may or may not run into the problem I'm describing above. Any change you can copy/paste the snippet of your ant build.xml that invokes your TestNG tests?

In your case, if I add a threshold field, you'll likely see a SUCCESS so long as your ant run does not terminate abnormally when it runs (i.e. - the ant exit code is non-zero). If it does, then you'll see a FAILURE. If you configure your ant build to fail when tests fails, then you'll always see a FAILURE (red) regardless of the new field that I'm adding. This is probably not the case for you since you said you get UNSTABLE (yellow) sometimes.

All that said, I'm happy to add the field, knowing that whether or not it functions as advertised is very specific to the build tool you're using to execute TestNG tests.

I'd like to know if Nalin has any opinion on this as the maintainer of the repo. I don't want to spend the time if the pull request won't be merged back into the mainline.

Nalin, can you comment on the discussion Nikolay and I have been having. I'll gladly finish the implementation, but I wan't know if there's any reason it doesn't make sense to have such a feature in the plugin. I don't want to spend the time if the pull request doesn't make sense in the first place from your point of view. Thanks!

I have a working implementation. See attached screenshot.

Just need to add some validation and I'll submit a pull request.

I am not really sold on this feature. It's not right to mark a build as successful based on a percentage as it could mislead the user. E.g. If the threshold is set at 95% and if 96% tests pass, the build would be marked success, but the failing 4% tests could be the most important tests of the regression.

Do you have a concrete use case in mind? What is the issue with a build being marked unstable? IMO, implementing JENKINS-9838, in some ways similar to how Emma Coverage Report plugin works would be better.

In our case, we have several thousand functional tests and an engineering organization of roughly 150 people that contribute to them. That said, we ocasionally have a an intermittent failure or two that we'd like to deal with in isolation, but we don't want the build to fail for hours while the fix is on its way. We're okay with tolerating roughly 1% of failures, but really nothing more.

Obviously, this is an opt-in feature, so I'm not sure how it could do anything but benefit others who have a similar use case as us. Fundamentally I agree with you that we might not want to ignore a certain percentage of failures, but each organization is different, so shouldn't we let them make that decision?

This feature works similarly to the xUnit-plugin and we could use that instead, but it currently doesn't have TestNG support. We'd prefer to continue to use the TestNG plugin instead of extending xUnit to support TestNG.

Finally, I think that the health extension a la emma is good, but it's not as valuable for our purposes as success vs. unstable vs. failure.

Thanks for responding. Before I move forward, I'll let Nikolay share his experiences too (he originally filed the feature request).

I have several jobs with different number of skipped and failed tests on each of them. I need to have a possibility to set an acceptable number of the failed and skipped tests. When number of skipped/failed tests is more than threshold value then this job should be set to unstable/failed. This is my point of view. Also would be good to have different strategies e.g. to have a possibility to set number or percentage of the skipped/failed tests depending on requirements.

ok. Please submit a pull request. Lets make sure that the configuration is similar to that in xUnit plugin. Thanks.

Hello guys, are there any updates?

Any update to this?

During a company hackathon, a team of co-workers and myself worked on this ticket (as well as others). Here is the PR link: https://github.com/jenkinsci/testng-plugin-plugin/pull/20/files

Is there any update is available to mark build as Stable if some configuration test cases failed ?

https://github.com/jenkinsci/testng-plugin-plugin/commit/52a4637968ea1407ae09be257992e4c0ef4d3c41

merged pull #20.

Hey nalin_makar, I don't think the default threshold value should be 100%. We updated the plugin without noticing this feature, and this was causing tests that failed to show as stable.

cwcam, can you please file a separate bug? And, it'll be great if you can give specific information about the properties you are referring to. Thanks!

cwcam, nalin_makar, I don't think that is a bug. The default behavior before the update was to mark the build as UNSTABLE whenever there were test failures. The default failure threshold to FAIL the build is 100%, but the default failure threshold to mark the build as UNSTABLE is 0. The default behavior should not have changed with the update.

The default is fixed, but now I am stuck in a situation where all my jobs are set to 100 for failed tests to be considered unstable. Ungrading to 1.13 fixes the default for new jobs. But how about the rest of us that upgraded to 1.12 and ALL our jobs are stuck with 100% for this value? I guess I am stuck with having the server admin help me change all the job xmls directly (since I dont have access). Or maybe you want to come up with a config slicer or something for this. Any other suggestions on how I can correct this now since your initial release was very poorly configured and tested?

original issue reported here is fixed. Editing config xml is the way to go (sorry for the trouble).