Details
-
Type:
Bug
-
Status: Resolved (View Workflow)
-
Priority:
Major
-
Resolution: Fixed
-
Component/s: xunit-plugin
-
Labels:None
-
Environment:xunit-plugin 2.2.3
nunit3-console.exe 3.7.0
-
Similar Issues:
Description
We are running `nunit3-console -result:TestResult.xml;format=nunit2` to produce NUnit2 compatible test reports with the NUnit3 runner.
With the earlier xunit-plugin 1.102 version this worked just fine. Now with xunit-plugin 2.2.3, if we have errors / failures in the test results, these are not reported as failures anymore. Even worse, they are reported as success!
TestResults.xml report file says:
Jenkins / xunit-plugin 2.2.3 says:
See the full example XML report attached
Attachments
Activity
Torben Knerr
created issue -
Torben Knerr
made changes -
| Field | Original Value | New Value |
|---|---|---|
| Attachment | Nunit3-in-nunit2-format-with-failures-TestReport.xml [ 43819 ] |
Torben Knerr
made changes -
| Attachment | Nunit3-in-nunit2-format-with-failures-TestReport.xml [ 43820 ] |
Torben Knerr
made changes -
| Attachment | screenshot-1.png [ 43821 ] |
Torben Knerr
made changes -
| Attachment | screenshot-2.png [ 43822 ] |
Torben Knerr
made changes -
| Description |
We are running `nunit3-console -result:TestResult.xml;format=nunit2` to produce NUnit2 compatible test reports with the NUnit3 runner. With the earlier xunit-plugin 1.102 version this worked just fine. Now with xunit-plugin 2.2.3, if we have errors / failures in the test results, these are not reported anymore. See the example XML report attached |
We are running `nunit3-console -result:TestResult.xml;format=nunit2` to produce NUnit2 compatible test reports with the NUnit3 runner. With the earlier xunit-plugin 1.102 version this worked just fine. Now with xunit-plugin 2.2.3, if we have errors / failures in the test results, these are not reported as failures anymore. Even worse, they are reported as success! TestResults.xml report file says: !screenshot-2.png|thumbnail! Jenkins / xunit-plugin 2.2.3 says: !screenshot-1.png|thumbnail! See the full example XML report attached |
Torben Knerr
made changes -
| Summary | Test Failures not reported when using nunit3-console with nunit2 output formatter | Test Failures reported as Success when using nunit3-console with nunit2 output formatter |
Torben Knerr
made changes -
| Issue Type | Improvement [ 4 ] | Bug [ 1 ] |
Torben Knerr
made changes -
| Priority | Minor [ 4 ] | Major [ 3 ] |
If you produce the NUnit3 why do you convert to NUni2 instead of configure the tool type to NUnit 3.x?
Anyway let me have a look if result Error is a valid kind in NUnit2 and in case handle properly or it is a status in NUnit3 that is handled in wrong manner in the converter.
I suggest if you have NUni3 to do not convert and use the tool type set to NUnit 3.x
"my first impression was that the "nunit-version" field (which is set to "3.5.0.0" in the attached example) might get interpreted"
This is a bug of adapter https://github.com/nunit/nunit-v2-result-writer/issues/11
| Summary | Test Failures reported as Success when using nunit3-console with nunit2 output formatter | NUnit 2.x test cases with result Error are reported as Success |
| Status | Open [ 1 ] | In Progress [ 3 ] |
Nikolas Falco the reason we are converting to nunit2 format is actually that the previous version of xunit-plugin we used (1.102) did not support nunit3 yet. So while we are using nunit2 format in other places, we could indeed try to use nunit3 format here to work around that issue with xunit-plugin 2.2.3.
Regarding the nunit-version attribute, I suspected that the "3.5.0.0" version was the culprit. However, I changed that to "2.5.0.0" before the report publishing and still got the same misbehaviour. So I concluded that even though this is a bug in nunit-v2-result-writer (thanks for digging out the link to the github issue!) it would not resolve the issue reported here.
Torben Knerr
added a comment - - edited Nikolas Falco the reason we are converting to nunit2 format is actually that the previous version of xunit-plugin we used (1.102) did not support nunit3 yet. So while we are using nunit2 format in other places, we could indeed try to use nunit3 format here to work around that issue with xunit-plugin 2.2.3.
Regarding the nunit-version attribute, I suspected that the "3.5.0.0" version was the culprit. However, I changed that to "2.5.0.0" before the report publishing and still got the same misbehaviour. So I concluded that even though this is a bug in nunit-v2-result-writer (thanks for digging out the link to the github issue!) it would not resolve the issue reported here. | Resolution | Fixed [ 1 ] | |
| Status | In Progress [ 3 ] | Fixed but Unreleased [ 10203 ] |
Thanks Nikolas Falco, just saw you already fixed it in master! 
Any plans for pushing out a 2.2.4 release with that included anytime soon?
Torben Knerr
added a comment - Thanks Nikolas Falco , just saw you already fixed it in master!
Any plans for pushing out a 2.2.4 release with that included anytime soon? Hi Nikolas Falco, any news yet about a potential 2.2.4 release or expected timeline?
Cheers, Torben
Torben Knerr
added a comment - Hi Nikolas Falco , any news yet about a potential 2.2.4 release or expected timeline?
Cheers, Torben released in 2.2.4
| Status | Fixed but Unreleased [ 10203 ] | Resolved [ 5 ] |
Nikolas Falco my first impression was that the "nunit-version" field (which is set to "3.5.0.0" in the attached example) might get interpreted. However, I manually changed this to "2.5.0.0" but still the same issue, failures were not reported correctly
Note: we use "NUnitJunitHudsonTestType" assuming this will handle the NUnit2 format correctly (which it did in the xunit-plugin 1.102 version at least)