Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47367

Parameterized test cases mixed with non-parameterized are listed wrong in test summary

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: nunit-plugin
    • Labels:
      None
    • Environment:
      nunit-plugin 0.21, Jenkins 2.83 on a Windows Server 2008 R2
    • Similar Issues:

      Description

      Using the latest nunit plugin on latest (not LTS) stable Jenkins leads to a wrong displayed test case summary. 

      In our example the non-parameterized test cases which are mixed in one test class with parameterized test cases are referenced to a wrong class. 

      In our example the parameterized test cases are referenced to the package "Base.Tests" and the non-parameterized test cases are referenced to the package "Base" which is wrong. They are in the package "Base.Tests". You can see this in the screenshots, every test which is displayed in "Base" is wrong and belongs to "Base.Tests"

      This is our test result output xml.

       <test-suite type="TestFixture" name="CryptUtilsTests" executed="True" result="Success" success="True" time="0.308" asserts="0">
      	<results>
      	  <test-suite type="ParameterizedTest" name="ComputerSha256HashAndCompareWithExpectedResult" executed="True" result="Success" success="True" time="0.010" asserts="0">
      		<results>
      		  <test-case name="Base.Tests.CryptUtilsTests.ComputerSha256HashAndCompareWithExpectedResult(&quot;pass&quot;,&quot;D74FF0BDE50BE8F1&quot;)" executed="True" result="Success" success="True" time="0.005" asserts="1" />
      		  <test-case name="Base.Tests.CryptUtilsTests.ComputerSha256HashAndCompareWithExpectedResult(&quot;Horst&quot;,&quot;1FC1E73359247428B9855DFD&quot;)" executed="True" result="Success" success="True" time="0.000" asserts="1" />
      		  <test-case name="Base.Tests.CryptUtilsTests.ComputerSha256HashAndCompareWithExpectedResult(&quot;Hugo&quot;,&quot;D70B489BB97D74C8E69A60EA62DC8FFD&quot;)" executed="True" result="Success" success="True" time="0.001" asserts="1" />
      		  <test-case name="Base.Tests.CryptUtilsTests.ComputerSha256HashAndCompareWithExpectedResult(&quot;GeheimesPasswort!&quot;,&quot;2600F32BA82835F62D46B7&quot;)" executed="True" result="Success" success="True" time="0.000" asserts="1" />
      		</results>
      	  </test-suite>
      	  <test-case name="Base.Tests.CryptUtilsTests.ComputesSaltAndPasswordHashCorrectly" executed="True" result="Success" success="True" time="0.024" asserts="4" />
      	  <test-case name="Base.Tests.CryptUtilsTests.ComputesSaltAndPasswordHashFromDifferentTasksCorrectly" executed="True" result="Success" success="True" time="0.135" asserts="1" />
      	  <test-case name="Base.Tests.CryptUtilsTests.ComputesSaltAndPasswordHashFromDifferentThreadsCorrectly" executed="True" result="Success" success="True" time="0.095" asserts="1" />
      	  <test-case name="Base.Tests.CryptUtilsTests.EncyrptAndDecryptWindows" executed="True" result="Success" success="True" time="0.034" asserts="15" />
      	</results>
       </test-suite>

        Attachments

          Activity

          Hide
          slide_o_mix Alex Earl added a comment -

          Did this behave differently with a different version of Jenkins, or are you just saying it doesn't match things up in general?

          Show
          slide_o_mix Alex Earl added a comment - Did this behave differently with a different version of Jenkins, or are you just saying it doesn't match things up in general?
          Hide
          ise Daniel Isenmann added a comment -

          Can't test it with different version because it's our production server. We just discovered it with the version I have mentioned.

          Show
          ise Daniel Isenmann added a comment - Can't test it with different version because it's our production server. We just discovered it with the version I have mentioned.
          Hide
          slide_o_mix Alex Earl added a comment -

          That isn't even valid XML though (what you pasted). There are quoted strings inside of attributes. When I try and run this through the plugin locally, it just dies on trying to process the XML itself.

          Show
          slide_o_mix Alex Earl added a comment - That isn't even valid XML though (what you pasted). There are quoted strings inside of attributes. When I try and run this through the plugin locally, it just dies on trying to process the XML itself.
          Hide
          ise Daniel Isenmann added a comment -

          Changed the XML accordingly. Seems that the quotes were insert during copy, they are now masked correctly. 

          Show
          ise Daniel Isenmann added a comment - Changed the XML accordingly. Seems that the quotes were insert during copy, they are now masked correctly. 
          Hide
          slide_o_mix Alex Earl added a comment -

          This doesn't look like valid nunit3 output, is it nunit3 or nunit2? The <results> nodes are not part of the NUnit3 XML specifications. https://github.com/nunit/docs/wiki/Test-Result-XML-Format

          Show
          slide_o_mix Alex Earl added a comment - This doesn't look like valid nunit3 output, is it nunit3 or nunit2? The <results> nodes are not part of the NUnit3 XML specifications.  https://github.com/nunit/docs/wiki/Test-Result-XML-Format
          Hide
          ise Daniel Isenmann added a comment -

          We are running nunit2

          Show
          ise Daniel Isenmann added a comment - We are running nunit2
          Hide
          slide_o_mix Alex Earl added a comment -

          Is that really the full output xml from nunit? With nunit2, there should be a <test-results> node at the outside.

          Show
          slide_o_mix Alex Earl added a comment - Is that really the full output xml from nunit? With nunit2, there should be a <test-results> node at the outside.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            ise Daniel Isenmann
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated: