Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-26000

Invalid processing of suite-files in testng

    XMLWordPrintable

Details

    Description

      There is a problem while retrieving results for testng when the testng XML file contains the constructions <suite-file path="./suite1.xml"/> with multiple suites.
      Here are the details.

      At first, this is the example that works OK:
      <suite name="TAM TestSuite">
      <test name="MRCT1100"><classes><class name="com.ingenico.tam.testsuite.merchants.MRCT1101"></class></classes></test>
      <test name="SCNWZD1100"><classes><class name="com.ingenico.tam.testsuite.scenariowizard.SCNWZD1100"></class></classes></test>
      </suite>
      Here there are two tests, the first one is successful, the second one fails. In this format everything is OK, see the screenshot jenkins-ok.png.
      I attach also the file testng-results.ok for this case.

      Next, I do the same tests java with the same confuguration of testlink and plugin. I replace only the testng XML file by three files (attached): All.xml, TAMMerchants.xml, TAMScenarioWizard.xml.
      The tests are the same, but they are organized in separate files:
      <suite name="TAM TestSuite" parallel="false" thread-count="1" preserve-order="true">
      <suite-files>
      <suite-file path="./TAMMerchants.xml"/>
      <suite-file path="./TAMScenarioWizard.xml"/>
      </suite-files>
      </suite>
      In this case both tests are executed by testng, but the problem appears when the plugin retrieves the result, see the screenshot jenkins-ko.PNG attached.
      The test from TAMScenarioWizard.xml is Not Run.
      However it was executed, see the file testng-results.ko attached.

      In TestLink also:

      • for the first case the test MRCT1100 is successful, and SCNWZD1100 failes
      • in the second case the test MRCT1100 is successful, and SCNWZD1100 Not Run

      It seems that there is a problem when processing testng-results.xml: in first case it contains only one suite ,and it works, and in the seconde case there are three suite tags, and only one result is retrieved.

      I join also the file config.xml

      Attachments

        1. All.xml
          0.3 kB
        2. config.xml
          4 kB
        3. jenkins-ko.PNG
          jenkins-ko.PNG
          25 kB
        4. jenkins-ok.PNG
          jenkins-ok.PNG
          24 kB
        5. ko62.xml
          18 kB
        6. ok63.xml
          21 kB
        7. TAMMerchants.xml
          0.8 kB
        8. TAMScenarioWizard.xml
          1.0 kB
        9. testng-results.ko
          3 kB
        10. testng-results.ok
          4 kB

        Activity

          ab_ing A B added a comment - - edited

          The file ko62.xml is the XML RPC exchange between the plugin and testlink for build 62 where the second test is not run

          ab_ing A B added a comment - - edited The file ko62.xml is the XML RPC exchange between the plugin and testlink for build 62 where the second test is not run
          ab_ing A B added a comment -

          The file ok63.xml is the XML RPC exchange between the plugin and testlink for build 63 where the second test is present

          ab_ing A B added a comment - The file ok63.xml is the XML RPC exchange between the plugin and testlink for build 63 where the second test is present
          ab_ing A B added a comment -

          A added two more files: ko62.xml and ok63.xml containing the XML RPC messages exchanged between the plugin and testlink.
          The main difference is at the end:

          • ko62.xml: the plugin sends back only one test case result
          • ok63.xml: the plugin sends back two test case results

          the builds 62 and 63 use the same set of test cases, but for 62 they are structured into suites, and in 63 they are in the same suite.

          ab_ing A B added a comment - A added two more files: ko62.xml and ok63.xml containing the XML RPC messages exchanged between the plugin and testlink. The main difference is at the end: ko62.xml: the plugin sends back only one test case result ok63.xml: the plugin sends back two test case results the builds 62 and 63 use the same set of test cases, but for 62 they are structured into suites, and in 63 they are in the same suite.

          Hello.

          I have exactly the same problem described below
          Jenkins Version : 1.613
          Plugin Version : 3.10

          A correction time is it planned?

          neril aurelien dubois added a comment - Hello. I have exactly the same problem described below Jenkins Version : 1.613 Plugin Version : 3.10 A correction time is it planned?

          People

            kinow Bruno P. Kinoshita
            ab_ing A B
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: