Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12417

xUnit problems with CppTest reports generated with default options

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • xunit-plugin
    • None
    • xUnit 1.32
      Jenkins 1.424.1
      C++test 7.3.2.97

      We are encountering some problems of xUnit when parsing Parasoft C++test results.

      • When parsing results output with C++test default options, xUnit fails when all tests succeed.
      • xUnit miscounts the number of failures and total, when multiple assertions fail in one test case.
      • When a certain C++test report option enabled, xUnit outputs "ERROR" to console log.

      To help you reproduce above problems, I attach a zip file,
      which includes some report.xml files output by C++test, (and also report.html)

      Below are expected results and obtained ones by xUnit,
      using each report.xml file in each folder of the zip as an input for xUnit.

      A) report_with_no_coverage_detail/

                               Expected   xUnit result
      a1) all_success_no_opt/    0/20      ERROR
      a2) all_success/           0/20      OK (0/20)
      a3) mix/                   1/20      OK (1/20)
      a4) 2assertion_failure/    1/20      NG (2/21)
                                (M/N means failures/total of tests)
      

      a1) is the case when all tests succeeded and report.xml was output with C++test default options.
      This case caused an xUnit error below:

      [xUnit] [INFO] - Starting to record.
      [xUnit] [INFO] - Processing CppTest-7.3 (default)
      [xUnit] [INFO] - [CppTest-7.3 (default)] - 1 test report file(s) were found with the pattern 'repout/report.xml' relative to '/home/user/slave01/workspace/test-cpptest-unit' for the testing framework 'CppTest-7.3 (default)'.
      [xUnit] [ERROR] - The plugin hasn't been performed correctly: None of the test reports contained any result
      Build step 'Publish testing tools result report' changed build result to FAILURE
      Build step 'Publish testing tools result report' marked build as failure
      Notifying upstream projects of job completion
      Finished: FAILURE
      

      When we used xUnit 1.34 insted of 1.32, xUnit 1.34 didn't cause an error,
      but the result was "no tests" though 20 tests existed.

      [xUnit] [WARNING] - All test reports are empty
      

      a2) is same to a1), but report.xml was output with C++test report option
      "Overview of checked files and executed tests" enabled.
      This case looks OK.
      (In this case, report.xml looks to have additional <ExecutedTestsDetails> elements)

      a3) is the case when one test failed.
      This looks OK.

      a4) is the case when one test failed, but two assertions failed in that one failed test.
      xUnit seems to wrongly increase the number of failures and total for each assertion failure in one failed test.

      B) report_with_coverage_detail/

      This folder contains the cases same to A),
      but report.xml files were output with C++test report option "Detailed report for Line coverage" enabled.

      Each report.xml file with this option looks to contain an additional <Files> element,
      as the last child of Exec/Coverage element.

      The results are same to A), but xUnit output ERROR message to console log:

      [xUnit] [INFO] - Starting to record.
      [xUnit] [INFO] - Processing CppTest-7.3 (default)
      [xUnit] [INFO] - [CppTest-7.3 (default)] - 1 test report file(s) were found with the pattern 'repout/report.xml' relative to '/home/baba/slave01/workspace/test-cpptest-unit' for the testing framework 'CppTest-7.3 (default)'.
      [xUnit] [ERROR] - The converted file for the result file '/home/user/slave01/workspace/test-cpptest-unit/repout/report.xml' (during conversion process for the metric 'CppTest') is not valid. The report file has been skipped.
      [xUnit] [INFO] - Check 'Failed Tests' threshold.
      [xUnit] [INFO] - Check 'Skipped Tests' threshold.
      [xUnit] [INFO] - Setting the build status to SUCCESS
      [xUnit] [INFO] - Stopping recording.
      

      xUnit seems to have failed to parse report.xml, but continue processing regardless of "skipped" and "ERROR".


      If you need some more information, please let me know.
      Thanks in advance.

            nfalco Nikolas Falco
            yamabb Yama Tksh
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: