• Type: Bug
• Resolution: Done
• Priority: Minor
• Component/s: xunit-plugin
• Labels:

  • junit
  • parse-error
  • test-results
  • xunit
• Environment:

  Jenkins version 2.179, xunit plugin version 2.3.1

1. junit_test_good.xml

   0.3 kB

   2019-07-06 17:43

2. junit_test_bad.xml

   0.5 kB

   2019-07-06 17:42

3. xunit_test_bad.xml

   0.6 kB

   2019-07-06 17:42

4. xunit_test_good.xml

   0.5 kB

   2019-07-06 17:42

[JENKINS-58369] xUnit Plugin problem with Boost test XML output that contains messages

Francis Duffy created issue - 2019-07-06 17:45

Francis Duffy made changes - 2019-07-06 17:46

Description

Original: I compile the following minimal C++ Boost test program   {code:java} #define BOOST_TEST_MODULE ExampleTestSuite #include <boost/test/included/unit_test.hpp> #include <boost/test/unit_test.hpp> BOOST_AUTO_TEST_SUITE(MyTestSuite_1) BOOST_AUTO_TEST_CASE(MyTest_1_1) {     // BOOST_TEST_MESSAGE("Running MyTest_1_1");     int i = 1;     BOOST_CHECK_EQUAL(1, i); } BOOST_AUTO_TEST_SUITE_END() BOOST_AUTO_TEST_SUITE(MyTestSuite_2) BOOST_AUTO_TEST_CASE(MyTest_2_1) {     int i = 1;     BOOST_CHECK_EQUAL(1, i); } BOOST_AUTO_TEST_SUITE_END() {code} and run the test using the parameters     {code:java} --log_format=XML --log_level=test_suite --report_level=no --result_code=no --log_sink=xunit_test.xml{code} The XML output is in the attached file {{xunit_test_good.xml}} and the xUnit plugin generated JUnit output is in the attached file {{junit_good.xml}}. Everything is as I would expect here and the table that is rendered on the Jenkins GUI has {{ExampleTestSuite}} at the top level with {{MyTestSuite_1}} and {{MyTestSuite_2}} nested below it. However, if I comment in the {{BOOST_TEST_MESSAGE}} line in the code, I get the message in the XML output as in the attached file {{xunit_test_bad.xml}}. This gets converted to the JUnit output in {{junit_test_bad.xml}} by the xUnit plugin. The nesting is now messed up in that {{ExampleTestSuite}} appears with a nested {{MyTestSuite_2}} but {{ExampleTestSuite.MyTestSuite_1}} appears alongside {{ExampleTestSuite}}. This is coming from the {{classname}} attribute in the generated JUnit output. Would it be possible to fix this so that test message output does not interfere with the generation of the {{classname}} attribute and hence the nesting of the test suites in the report? Alternatively, is there a workaround that I can apply to keep the log message but still have the expected nesting of the test suites?

New: I compile the following minimal C++ Boost test program  {code:java} #define BOOST_TEST_MODULE ExampleTestSuite #include <boost/test/included/unit_test.hpp> #include <boost/test/unit_test.hpp> BOOST_AUTO_TEST_SUITE(MyTestSuite_1) BOOST_AUTO_TEST_CASE(MyTest_1_1) {     // BOOST_TEST_MESSAGE("Running MyTest_1_1");     int i = 1;     BOOST_CHECK_EQUAL(1, i); } BOOST_AUTO_TEST_SUITE_END() BOOST_AUTO_TEST_SUITE(MyTestSuite_2) BOOST_AUTO_TEST_CASE(MyTest_2_1) {     int i = 1;     BOOST_CHECK_EQUAL(1, i); } BOOST_AUTO_TEST_SUITE_END() {code} and run the test using the parameters {code:java} --log_format=XML --log_level=test_suite --report_level=no --result_code=no --log_sink=xunit_test.xml{code} The XML output is in the attached file {{xunit_test_good.xml}} and the xUnit plugin generated JUnit output is in the attached file {{junit_good.xml}}. Everything is as I would expect here and the table that is rendered on the Jenkins GUI has {{ExampleTestSuite}} at the top level with {{MyTestSuite_1}} and {{MyTestSuite_2}} nested below it. However, if I comment in the {{BOOST_TEST_MESSAGE}} line in the code, I get the message in the XML output as in the attached file {{xunit_test_bad.xml}}. This gets converted to the JUnit output in {{junit_test_bad.xml}} by the xUnit plugin. The nesting is now messed up in that {{ExampleTestSuite}} appears with a nested {{MyTestSuite_2}} but {{ExampleTestSuite.MyTestSuite_1}} appears alongside {{ExampleTestSuite}}. This is coming from the {{classname}} attribute in the generated JUnit output. Would it be possible to fix this so that test message output does not interfere with the generation of the {{classname}} attribute and hence the nesting of the test suites in the report? Alternatively, is there a workaround that I can apply to keep the log message but still have the expected nesting of the test suites?

Collapse comment: Kim Randell added a comment - 2019-10-14 15:31

Kim Randell added a comment - 2019-10-14 15:31

This comes from this line:

https://github.com/jenkinsci/xunit-plugin/blob/5f017f9da41e987f3644c65f10658660519abfe5/src/main/resources/org/jenkinsci/plugins/xunit/types/boosttest-1.2-to-junit-4.xsl#L176

Note that $elt/@file is only populated if the TestCase element has a first child with a file attribute (line 172). Therefore, if there is a message or assertion failure in the test, classname is the suite name concatenated with the filename (minus extension), otherwise it is just the suite name. It's unclear to me why the filename should be used in the classname in any case.

 

Expand comment: Kim Randell added a comment - 2019-10-14 15:31

Kim Randell added a comment - 2019-10-14 15:31 This comes from this line: https://github.com/jenkinsci/xunit-plugin/blob/5f017f9da41e987f3644c65f10658660519abfe5/src/main/resources/org/jenkinsci/plugins/xunit/types/boosttest-1.2-to-junit-4.xsl#L176 Note that $elt/@file is only populated if the TestCase element has a first child with a file attribute (line 172). Therefore, if there is a message or assertion failure in the test, classname is the suite name concatenated with the filename (minus extension), otherwise it is just the suite name. It's unclear to me why the filename should be used in the classname in any case.  

Nikolas Falco made changes - 2022-10-04 12:59

Released As		New: 3.1.2
Resolution		New: Done [ 10000 ]
Status	Original: Open [ 1 ]	New: Resolved [ 5 ]