• Icon: New Feature New Feature
    • Resolution: Unresolved
    • Icon: Minor Minor
    • xunit-plugin
    • None

      When a CTest fails on a segfault, xUnit reports this as a failure, but in the Stack Trace, doesn't show that it failed because of a segfault. GTest shows that it PASSED, which our logs show that as well, but behind the scenes, xUnit must know that it segfaulted as well to put it into the Failed Test section. The exit code of the test shows that it is a SEGFAULT as well. Was hoping to have some sort of indication that a test failed because of a SEGFAULT displayed in the xUnit failed test results.

          [JENKINS-66695] SEGFAULT on CTest should be reported

          Nikolas Falco added a comment -

          I need a complete xml test report (produced by CTest) so I can look into and prepare an appropriate xlst to generate a failure case

          Nikolas Falco added a comment - I need a complete xml test report (produced by CTest) so I can look into and prepare an appropriate xlst to generate a failure case

          David added a comment -

          I've attached an XML that contains only the test that I am talking about. Two tests in my repo show this same behavior though, with possible others throughout my project.

          David added a comment - I've attached an XML that contains only the test that I am talking about. Two tests in my repo show this same behavior though, with possible others throughout my project.

          David added a comment - - edited

          Is there any status on this. Do you need any more info than what I have provided?

          David added a comment - - edited Is there any status on this. Do you need any more info than what I have provided?

          Nikolas Falco added a comment -

          no movement on this. I have all informations thanks

          Nikolas Falco added a comment - no movement on this. I have all informations thanks

          Nikolas Falco added a comment - - edited

          The output you attach is about CTest I do not know why in the description you mention GTest. In xUnit there are GoogleTest and GTester which formats are completly different.
          Anyway to show the Exit Code value implies change actual format (before failure = system-out).

          <?xml version="1.0" encoding="UTF-8"?>
          <testsuites xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xunit="http://www.xunit.org">
             <testsuite name="CTest"
                        tests="1"
                        time="88.000"
                        failures="1"
                        errors="0"
                        skipped="0">
                <testcase classname="projectroot.tests" name="test1.realStdin" time="0.025">
                   <failure>SEGFAULT</failure>
                   <system-out><![CDATA[Running main() from /home/user/googletest-release-1.10.0/googletest/src/gtest_main.cc
          Note: Google Test filter = test1.realStdin
          [==========] Running 1 test from 1 test suite.
          [----------] Global test environment set-up.
          [----------] 1 test from StdinComponent
          [ RUN      ] test1.realStdin
          [       OK ] test1.realStdin (1 ms)
          [----------] 1 test from StdinComponent (1 ms total)
          
          [----------] Global test environment tear-down
          [==========] 1 test from 1 test suite ran. (1 ms total)
          [  PASSED  ] 1 test.
          ]]></system-out>
                </testcase>
             </testsuite>
          </testsuites>
          

          I do not know if report SEGFAULT as error instead of failure and if there are other possible values as Exit Code that will complicate the way to understand what is error and what failure

          Nikolas Falco added a comment - - edited The output you attach is about CTest I do not know why in the description you mention GTest. In xUnit there are GoogleTest and GTester which formats are completly different. Anyway to show the Exit Code value implies change actual format (before failure = system-out). <?xml version= "1.0" encoding= "UTF-8" ?> <testsuites xmlns:xs = "http://www.w3.org/2001/XMLSchema" xmlns:xunit = "http://www.xunit.org" > <testsuite name= "CTest" tests= "1" time= "88.000" failures= "1" errors= "0" skipped= "0" > <testcase classname= "projectroot.tests" name= "test1.realStdin" time= "0.025" > <failure> SEGFAULT </failure> <system-out> <![CDATA[Running main() from /home/user/googletest-release-1.10.0/googletest/src/gtest_main.cc Note: Google Test filter = test1.realStdin [==========] Running 1 test from 1 test suite. [----------] Global test environment set-up. [----------] 1 test from StdinComponent [ RUN ] test1.realStdin [ OK ] test1.realStdin (1 ms) [----------] 1 test from StdinComponent (1 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test suite ran. (1 ms total) [ PASSED ] 1 test. ]]> </system-out> </testcase> </testsuite> </testsuites> I do not know if report SEGFAULT as error instead of failure and if there are other possible values as Exit Code that will complicate the way to understand what is error and what failure

          David added a comment -

          So maybe what i sent didnt have enough clarity in it but in the end, XUnit in the reports shows that this test Fails but in the details of it, it says that it passes. I have blurred out sensitive information to show you what I mean. This is the reporting of the xUnit plugin for failed tests, but in the end, it says that the test has passed. I'm not 100% clear on what you mean by "change actual format." If you means, changing the HTML reporting format, I think that might be the path that should be taken since there's vital information that isn't being presented. 

          David added a comment - So maybe what i sent didnt have enough clarity in it but in the end, XUnit in the reports shows that this test Fails but in the details of it, it says that it passes. I have blurred out sensitive information to show you what I mean. This is the reporting of the xUnit plugin for failed tests, but in the end, it says that the test has passed. I'm not 100% clear on what you mean by "change actual format." If you means, changing the HTML reporting format, I think that might be the path that should be taken since there's vital information that isn't being presented. 

          Nikolas Falco added a comment -

          xUnit reported those tests as failed but you complained that in the "Stack Trace" and "Standard Output" show the same content (the output of the command line).
          Now with the changes in the branch the reports will show as failure (Stack Trace) SEGFAULT and as "Standard Output" the usual cli output, this means changes the XML produced.
          I can not change the output of the command line, so if the logs is

          ...
          [  PASSES  ] 1 test.
          

          will be keept as is. It makes no sense for me to interpolate the meaning of the logs to show the desired one. The log could vary with the version of the tool and have too many special cases, it would be a bloodbath. You are free to use a your custom XLST to archive this.

          Nikolas Falco added a comment - xUnit reported those tests as failed but you complained that in the "Stack Trace" and "Standard Output" show the same content (the output of the command line). Now with the changes in the branch the reports will show as failure (Stack Trace) SEGFAULT and as "Standard Output" the usual cli output, this means changes the XML produced. I can not change the output of the command line, so if the logs is ... [ PASSES ] 1 test. will be keept as is. It makes no sense for me to interpolate the meaning of the logs to show the desired one. The log could vary with the version of the tool and have too many special cases, it would be a bloodbath. You are free to use a your custom XLST to archive this.

            nfalco Nikolas Falco
            venn David
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: