• Icon: Improvement Improvement
    • Resolution: Fixed
    • Icon: Major Major
    • junit-plugin
    • None

      JUnit report XML can contain a message for skipped tests, which often contains information regarding why the test is being skipped. It would be very useful if this message could be output in the test results in a similar way to how the failure are shown.

      I've attached an example JUnit style report with a skipped test.

          [JENKINS-8713] Display skipped test messages in results

          Gregory Nofi added a comment - - edited

          The issue type is Improvement but I think this might be a bug because this used to work quite well.

          Any skipped tests were listed as such in the Test Report, and the number of skipped tests was represented by a nice yellow color in the Test Result Trend chart. Even now the Test Result still has a column for number of Skipped tests, but it is always set to 0 in each row even when there are skipped tests in the JUnit report.

          One theory: this may have started happening to our team after the Performance Plugin was installed, which I think has its own JUnit parser. However, we have this problem even in jobs that aren't using the Performance Plugin.

          Update: Skipped tests started showing up again in the Test Result Trend Chart for us again. Please disregard my comment, which should have been logged as a separate, new bug report instead.

          Gregory Nofi added a comment - - edited The issue type is Improvement but I think this might be a bug because this used to work quite well. Any skipped tests were listed as such in the Test Report, and the number of skipped tests was represented by a nice yellow color in the Test Result Trend chart. Even now the Test Result still has a column for number of Skipped tests, but it is always set to 0 in each row even when there are skipped tests in the JUnit report. One theory: this may have started happening to our team after the Performance Plugin was installed, which I think has its own JUnit parser. However, we have this problem even in jobs that aren't using the Performance Plugin. Update: Skipped tests started showing up again in the Test Result Trend Chart for us again. Please disregard my comment, which should have been logged as a separate, new bug report instead.

          redsolo added a comment -

          Hi gregory, this issue should be marked as an improvement as it describes a feature that does not yet exist. Your problem sounds like a bug in an existing feature, so please create a new bug-report describing the problem and what plugins you are using (and version). Plus if you can, add a sample report that contains skipped tests to make easy to reproduce the issue.

          redsolo added a comment - Hi gregory, this issue should be marked as an improvement as it describes a feature that does not yet exist. Your problem sounds like a bug in an existing feature, so please create a new bug-report describing the problem and what plugins you are using (and version). Plus if you can, add a sample report that contains skipped tests to make easy to reproduce the issue.

          Ignas Vyšnu added a comment -

          Bumping this and would like to clear some confusion: skipped tests are counted and the total number is displayed, I think there is no "bug" here, but the feature improvement would be to actually list/display (or at least have an option to do so) the skipped tests the same way failed tests are being listed under Test Result view.

          Ignas Vyšnu added a comment - Bumping this and would like to clear some confusion: skipped tests are counted and the total number is displayed, I think there is no "bug" here, but the feature improvement would be to actually list/display (or at least have an option to do so) the skipped tests the same way failed tests are being listed under Test Result view.

          Code changed in jenkins
          User: Michael Clarke
          Path:
          core/src/main/java/hudson/tasks/junit/CaseResult.java
          core/src/main/resources/hudson/tasks/junit/CaseResult/index.jelly
          http://jenkins-ci.org/commit/jenkins/3b58cd294f8266f2e4dc80356bf172c37769030e
          Log:
          [FIXED JENKINS-8713] Display test skipped messages in test results

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Michael Clarke Path: core/src/main/java/hudson/tasks/junit/CaseResult.java core/src/main/resources/hudson/tasks/junit/CaseResult/index.jelly http://jenkins-ci.org/commit/jenkins/3b58cd294f8266f2e4dc80356bf172c37769030e Log: [FIXED JENKINS-8713] Display test skipped messages in test results

          Code changed in jenkins
          User: Michael Clarke
          Path:
          changelog.html
          http://jenkins-ci.org/commit/jenkins/2aa0293858d50b064128a9a74f4d25d3f5c4a5c8
          Log:
          JENKINS-8713 Add changelog entry

          Compare: https://github.com/jenkinsci/jenkins/compare/fe9f67602877...2aa0293858d5


          You received this message because you are subscribed to the Google Groups "Jenkins Commits" group.
          To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com.
          For more options, visit https://groups.google.com/groups/opt_out.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Michael Clarke Path: changelog.html http://jenkins-ci.org/commit/jenkins/2aa0293858d50b064128a9a74f4d25d3f5c4a5c8 Log: JENKINS-8713 Add changelog entry Compare: https://github.com/jenkinsci/jenkins/compare/fe9f67602877...2aa0293858d5 – You received this message because you are subscribed to the Google Groups "Jenkins Commits" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out .

          dogfood added a comment -

          Integrated in jenkins_main_trunk #2369
          [FIXED JENKINS-8713] Display test skipped messages in test results (Revision 3b58cd294f8266f2e4dc80356bf172c37769030e)
          JENKINS-8713 Add changelog entry (Revision 2aa0293858d50b064128a9a74f4d25d3f5c4a5c8)

          Result = UNSTABLE
          michael.m.clarke : 3b58cd294f8266f2e4dc80356bf172c37769030e
          Files :

          • core/src/main/java/hudson/tasks/junit/CaseResult.java
          • core/src/main/resources/hudson/tasks/junit/CaseResult/index.jelly

          michael.m.clarke : 2aa0293858d50b064128a9a74f4d25d3f5c4a5c8
          Files :

          • changelog.html

          dogfood added a comment - Integrated in jenkins_main_trunk #2369 [FIXED JENKINS-8713] Display test skipped messages in test results (Revision 3b58cd294f8266f2e4dc80356bf172c37769030e) JENKINS-8713 Add changelog entry (Revision 2aa0293858d50b064128a9a74f4d25d3f5c4a5c8) Result = UNSTABLE michael.m.clarke : 3b58cd294f8266f2e4dc80356bf172c37769030e Files : core/src/main/java/hudson/tasks/junit/CaseResult.java core/src/main/resources/hudson/tasks/junit/CaseResult/index.jelly michael.m.clarke : 2aa0293858d50b064128a9a74f4d25d3f5c4a5c8 Files : changelog.html

          sahumada added a comment -

          Hi,

          I've been trying to show my skipped tests without much success. My XML looks like:

          <?xml version="1.0" encoding="UTF-8"?>
          <testsuite name="tst_XmlPatterns" tests="2" failures="0" errors="0" skipped="1" time="0.000">
             <properties>
                <property name="QtVersion" value="5.2.2"/>
                <property name="QTestVersion" value="5.2.2"/>
             </properties>
             <testcase classname="tst_XmlPatterns" name="initTestCase" time="0.000">
                <skipped>tst_xmlpatterns.cpp:120 :: [] Device &#x9;IP:169.254.0.13&#xA;The command line tool (/usr/lib/qt5/bin/xmlpatterns) could not be run, possibly because Qt was incompletely built or installed. No tests will be run.</skipped>
             </testcase>
             <testcase classname="tst_XmlPatterns" name="cleanupTestCase" time="0.000"/>
             <system-out/>
             <system-err/>
          </testsuite>
          

          but it doesn't show anything.

          Looking at the source code, I see that you added an attribute called "message", so if I change my XML to

          <?xml version="1.0" encoding="UTF-8"?>
          <testsuite name="tst_XmlPatterns" tests="2" failures="0" errors="0" skipped="1" time="0.000">
             <properties>
                <property name="QtVersion" value="5.2.2"/>
                <property name="QTestVersion" value="5.2.2"/>
             </properties>
             <testcase classname="tst_XmlPatterns" name="initTestCase" time="0.000">
                <skipped message="tst_xmlpatterns.cpp:120 :: [] Device &#x9;IP:169.254.0.13&#xA;The command line tool (/usr/lib/qt5/bin/xmlpatterns) could not be run, possibly because Qt was incompletely built or installed. No tests will be run."/>
             </testcase>
             <testcase classname="tst_XmlPatterns" name="cleanupTestCase" time="0.000"/>
             <system-out/>
             <system-err/>
          </testsuite>
          

          then it fails because the XML is not valid according to junit-4.xsd for the xUnit plugin I am using.

          The error from the xUnit plugin I get is:

          13:03:18 [xUnit] [INFO] - Starting to record.
          13:03:18 [xUnit] [INFO] - Processing Custom Tool
          13:03:18 [xUnit] [INFO] - [Custom Tool] - 29 test report file(s) were found with the pattern 'test-results/tst_*.xml' relative to '/var/lib/jenkins/jobs/qt-5-qtxmlpatterns-dev/workspace/ARCH/ARM' for the testing framework 'Custom Tool'.
          13:03:18 [xUnit] [ERROR] - The converted file for the result file '/var/lib/jenkins/jobs/qt-5-qtxmlpatterns-dev/workspace/ARCH/ARM/test-results/tst_xmlpatterns.xml' (during conversion process for the metric 'Custom Tool') is not valid. The report file has been skipped.
          13:03:18 [xUnit] [INFO] - Failing BUILD because 'set build failed if errors' option is activated.
          13:03:18 [xUnit] [INFO] - There are errors when processing test results.
          13:03:18 [xUnit] [INFO] - Skipping tests recording.
          13:03:18 [xUnit] [INFO] - Stop build.
          

          So, is this change incorrect/incomplete or does junit-4.xsd/xUnit need to be updated ?

          sahumada added a comment - Hi, I've been trying to show my skipped tests without much success. My XML looks like: <?xml version="1.0" encoding="UTF-8"?> <testsuite name="tst_XmlPatterns" tests="2" failures="0" errors="0" skipped="1" time="0.000"> <properties> <property name="QtVersion" value="5.2.2"/> <property name="QTestVersion" value="5.2.2"/> </properties> <testcase classname="tst_XmlPatterns" name="initTestCase" time="0.000"> <skipped>tst_xmlpatterns.cpp:120 :: [] Device &#x9;IP:169.254.0.13&#xA;The command line tool (/usr/lib/qt5/bin/xmlpatterns) could not be run, possibly because Qt was incompletely built or installed. No tests will be run.</skipped> </testcase> <testcase classname="tst_XmlPatterns" name="cleanupTestCase" time="0.000"/> <system-out/> <system-err/> </testsuite> but it doesn't show anything. Looking at the source code, I see that you added an attribute called "message", so if I change my XML to <?xml version="1.0" encoding="UTF-8"?> <testsuite name="tst_XmlPatterns" tests="2" failures="0" errors="0" skipped="1" time="0.000"> <properties> <property name="QtVersion" value="5.2.2"/> <property name="QTestVersion" value="5.2.2"/> </properties> <testcase classname="tst_XmlPatterns" name="initTestCase" time="0.000"> <skipped message="tst_xmlpatterns.cpp:120 :: [] Device &#x9;IP:169.254.0.13&#xA;The command line tool (/usr/lib/qt5/bin/xmlpatterns) could not be run, possibly because Qt was incompletely built or installed. No tests will be run."/> </testcase> <testcase classname="tst_XmlPatterns" name="cleanupTestCase" time="0.000"/> <system-out/> <system-err/> </testsuite> then it fails because the XML is not valid according to junit-4.xsd for the xUnit plugin I am using. The error from the xUnit plugin I get is: 13:03:18 [xUnit] [INFO] - Starting to record. 13:03:18 [xUnit] [INFO] - Processing Custom Tool 13:03:18 [xUnit] [INFO] - [Custom Tool] - 29 test report file(s) were found with the pattern 'test-results/tst_*.xml' relative to '/var/lib/jenkins/jobs/qt-5-qtxmlpatterns-dev/workspace/ARCH/ARM' for the testing framework 'Custom Tool'. 13:03:18 [xUnit] [ERROR] - The converted file for the result file '/var/lib/jenkins/jobs/qt-5-qtxmlpatterns-dev/workspace/ARCH/ARM/test-results/tst_xmlpatterns.xml' (during conversion process for the metric 'Custom Tool') is not valid. The report file has been skipped. 13:03:18 [xUnit] [INFO] - Failing BUILD because 'set build failed if errors' option is activated. 13:03:18 [xUnit] [INFO] - There are errors when processing test results. 13:03:18 [xUnit] [INFO] - Skipping tests recording. 13:03:18 [xUnit] [INFO] - Stop build. So, is this change incorrect/incomplete or does junit-4.xsd/xUnit need to be updated ?

            mc1arke Michael Clarke
            davehunt Dave Hunt
            Votes:
            10 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated:
              Resolved: