Successful testscase incorrectly identified as failure

ExportXMLWordPrintable

      Some test results may include testcases containing system output and system error messages despite being successful, and are incorrectly reported as failure.

      See for instance, the case titled "windows-11 / Build (windows-11) / org\jenkinsci\plugins\genexus\GeneXusInstallation\config.jelly – org.jvnet.hudson.test.JellyTestSuiteBuilder$JellyCheck" at [https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fgenexus-plugin/detail/JakartaEE9/12/tests/

      The result file TEST-org.jenkinsci.plugins.genexus.GeneXusInstallationTest.xml for this test case looks like this:

      <testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report-3.0.xsd" version="3.0" name="org.jenkinsci.plugins.genexus.GeneXusInstallationTest" time="2.25" tests="1" errors="0" skipped="0" failures="0">
   <properties>
   <!- ... -->
   </properties>
   <testcase name="testConfigRoundtrip" classname="org.jenkinsci.plugins.genexus.GeneXusInstallationTest" time="2.25">
      <system-out><![CDATA[=== Starting testConfigRoundtrip(org.jenkinsci.plugins.genexus.GeneXusInstallationTest) ]]>
      </system-out>
      <system-err><![CDATA[   0.000 [id=73] INFO o.jvnet.hudson.test.JenkinsRule#createWebServer: Running on http://localhost:59339/jenkins/
    0.078 [id=86] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization
    0.078 [id=100] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins
    0.203 [id=99] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
    0.203 [id=87] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins
    0.203 [id=91] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
    1.219 [id=86] INFO jenkins.InitReactorRunner$1#onAttained: System config loaded
    1.219 [id=94] INFO jenkins.InitReactorRunner$1#onAttained: System config adapted
    1.219 [id=94] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
    1.219 [id=96] INFO jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
    1.297 [id=87] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization
    1.953 [id=73] INFO jenkins.model.Jenkins#cleanUp: Stopping Jenkins
    2.078 [id=73] INFO jenkins.model.Jenkins#cleanUp: Jenkins stopped]]>
      </system-err>
   </testcase>
</testsuite>

      Test cases like these, which incude <system-out> and <system-err> mesages, are incorrectly interpreted as failures:
       

      [INFO] -------------------------------------------------------
[INFO] Solving test names
[INFO] -------------------------------------------------------
[INFO] Reading E:\Dev\plugin-compat-tester\tmp\work\genexus\target\failsafe-reports
[WARNING] Unable to retrieve info from: E:\Dev\plugin-compat-tester\tmp\work\genexus\target\failsafe-reports\TEST-org.jenkinsci.plugins.genexus.server.info.VersionListTest.xml
[WARNING] Unable to retrieve info from: E:\Dev\plugin-compat-tester\tmp\work\genexus\target\failsafe-reports\TEST-org.jenkinsci.plugins.genexus.builders.GeneXusBuilderTest.xml
[WARNING] Unable to retrieve info from: E:\Dev\plugin-compat-tester\tmp\work\genexus\target\failsafe-reports\TEST-org.jenkinsci.plugins.genexus.server.info.RevisionListTest.xml
[WARNING] Unable to retrieve info from: E:\Dev\plugin-compat-tester\tmp\work\genexus\target\failsafe-reports\TEST-org.jenkinsci.plugins.genexus.server.info.KBListTest.xml
[WARNING] Unable to retrieve info from: E:\Dev\plugin-compat-tester\tmp\work\genexus\target\failsafe-reports\TEST-org.jenkinsci.plugins.genexus.GeneXusInstallationTest.xml
[INFO] Reading E:\Dev\plugin-compat-tester\tmp\work\genexus\target\surefire-reports
[INFO] Extracted 1 testnames from E:\Dev\plugin-compat-tester\tmp\work\genexus\target\surefire-reports\TEST-org.jenkinsci.plugins.genexus.server.info.VersionListTest.xml
[INFO] Extracted 2 testnames from E:\Dev\plugin-compat-tester\tmp\work\genexus\target\surefire-reports\TEST-org.jenkinsci.plugins.genexus.builders.GeneXusBuilderTest.xml
[INFO] Extracted 1 testnames from E:\Dev\plugin-compat-tester\tmp\work\genexus\target\surefire-reports\TEST-org.jenkinsci.plugins.genexus.server.info.RevisionListTest.xml
[INFO] Extracted 1 testnames from E:\Dev\plugin-compat-tester\tmp\work\genexus\target\surefire-reports\TEST-org.jenkinsci.plugins.genexus.server.info.KBListTest.xml
[INFO] Extracted 1 testnames from E:\Dev\plugin-compat-tester\tmp\work\genexus\target\surefire-reports\TEST-org.jenkinsci.plugins.genexus.GeneXusInstallationTest.xml
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Executed: 0
[INFO] 
[INFO] Failed: 6
[INFO] - org.jenkinsci.plugins.genexus.GeneXusInstallationTest.testConfigRoundtrip
[INFO] - org.jenkinsci.plugins.genexus.builders.GeneXusBuilderTest.testConfigRoundtrip
[INFO] - org.jenkinsci.plugins.genexus.builders.GeneXusBuilderTest.testGetGxInstallationId
[INFO] - org.jenkinsci.plugins.genexus.server.info.KBListTest.testXmlSerialization
[INFO] - org.jenkinsci.plugins.genexus.server.info.RevisionListTest.testXmlSerialization
[INFO] - org.jenkinsci.plugins.genexus.server.info.VersionListTest.testXmlSerialization

       

      To reproduce, you may just run PCT for artifact-id genexus. 

       

       

       

       

       

            Assignee:
            Unassigned
            Reporter:
            José Lamas
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: