-
Bug
-
Resolution: Unresolved
-
Minor
-
None
Some test results may include testcases containing system output and system error messages despite being successful, and are incorrectly reported as failure.
See for instance, the case titled "windows-11 / Build (windows-11) / org\jenkinsci\plugins\genexus\GeneXusInstallation\config.jelly – org.jvnet.hudson.test.JellyTestSuiteBuilder$JellyCheck" at [https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fgenexus-plugin/detail/JakartaEE9/12/tests/
The result file TEST-org.jenkinsci.plugins.genexus.GeneXusInstallationTest.xml for this test case looks like this:
<testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report-3.0.xsd" version="3.0" name="org.jenkinsci.plugins.genexus.GeneXusInstallationTest" time="2.25" tests="1" errors="0" skipped="0" failures="0"> <properties> <!- ... --> </properties> <testcase name="testConfigRoundtrip" classname="org.jenkinsci.plugins.genexus.GeneXusInstallationTest" time="2.25"> <system-out><![CDATA[=== Starting testConfigRoundtrip(org.jenkinsci.plugins.genexus.GeneXusInstallationTest) ]]> </system-out> <system-err><![CDATA[ 0.000 [id=73] INFO o.jvnet.hudson.test.JenkinsRule#createWebServer: Running on http://localhost:59339/jenkins/ 0.078 [id=86] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization 0.078 [id=100] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins 0.203 [id=99] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins 0.203 [id=87] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins 0.203 [id=91] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions 1.219 [id=86] INFO jenkins.InitReactorRunner$1#onAttained: System config loaded 1.219 [id=94] INFO jenkins.InitReactorRunner$1#onAttained: System config adapted 1.219 [id=94] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs 1.219 [id=96] INFO jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated 1.297 [id=87] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization 1.953 [id=73] INFO jenkins.model.Jenkins#cleanUp: Stopping Jenkins 2.078 [id=73] INFO jenkins.model.Jenkins#cleanUp: Jenkins stopped]]> </system-err> </testcase> </testsuite>
Test cases like these, which incude <system-out> and <system-err> mesages, are incorrectly interpreted as failures:
[INFO] ------------------------------------------------------- [INFO] Solving test names [INFO] ------------------------------------------------------- [INFO] Reading E:\Dev\plugin-compat-tester\tmp\work\genexus\target\failsafe-reports [WARNING] Unable to retrieve info from: E:\Dev\plugin-compat-tester\tmp\work\genexus\target\failsafe-reports\TEST-org.jenkinsci.plugins.genexus.server.info.VersionListTest.xml [WARNING] Unable to retrieve info from: E:\Dev\plugin-compat-tester\tmp\work\genexus\target\failsafe-reports\TEST-org.jenkinsci.plugins.genexus.builders.GeneXusBuilderTest.xml [WARNING] Unable to retrieve info from: E:\Dev\plugin-compat-tester\tmp\work\genexus\target\failsafe-reports\TEST-org.jenkinsci.plugins.genexus.server.info.RevisionListTest.xml [WARNING] Unable to retrieve info from: E:\Dev\plugin-compat-tester\tmp\work\genexus\target\failsafe-reports\TEST-org.jenkinsci.plugins.genexus.server.info.KBListTest.xml [WARNING] Unable to retrieve info from: E:\Dev\plugin-compat-tester\tmp\work\genexus\target\failsafe-reports\TEST-org.jenkinsci.plugins.genexus.GeneXusInstallationTest.xml [INFO] Reading E:\Dev\plugin-compat-tester\tmp\work\genexus\target\surefire-reports [INFO] Extracted 1 testnames from E:\Dev\plugin-compat-tester\tmp\work\genexus\target\surefire-reports\TEST-org.jenkinsci.plugins.genexus.server.info.VersionListTest.xml [INFO] Extracted 2 testnames from E:\Dev\plugin-compat-tester\tmp\work\genexus\target\surefire-reports\TEST-org.jenkinsci.plugins.genexus.builders.GeneXusBuilderTest.xml [INFO] Extracted 1 testnames from E:\Dev\plugin-compat-tester\tmp\work\genexus\target\surefire-reports\TEST-org.jenkinsci.plugins.genexus.server.info.RevisionListTest.xml [INFO] Extracted 1 testnames from E:\Dev\plugin-compat-tester\tmp\work\genexus\target\surefire-reports\TEST-org.jenkinsci.plugins.genexus.server.info.KBListTest.xml [INFO] Extracted 1 testnames from E:\Dev\plugin-compat-tester\tmp\work\genexus\target\surefire-reports\TEST-org.jenkinsci.plugins.genexus.GeneXusInstallationTest.xml [INFO] [INFO] Results: [INFO] [INFO] Executed: 0 [INFO] [INFO] Failed: 6 [INFO] - org.jenkinsci.plugins.genexus.GeneXusInstallationTest.testConfigRoundtrip [INFO] - org.jenkinsci.plugins.genexus.builders.GeneXusBuilderTest.testConfigRoundtrip [INFO] - org.jenkinsci.plugins.genexus.builders.GeneXusBuilderTest.testGetGxInstallationId [INFO] - org.jenkinsci.plugins.genexus.server.info.KBListTest.testXmlSerialization [INFO] - org.jenkinsci.plugins.genexus.server.info.RevisionListTest.testXmlSerialization [INFO] - org.jenkinsci.plugins.genexus.server.info.VersionListTest.testXmlSerialization
To reproduce, you may just run PCT for artifact-id genexus.