Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-6700

Failure in test class constructor or @Before method is not reported (was: Maven plugin doesn't set build to unstable on SurefireExecutionException)

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • junit-plugin
    • None
    • Hudson 1.351, Maven 2.2.1, Surefire 2.5, TestNG 5.9
      Jenkins 1.420

      Occasionally we have some test which lets Surefire fail hard which Hudson doesn't recognize. I.e. the tests are not executed, but the build result is still success.
      E.g. we had a test which failed with a NoSuchFieldException in the constructor of the test class:

      Running TestSuite
      log4j:WARN No appenders could be found for logger (...).
      log4j:WARN Please initialize the log4j system properly.
      org.apache.maven.surefire.booter.SurefireExecutionException:
      Cannot instantiate class XyzTest; nested exception is org.testng.TestNGException:
      Cannot instantiate class XyzTest
      org.testng.TestNGException:
      Cannot instantiate class XyzTest
      at org.testng.internal.ObjectFactoryImpl.newInstance(ObjectFactoryImpl.java:35)
      at org.testng.internal.ClassHelper.createInstance(ClassHelper.java:330)
      at org.testng.internal.ClassImpl.getDefaultInstance(ClassImpl.java:62)
      at org.testng.internal.ClassImpl.getInstances(ClassImpl.java:81)
      at org.testng.internal.TestNGClassFinder.<init>(TestNGClassFinder.java:114)
      at org.testng.TestRunner.initMethods(TestRunner.java:289)
      at org.testng.TestRunner.init(TestRunner.java:235)
      at org.testng.TestRunner.init(TestRunner.java:197)
      at org.testng.TestRunner.<init>(TestRunner.java:141)
      at org.testng.SuiteRunner$DefaultTestRunnerFactory.newTestRunner(SuiteRunner.java:488)
      at org.testng.SuiteRunner.privateRun(SuiteRunner.java:250)
      at org.testng.SuiteRunner.run(SuiteRunner.java:204)
      at org.testng.TestNG.createAndRunSuiteRunners(TestNG.java:877)
      at org.testng.TestNG.runSuitesLocally(TestNG.java:842)
      at org.testng.TestNG.run(TestNG.java:751)
      at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:62)
      at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:141)
      at org.apache.maven.surefire.Surefire.run(Surefire.java:180)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:350)
      at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1021)
      Caused by: java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      at org.testng.internal.ObjectFactoryImpl.newInstance(ObjectFactoryImpl.java:26)
      ... 23 more
      Caused by: java.lang.NoSuchFieldException: cipherService
      at java.lang.Class.getDeclaredField(Class.java:1882)
      at XyzTest.<init>(XyzTest.java:91)
      ... 28 more
      [ERROR] There are test failures.

          [JENKINS-6700] Failure in test class constructor or @Before method is not reported (was: Maven plugin doesn't set build to unstable on SurefireExecutionException)

          kutzi created issue -
          flottis made changes -
          Link New: This issue duplicates JENKINS-4378 [ JENKINS-4378 ]
          kutzi made changes -
          Link New: This issue is related to JENKINS-9770 [ JENKINS-9770 ]

          This problem is broader than the problem description indicates. It's nothing specifically to do with Maven. Here's my version of the problem:

          We're running some very standard Java JUnit4 tests, and using the "Publish JUnit test result report" option. I only just noticed we have failing tests that are never reported in the build results.

          The problem is with some tests that have a @Before method, which gets an exception. The JUnit runner writes an XML report file in the expected place, with the error information, but it appears that Jenkins does not do anything useful with it. The XML file looks like:

          <?xml version="1.0" encoding="UTF-8" ?>
          <testsuite errors="1" failures="0" hostname="whatever" name="gda.device.detector.XHDetectorTest" tests="1" time="0.0" timestamp="2011-07-22T13:12:19">
            <properties>
              <property name="sun.arch.data.model" value="32" />
               Etc etc
            </properties>
            <error message="gda/device/detector/DAServer" type="java.lang.NoClassDefFoundError">java.lang.NoClassDefFoundError: gda/device/detector/DAServer
          	at java.lang.Class.forName0(Native Method)
          	at java.lang.Class.forName(Class.java:169)
          Caused by: java.lang.ClassNotFoundException: gda.device.detector.DAServer
          	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
          	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
          </error>
            <system-out><![CDATA[]]></system-out>
            <system-err><![CDATA[]]></system-err>
          </testsuite>
          

          I presume that the absence of any <testcase classname="blah" name="blah" time="1.249" /> means that the report file is simply ignored.

          I'm somewhat concerned that we have had tests failing for some time, with no indication.

          Matthew Webber added a comment - This problem is broader than the problem description indicates. It's nothing specifically to do with Maven. Here's my version of the problem: We're running some very standard Java JUnit4 tests, and using the "Publish JUnit test result report" option. I only just noticed we have failing tests that are never reported in the build results. The problem is with some tests that have a @Before method, which gets an exception. The JUnit runner writes an XML report file in the expected place, with the error information, but it appears that Jenkins does not do anything useful with it. The XML file looks like: <?xml version="1.0" encoding="UTF-8" ?> <testsuite errors="1" failures="0" hostname="whatever" name="gda.device.detector.XHDetectorTest" tests="1" time="0.0" timestamp="2011-07-22T13:12:19"> <properties> <property name="sun.arch.data.model" value="32" /> Etc etc </properties> <error message="gda/device/detector/DAServer" type="java.lang.NoClassDefFoundError">java.lang.NoClassDefFoundError: gda/device/detector/DAServer at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:169) Caused by: java.lang.ClassNotFoundException: gda.device.detector.DAServer at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) </error> <system-out><![CDATA[]]></system-out> <system-err><![CDATA[]]></system-err> </testsuite> I presume that the absence of any <testcase classname="blah" name="blah" time="1.249" /> means that the report file is simply ignored. I'm somewhat concerned that we have had tests failing for some time, with no indication.
          Matthew Webber made changes -
          Component/s New: junit [ 15499 ]
          Component/s Original: maven2 [ 15487 ]
          Environment Original: Hudson 1.351, Maven 2.2.1, Surefire 2.5, TestNG 5.9 New: Hudson 1.351, Maven 2.2.1, Surefire 2.5, TestNG 5.9
          Jenkins 1.420
          Summary Original: Maven plugin doesn't set build to unstable on SurefireExecutionException New: Failure in test class constructor or @Before method is not reported (was: Maven plugin doesn't set build to unstable on SurefireExecutionException)
          kutzi made changes -
          Assignee New: kutzi [ kutzi ]

          Code changed in jenkins
          User: Christoph Kutzinski
          Path:
          changelog.html
          core/src/main/java/hudson/tasks/junit/SuiteResult.java
          test/src/test/java/hudson/tasks/junit/SuiteResultTest.java
          http://jenkins-ci.org/commit/jenkins/a2c8673da13b032ea0afc745219c9c021a93efb3
          Log:
          [FIXED JENKINS-6700] Failure in test class constructor or @Before method was not reported

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: changelog.html core/src/main/java/hudson/tasks/junit/SuiteResult.java test/src/test/java/hudson/tasks/junit/SuiteResultTest.java http://jenkins-ci.org/commit/jenkins/a2c8673da13b032ea0afc745219c9c021a93efb3 Log: [FIXED JENKINS-6700] Failure in test class constructor or @Before method was not reported
          SCM/JIRA link daemon made changes -
          Resolution New: Fixed [ 1 ]
          Status Original: Open [ 1 ] New: Resolved [ 5 ]

          Code changed in jenkins
          User: Christoph Kutzinski
          Path:
          test/src/test/resources/hudson/tasks/junit/junit-report-6700.xml
          http://jenkins-ci.org/commit/jenkins/6168fba02dbd6100f17540a00bbe8c6094dafd9c
          Log:
          [FIXED JENKINS-6700] Failure in test class constructor or @Before method was not reported

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: test/src/test/resources/hudson/tasks/junit/junit-report-6700.xml http://jenkins-ci.org/commit/jenkins/6168fba02dbd6100f17540a00bbe8c6094dafd9c Log: [FIXED JENKINS-6700] Failure in test class constructor or @Before method was not reported

          kutzi added a comment - - edited

          Should be fixed in 1.423

          kutzi added a comment - - edited Should be fixed in 1.423

            kutzi kutzi
            kutzi kutzi
            Votes:
            3 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: