[JENKINS-6700] Failure in test class constructor or @Before method is not reported (was: Maven plugin doesn't set build to unstable on SurefireExecutionException) - Jenkins Jira

XML

Word

Printable

Details

Type: Bug
Status: Resolved (View Workflow)
Priority: Critical
Resolution: Fixed
Component/s: junit-plugin
Labels:
None
Environment:
Hudson 1.351, Maven 2.2.1, Surefire 2.5, TestNG 5.9
Jenkins 1.420

Similar Issues:

Show

Description

Occasionally we have some test which lets Surefire fail hard which Hudson doesn't recognize. I.e. the tests are not executed, but the build result is still success.
E.g. we had a test which failed with a NoSuchFieldException in the constructor of the test class:

Running TestSuite
log4j:WARN No appenders could be found for logger (...).
log4j:WARN Please initialize the log4j system properly.
org.apache.maven.surefire.booter.SurefireExecutionException:
Cannot instantiate class XyzTest; nested exception is org.testng.TestNGException:
Cannot instantiate class XyzTest
org.testng.TestNGException:
Cannot instantiate class XyzTest
at org.testng.internal.ObjectFactoryImpl.newInstance(ObjectFactoryImpl.java:35)
at org.testng.internal.ClassHelper.createInstance(ClassHelper.java:330)
at org.testng.internal.ClassImpl.getDefaultInstance(ClassImpl.java:62)
at org.testng.internal.ClassImpl.getInstances(ClassImpl.java:81)
at org.testng.internal.TestNGClassFinder.<init>(TestNGClassFinder.java:114)
at org.testng.TestRunner.initMethods(TestRunner.java:289)
at org.testng.TestRunner.init(TestRunner.java:235)
at org.testng.TestRunner.init(TestRunner.java:197)
at org.testng.TestRunner.<init>(TestRunner.java:141)
at org.testng.SuiteRunner$DefaultTestRunnerFactory.newTestRunner(SuiteRunner.java:488)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:250)
at org.testng.SuiteRunner.run(SuiteRunner.java:204)
at org.testng.TestNG.createAndRunSuiteRunners(TestNG.java:877)
at org.testng.TestNG.runSuitesLocally(TestNG.java:842)
at org.testng.TestNG.run(TestNG.java:751)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:62)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:141)
at org.apache.maven.surefire.Surefire.run(Surefire.java:180)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:350)
at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1021)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.testng.internal.ObjectFactoryImpl.newInstance(ObjectFactoryImpl.java:26)
... 23 more
Caused by: java.lang.NoSuchFieldException: cipherService
at java.lang.Class.getDeclaredField(Class.java:1882)
at XyzTest.<init>(XyzTest.java:91)
... 28 more
[ERROR] There are test failures.

Attachments

Issue Links

is duplicated by

JENKINS-9770 Maven 2/3 job not failing when tests cannot be run

Resolved

Activity

Ascending order - Click to sort in descending order

Matthew Webber added a comment - 2011-07-22 14:56

This problem is broader than the problem description indicates. It's nothing specifically to do with Maven. Here's my version of the problem:

We're running some very standard Java JUnit4 tests, and using the "Publish JUnit test result report" option. I only just noticed we have failing tests that are never reported in the build results.

The problem is with some tests that have a @Before method, which gets an exception. The JUnit runner writes an XML report file in the expected place, with the error information, but it appears that Jenkins does not do anything useful with it. The XML file looks like:

<?xml version="1.0" encoding="UTF-8" ?>
<testsuite errors="1" failures="0" hostname="whatever" name="gda.device.detector.XHDetectorTest" tests="1" time="0.0" timestamp="2011-07-22T13:12:19">
  <properties>
    <property name="sun.arch.data.model" value="32" />
     Etc etc
  </properties>
  <error message="gda/device/detector/DAServer" type="java.lang.NoClassDefFoundError">java.lang.NoClassDefFoundError: gda/device/detector/DAServer
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:169)
Caused by: java.lang.ClassNotFoundException: gda.device.detector.DAServer
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
</error>
  <system-out><![CDATA[]]></system-out>
  <system-err><![CDATA[]]></system-err>
</testsuite>

I presume that the absence of any <testcase classname="blah" name="blah" time="1.249" /> means that the report file is simply ignored.

I'm somewhat concerned that we have had tests failing for some time, with no indication.

Matthew Webber added a comment - 2011-07-22 14:56 This problem is broader than the problem description indicates. It's nothing specifically to do with Maven. Here's my version of the problem: We're running some very standard Java JUnit4 tests, and using the "Publish JUnit test result report" option. I only just noticed we have failing tests that are never reported in the build results. The problem is with some tests that have a @Before method, which gets an exception. The JUnit runner writes an XML report file in the expected place, with the error information, but it appears that Jenkins does not do anything useful with it. The XML file looks like: <?xml version="1.0" encoding="UTF-8" ?> <testsuite errors="1" failures="0" hostname="whatever" name="gda.device.detector.XHDetectorTest" tests="1" time="0.0" timestamp="2011-07-22T13:12:19"> <properties> <property name="sun.arch.data.model" value="32" /> Etc etc </properties> <error message="gda/device/detector/DAServer" type="java.lang.NoClassDefFoundError">java.lang.NoClassDefFoundError: gda/device/detector/DAServer at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:169) Caused by: java.lang.ClassNotFoundException: gda.device.detector.DAServer at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) </error> <system-out><![CDATA[]]></system-out> <system-err><![CDATA[]]></system-err> </testsuite> I presume that the absence of any <testcase classname="blah" name="blah" time="1.249" /> means that the report file is simply ignored. I'm somewhat concerned that we have had tests failing for some time, with no indication.

SCM/JIRA link daemon added a comment - 2011-07-23 09:26

Code changed in jenkins
User: Christoph Kutzinski
Path:
changelog.html
core/src/main/java/hudson/tasks/junit/SuiteResult.java
test/src/test/java/hudson/tasks/junit/SuiteResultTest.java
http://jenkins-ci.org/commit/jenkins/a2c8673da13b032ea0afc745219c9c021a93efb3
Log:
[FIXED JENKINS-6700] Failure in test class constructor or @Before method was not reported

SCM/JIRA link daemon added a comment - 2011-07-23 09:26 Code changed in jenkins User: Christoph Kutzinski Path: changelog.html core/src/main/java/hudson/tasks/junit/SuiteResult.java test/src/test/java/hudson/tasks/junit/SuiteResultTest.java http://jenkins-ci.org/commit/jenkins/a2c8673da13b032ea0afc745219c9c021a93efb3 Log: [FIXED JENKINS-6700] Failure in test class constructor or @Before method was not reported

SCM/JIRA link daemon added a comment - 2011-07-23 09:30

Code changed in jenkins
User: Christoph Kutzinski
Path:
test/src/test/resources/hudson/tasks/junit/junit-report-6700.xml
http://jenkins-ci.org/commit/jenkins/6168fba02dbd6100f17540a00bbe8c6094dafd9c
Log:
[FIXED JENKINS-6700] Failure in test class constructor or @Before method was not reported

SCM/JIRA link daemon added a comment - 2011-07-23 09:30 Code changed in jenkins User: Christoph Kutzinski Path: test/src/test/resources/hudson/tasks/junit/junit-report-6700.xml http://jenkins-ci.org/commit/jenkins/6168fba02dbd6100f17540a00bbe8c6094dafd9c Log: [FIXED JENKINS-6700] Failure in test class constructor or @Before method was not reported

kutzi added a comment - 2011-07-23 09:33 - edited

Should be fixed in 1.423

kutzi added a comment - 2011-07-23 09:33 - edited Should be fixed in 1.423

dogfood added a comment - 2011-07-23 10:04

Integrated in jenkins_main_trunk #966
[FIXED JENKINS-6700] Failure in test class constructor or @Before method was not reported
[FIXED JENKINS-6700] Failure in test class constructor or @Before method was not reported

Christoph Kutzinski : a2c8673da13b032ea0afc745219c9c021a93efb3
Files :

test/src/test/java/hudson/tasks/junit/SuiteResultTest.java
core/src/main/java/hudson/tasks/junit/SuiteResult.java
changelog.html

Christoph Kutzinski : 6168fba02dbd6100f17540a00bbe8c6094dafd9c
Files :

test/src/test/resources/hudson/tasks/junit/junit-report-6700.xml

dogfood added a comment - 2011-07-23 10:04 Integrated in jenkins_main_trunk #966 [FIXED JENKINS-6700] Failure in test class constructor or @Before method was not reported [FIXED JENKINS-6700] Failure in test class constructor or @Before method was not reported Christoph Kutzinski : a2c8673da13b032ea0afc745219c9c021a93efb3 Files : test/src/test/java/hudson/tasks/junit/SuiteResultTest.java core/src/main/java/hudson/tasks/junit/SuiteResult.java changelog.html Christoph Kutzinski : 6168fba02dbd6100f17540a00bbe8c6094dafd9c Files : test/src/test/resources/hudson/tasks/junit/junit-report-6700.xml

dogfood added a comment - 2011-07-23 23:51

Integrated in jenkins_main_trunk #975
[FIXED JENKINS-6700] Failure in test class constructor or @Before method was not reported
[FIXED JENKINS-6700] Failure in test class constructor or @Before method was not reported

Christoph Kutzinski : a2c8673da13b032ea0afc745219c9c021a93efb3
Files :

core/src/main/java/hudson/tasks/junit/SuiteResult.java
test/src/test/java/hudson/tasks/junit/SuiteResultTest.java
changelog.html

Christoph Kutzinski : 6168fba02dbd6100f17540a00bbe8c6094dafd9c
Files :

test/src/test/resources/hudson/tasks/junit/junit-report-6700.xml

dogfood added a comment - 2011-07-23 23:51 Integrated in jenkins_main_trunk #975 [FIXED JENKINS-6700] Failure in test class constructor or @Before method was not reported [FIXED JENKINS-6700] Failure in test class constructor or @Before method was not reported Christoph Kutzinski : a2c8673da13b032ea0afc745219c9c021a93efb3 Files : core/src/main/java/hudson/tasks/junit/SuiteResult.java test/src/test/java/hudson/tasks/junit/SuiteResultTest.java changelog.html Christoph Kutzinski : 6168fba02dbd6100f17540a00bbe8c6094dafd9c Files : test/src/test/resources/hudson/tasks/junit/junit-report-6700.xml

Matthew Webber added a comment - 2011-08-04 08:02

I can confirm that upgrading to Jenkins 1.424 fixed the problem for us - tests that were failing in the @BeforeClass are now correctly reported as failing. Thanks!

Matthew Webber added a comment - 2011-08-04 08:02 I can confirm that upgrading to Jenkins 1.424 fixed the problem for us - tests that were failing in the @BeforeClass are now correctly reported as failing. Thanks!

SCM/JIRA link daemon added a comment - 2011-08-19 00:03

Code changed in jenkins
User: Christoph Kutzinski
Path:
changelog.html
core/src/main/java/hudson/tasks/junit/SuiteResult.java
core/src/test/java/hudson/tasks/junit/SuiteResultTest.java
http://jenkins-ci.org/commit/jenkins/02486455fd1dae745baf09ee7e7a5e7bb89ca26d
Log:
[FIXED JENKINS-6700] Failure in test class constructor or @Before method was not reported(cherry picked from commit a2c8673da13b032ea0afc745219c9c021a93efb3)

Conflicts:

changelog.html
core/src/test/java/hudson/tasks/junit/SuiteResultTest.java

SCM/JIRA link daemon added a comment - 2011-08-19 00:03 Code changed in jenkins User: Christoph Kutzinski Path: changelog.html core/src/main/java/hudson/tasks/junit/SuiteResult.java core/src/test/java/hudson/tasks/junit/SuiteResultTest.java http://jenkins-ci.org/commit/jenkins/02486455fd1dae745baf09ee7e7a5e7bb89ca26d Log: [FIXED JENKINS-6700] Failure in test class constructor or @Before method was not reported(cherry picked from commit a2c8673da13b032ea0afc745219c9c021a93efb3) Conflicts: changelog.html core/src/test/java/hudson/tasks/junit/SuiteResultTest.java

SCM/JIRA link daemon added a comment - 2011-08-19 00:03

Code changed in jenkins
User: Christoph Kutzinski
Path:
test/src/test/resources/hudson/tasks/junit/junit-report-6700.xml
http://jenkins-ci.org/commit/jenkins/ce54cd0a86abb1c5c337e33f5aaa25a4869fd341
Log:
[FIXED JENKINS-6700] Failure in test class constructor or @Before method was not reported(cherry picked from commit 6168fba02dbd6100f17540a00bbe8c6094dafd9c)

SCM/JIRA link daemon added a comment - 2011-08-19 00:03 Code changed in jenkins User: Christoph Kutzinski Path: test/src/test/resources/hudson/tasks/junit/junit-report-6700.xml http://jenkins-ci.org/commit/jenkins/ce54cd0a86abb1c5c337e33f5aaa25a4869fd341 Log: [FIXED JENKINS-6700] Failure in test class constructor or @Before method was not reported(cherry picked from commit 6168fba02dbd6100f17540a00bbe8c6094dafd9c)

SCM/JIRA link daemon added a comment - 2011-09-12 02:50

SCM/JIRA link daemon added a comment - 2011-09-12 02:50 Code changed in jenkins User: Christoph Kutzinski Path: changelog.html core/src/main/java/hudson/tasks/junit/SuiteResult.java test/src/test/java/hudson/tasks/junit/SuiteResultTest.java http://jenkins-ci.org/commit/jenkins/a2c8673da13b032ea0afc745219c9c021a93efb3 Log: [FIXED JENKINS-6700] Failure in test class constructor or @Before method was not reported

SCM/JIRA link daemon added a comment - 2011-09-12 02:50

SCM/JIRA link daemon added a comment - 2011-09-12 02:50 Code changed in jenkins User: Christoph Kutzinski Path: test/src/test/resources/hudson/tasks/junit/junit-report-6700.xml http://jenkins-ci.org/commit/jenkins/6168fba02dbd6100f17540a00bbe8c6094dafd9c Log: [FIXED JENKINS-6700] Failure in test class constructor or @Before method was not reported

SCM/JIRA link daemon added a comment - 2011-10-20 17:41

SCM/JIRA link daemon added a comment - 2011-10-20 17:41 Code changed in jenkins User: Christoph Kutzinski Path: changelog.html core/src/main/java/hudson/tasks/junit/SuiteResult.java test/src/test/java/hudson/tasks/junit/SuiteResultTest.java http://jenkins-ci.org/commit/jenkins/a2c8673da13b032ea0afc745219c9c021a93efb3 Log: [FIXED JENKINS-6700] Failure in test class constructor or @Before method was not reported

SCM/JIRA link daemon added a comment - 2011-10-20 17:41

SCM/JIRA link daemon added a comment - 2011-10-20 17:41 Code changed in jenkins User: Christoph Kutzinski Path: test/src/test/resources/hudson/tasks/junit/junit-report-6700.xml http://jenkins-ci.org/commit/jenkins/6168fba02dbd6100f17540a00bbe8c6094dafd9c Log: [FIXED JENKINS-6700] Failure in test class constructor or @Before method was not reported

People

Assignee:: kutzi

Reporter:: kutzi

Votes:: 3 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 2010-06-04 10:24

Updated:: 2011-10-20 17:41

Resolved:: 2011-07-23 09:26