-
Bug
-
Resolution: Fixed
-
Major
-
None
-
Powered by SuggestiMate
The current implementation of Hudson's Maven support needlessly requires the use of maven-surefire-plugin for executing tests. If an alternative test runner plugin is used, Hudson's maven-plugin will not recognize test results, meaning that UNSTABLE builds are wrongly considered to be FAILED builds, and also that build status does not include test result reports.
This behavior is due to the fact that the SurefireArchiver class includes conditional logic that insists that the test runner plugin is Surefire. If this conditional check is removed, then test results are correctly found and interpreted, and unstable builds are no longer considered failed builds.
I've attached a simple patch that demonstrates this. Even though the contents of the patch may not be the correct fix, it at least illustrates the nature of the problem.
- maven-plugin-kutzi.hpi
- 8.82 MB
- SurfireArchiver.patch
- 0.7 kB
- is blocking
-
JENKINS-16121 maven-soapui-plugin configurable outputFolder
-
- Resolved
-
- is duplicated by
-
JENKINS-13188 Recognize test results from com.github.searls:jasmine-maven-plugin
-
- Resolved
-
- is related to
-
JENKINS-16776 surefire-reports not detected for android-maven-plugin
-
- Resolved
-
[JENKINS-8334] maven-plugin insists on Surefire to run tests
What kind of plugin are you using for the tests?
There are already additional plugins supported (maven-junit-plugin, flexmojos-maven-plugin), maybe we can support your plugin, too.
our plugin is really just a big hack and just used internally in our projects. It isn't likely ever going to become another "standard" test runner or anything.
We're essentially subclassing the standard surefire one so we can do some programmatic adjustments to the test classpath. I'd love to find another way to accomplish this but this is the only way we've found to run our special test setups effectively
I'm with timeck. If you could add our plugin into the list of recognized test plugins for now, that could work also:
<groupId>org.terracotta.maven.plugins</groupId>
<artifactId>toolkit-resolver-plugin</artifactId>
I don't think it's the right way to add every in-house plugin to the list of recognized plugins.
Also, I don't want to remove the check altogether, as I suppose it was added with a purpose (which I don't know)
Maybe there are some custom jenkins plugins which could help you? Have you looked for them? I'm thinking about something like the xUnit plugin. Not sure if it can help in your case.
Sorry, that I cannot do more fore your.
There was no comment as to why the check needed in the code but it seems like the author was making sure he's parsing JUnit reports format.
I understand your point but if there's away to disable the check or a configuration to add a test plugin to the recognized plugin list, it could be a compromise
Could you please add this option to SurefireArchiver.postExecute() bypass the check?
- if (!isSurefireTest(mojo)) return true; + if (!Boolean.getBoolean("jenkins.ignore.surefire.check")) { + if (!isSurefireTest(mojo)) return true; + }
This is super important for supporting alternate JVM languages in maven builds. The xUnit plugin plugs into the free-style build, so it doesn't help if your project is configured as a maven build.
Why not just check for junit xml files after all the plugins have executed?
> Why not just check for junit xml files after all the plugins have executed?
That's just not how the Maven jobs work and it would mean a major refactoring to change that - and I'm not even positive that this is a desirable change.
Using a startup switch - like hhuynh - is maybe not a good solution as this would AFAIK mean that every executed mojo (not just test mojos) is handled by the SurefireArchiver - which could mean a significant overhead.
Could we use a "duck-typing" approach?
Some conventions that could identify test plugins?
That is, a test goal is a maven goal that is called "test", then check for output xml files?
Likewise, if there exists a property testFailureIgnore, then set it to true?
I've created a pull request with a proposed fix: https://github.com/jenkinsci/jenkins/pull/668
I've tested it with the renjin R plugin and it works quite nicely. Thanks!
I you like to test the pull request on a 'real' Jenkins environment - which would be great - here's a link to a build
(deleted)
I've tried uploading the HPI to our 'real' jenkins server, but i'm not sure jenkins is using it-- will installation via upload work for a "core" plugin or do i have to replace it on the filesystem and restart jenkins?
Here's a build of the maven-plugin.hpi from the kutzi's pull request.
(see attachment at top of page)
Code changed in jenkins
User: Christoph Kutzinski
Path:
maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java
maven-plugin/src/test/java/hudson/maven/MojoInfoBuilder.java
maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java
maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java
http://jenkins-ci.org/commit/jenkins/900341f4912e3d055c30cc12ea5ed05310bf97df
Log:
Add regression test for SurefireArchiver test mojo detection JENKINS-8334
Code changed in jenkins
User: Christoph Kutzinski
Path:
maven-plugin/src/main/java/hudson/maven/MojoInfo.java
maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java
http://jenkins-ci.org/commit/jenkins/2c0b577aabbd7aa1518eaf2f8111f80f7c549f20
Log:
[FIXED JENKINS-8334] use heuristics to detect test-mojos instead of relying on hard-coded plugin names
Code changed in jenkins
User: Christoph Kutzinski
Path:
core/src/main/java/hudson/tasks/junit/TestResult.java
maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java
maven-plugin/src/main/java/hudson/maven/reporters/TestMojo.java
maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java
maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java
http://jenkins-ci.org/commit/jenkins/b68b84eaa04e521ac99c02257d91802d1ab9f3ff
Log:
Refactored detection of test mojos to simplify SurefireArchiver class JENKINS-8334
Integrated in jenkins_main_trunk #2200
Add regression test for SurefireArchiver test mojo detection JENKINS-8334 (Revision 900341f4912e3d055c30cc12ea5ed05310bf97df)
[FIXED JENKINS-8334] use heuristics to detect test-mojos instead of relying on hard-coded plugin names (Revision 2c0b577aabbd7aa1518eaf2f8111f80f7c549f20)
Refactored detection of test mojos to simplify SurefireArchiver class JENKINS-8334 (Revision b68b84eaa04e521ac99c02257d91802d1ab9f3ff)
Result = SUCCESS
Christoph Kutzinski : 900341f4912e3d055c30cc12ea5ed05310bf97df
Files :
- maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java
- maven-plugin/src/test/java/hudson/maven/MojoInfoBuilder.java
- maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java
- maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java
Christoph Kutzinski : 2c0b577aabbd7aa1518eaf2f8111f80f7c549f20
Files :
- maven-plugin/src/main/java/hudson/maven/MojoInfo.java
- maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java
Christoph Kutzinski : b68b84eaa04e521ac99c02257d91802d1ab9f3ff
Files :
- maven-plugin/src/main/java/hudson/maven/reporters/TestMojo.java
- maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java
- maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java
- maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java
- core/src/main/java/hudson/tasks/junit/TestResult.java
Code changed in jenkins
User: Christoph Kutzinski
Path:
changelog.html
core/src/main/java/hudson/tasks/junit/TestResult.java
maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java
http://jenkins-ci.org/commit/jenkins/6b1cc47024172238a65d45f28db84f13495d8780
Log:
Changelog and cleanup for JENKINS-8334
Integrated in jenkins_main_trunk #2201
Changelog and cleanup for JENKINS-8334 (Revision 6b1cc47024172238a65d45f28db84f13495d8780)
Result = SUCCESS
Christoph Kutzinski : 6b1cc47024172238a65d45f28db84f13495d8780
Files :
- maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java
- changelog.html
- core/src/main/java/hudson/tasks/junit/TestResult.java
Code changed in jenkins
User: Christoph Kutzinski
Path:
src/main/java/hudson/maven/reporters/SurefireArchiver.java
src/test/java/hudson/maven/MojoInfoBuilder.java
src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java
src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java
http://jenkins-ci.org/commit/maven-plugin/61f6325a51ca37b4663cc096a79056755c764b94
Log:
Add regression test for SurefireArchiver test mojo detection JENKINS-8334
Originally-Committed-As: 900341f4912e3d055c30cc12ea5ed05310bf97df
Code changed in jenkins
User: Christoph Kutzinski
Path:
src/main/java/hudson/maven/MojoInfo.java
src/main/java/hudson/maven/reporters/SurefireArchiver.java
http://jenkins-ci.org/commit/maven-plugin/05d64e51bb883e2d684980ad8a8f52ce89761226
Log:
[FIXED JENKINS-8334] use heuristics to detect test-mojos instead of relying on hard-coded plugin names
Originally-Committed-As: 2c0b577aabbd7aa1518eaf2f8111f80f7c549f20
Code changed in jenkins
User: Christoph Kutzinski
Path:
src/main/java/hudson/maven/reporters/SurefireArchiver.java
src/main/java/hudson/maven/reporters/TestMojo.java
src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java
src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java
http://jenkins-ci.org/commit/maven-plugin/4d603c4c96f155821f15b0298928a512eea44599
Log:
Refactored detection of test mojos to simplify SurefireArchiver class JENKINS-8334
Originally-Committed-As: b68b84eaa04e521ac99c02257d91802d1ab9f3ff
Code changed in jenkins
User: Christoph Kutzinski
Path:
src/main/java/hudson/maven/reporters/SurefireArchiver.java
http://jenkins-ci.org/commit/maven-plugin/487579aacc73afca228d4533598b4f44877da108
Log:
Changelog and cleanup for JENKINS-8334
Originally-Committed-As: 6b1cc47024172238a65d45f28db84f13495d8780
to be sure, using a different other than surefire for running tests is likely a very odd exception, but we are indeed doing that at the moment. What would be nice is way to disable the check via config, or a way to allow customization of the MojoInfo inspection so we could cover our plugin too.