• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • maven-plugin
    • None

      The current implementation of Hudson's Maven support needlessly requires the use of maven-surefire-plugin for executing tests. If an alternative test runner plugin is used, Hudson's maven-plugin will not recognize test results, meaning that UNSTABLE builds are wrongly considered to be FAILED builds, and also that build status does not include test result reports.

      This behavior is due to the fact that the SurefireArchiver class includes conditional logic that insists that the test runner plugin is Surefire. If this conditional check is removed, then test results are correctly found and interpreted, and unstable builds are no longer considered failed builds.

      I've attached a simple patch that demonstrates this. Even though the contents of the patch may not be the correct fix, it at least illustrates the nature of the problem.

          [JENKINS-8334] maven-plugin insists on Surefire to run tests

          timeck added a comment -

          to be sure, using a different other than surefire for running tests is likely a very odd exception, but we are indeed doing that at the moment. What would be nice is way to disable the check via config, or a way to allow customization of the MojoInfo inspection so we could cover our plugin too.

          timeck added a comment - to be sure, using a different other than surefire for running tests is likely a very odd exception, but we are indeed doing that at the moment. What would be nice is way to disable the check via config, or a way to allow customization of the MojoInfo inspection so we could cover our plugin too.

          kutzi added a comment -

          What kind of plugin are you using for the tests?
          There are already additional plugins supported (maven-junit-plugin, flexmojos-maven-plugin), maybe we can support your plugin, too.

          kutzi added a comment - What kind of plugin are you using for the tests? There are already additional plugins supported (maven-junit-plugin, flexmojos-maven-plugin), maybe we can support your plugin, too.

          timeck added a comment -

          our plugin is really just a big hack and just used internally in our projects. It isn't likely ever going to become another "standard" test runner or anything.

          We're essentially subclassing the standard surefire one so we can do some programmatic adjustments to the test classpath. I'd love to find another way to accomplish this but this is the only way we've found to run our special test setups effectively

          timeck added a comment - our plugin is really just a big hack and just used internally in our projects. It isn't likely ever going to become another "standard" test runner or anything. We're essentially subclassing the standard surefire one so we can do some programmatic adjustments to the test classpath. I'd love to find another way to accomplish this but this is the only way we've found to run our special test setups effectively

          hhuynh added a comment -

          I'm with timeck. If you could add our plugin into the list of recognized test plugins for now, that could work also:

          <groupId>org.terracotta.maven.plugins</groupId>
          <artifactId>toolkit-resolver-plugin</artifactId>

          hhuynh added a comment - I'm with timeck. If you could add our plugin into the list of recognized test plugins for now, that could work also: <groupId>org.terracotta.maven.plugins</groupId> <artifactId>toolkit-resolver-plugin</artifactId>

          kutzi added a comment -

          I don't think it's the right way to add every in-house plugin to the list of recognized plugins.
          Also, I don't want to remove the check altogether, as I suppose it was added with a purpose (which I don't know)

          Maybe there are some custom jenkins plugins which could help you? Have you looked for them? I'm thinking about something like the xUnit plugin. Not sure if it can help in your case.

          Sorry, that I cannot do more fore your.

          kutzi added a comment - I don't think it's the right way to add every in-house plugin to the list of recognized plugins. Also, I don't want to remove the check altogether, as I suppose it was added with a purpose (which I don't know) Maybe there are some custom jenkins plugins which could help you? Have you looked for them? I'm thinking about something like the xUnit plugin. Not sure if it can help in your case. Sorry, that I cannot do more fore your.

          hhuynh added a comment -

          There was no comment as to why the check needed in the code but it seems like the author was making sure he's parsing JUnit reports format.

          I understand your point but if there's away to disable the check or a configuration to add a test plugin to the recognized plugin list, it could be a compromise

          hhuynh added a comment - There was no comment as to why the check needed in the code but it seems like the author was making sure he's parsing JUnit reports format. I understand your point but if there's away to disable the check or a configuration to add a test plugin to the recognized plugin list, it could be a compromise

          hhuynh added a comment - - edited

          Could you please add this option to SurefireArchiver.postExecute() bypass the check?

          -        if (!isSurefireTest(mojo)) return true;
          +        if (!Boolean.getBoolean("jenkins.ignore.surefire.check")) {
          +    	      if (!isSurefireTest(mojo)) return true;
          +        }
          

          hhuynh added a comment - - edited Could you please add this option to SurefireArchiver.postExecute() bypass the check? - if (!isSurefireTest(mojo)) return true ; + if (! Boolean .getBoolean( "jenkins.ignore.surefire.check" )) { + if (!isSurefireTest(mojo)) return true ; + }

          This is super important for supporting alternate JVM languages in maven builds. The xUnit plugin plugs into the free-style build, so it doesn't help if your project is configured as a maven build.

          Why not just check for junit xml files after all the plugins have executed?

          Alexander Bertram added a comment - This is super important for supporting alternate JVM languages in maven builds. The xUnit plugin plugs into the free-style build, so it doesn't help if your project is configured as a maven build. Why not just check for junit xml files after all the plugins have executed?

          kutzi added a comment -

          > Why not just check for junit xml files after all the plugins have executed?

          That's just not how the Maven jobs work and it would mean a major refactoring to change that - and I'm not even positive that this is a desirable change.

          kutzi added a comment - > Why not just check for junit xml files after all the plugins have executed? That's just not how the Maven jobs work and it would mean a major refactoring to change that - and I'm not even positive that this is a desirable change.

          kutzi added a comment -

          Using a startup switch - like hhuynh - is maybe not a good solution as this would AFAIK mean that every executed mojo (not just test mojos) is handled by the SurefireArchiver - which could mean a significant overhead.

          kutzi added a comment - Using a startup switch - like hhuynh - is maybe not a good solution as this would AFAIK mean that every executed mojo (not just test mojos) is handled by the SurefireArchiver - which could mean a significant overhead.

          Could we use a "duck-typing" approach?
          Some conventions that could identify test plugins?

          That is, a test goal is a maven goal that is called "test", then check for output xml files?

          Likewise, if there exists a property testFailureIgnore, then set it to true?

          Alexander Bertram added a comment - Could we use a "duck-typing" approach? Some conventions that could identify test plugins? That is, a test goal is a maven goal that is called "test", then check for output xml files? Likewise, if there exists a property testFailureIgnore, then set it to true?

          kutzi added a comment -

          Sounds reasonable. I'll look into it.

          kutzi added a comment - Sounds reasonable. I'll look into it.

          kutzi added a comment -

          I've created a pull request with a proposed fix: https://github.com/jenkinsci/jenkins/pull/668

          kutzi added a comment - I've created a pull request with a proposed fix: https://github.com/jenkinsci/jenkins/pull/668

          I've tested it with the renjin R plugin and it works quite nicely. Thanks!

          Alexander Bertram added a comment - I've tested it with the renjin R plugin and it works quite nicely. Thanks!

          John Bito added a comment -

          I like kutzi's pull request.

          John Bito added a comment - I like kutzi's pull request.

          kutzi added a comment - - edited

          I you like to test the pull request on a 'real' Jenkins environment - which would be great - here's a link to a build
          (deleted)

          kutzi added a comment - - edited I you like to test the pull request on a 'real' Jenkins environment - which would be great - here's a link to a build (deleted)

          I've tried uploading the HPI to our 'real' jenkins server, but i'm not sure jenkins is using it-- will installation via upload work for a "core" plugin or do i have to replace it on the filesystem and restart jenkins?

          Alexander Bertram added a comment - I've tried uploading the HPI to our 'real' jenkins server, but i'm not sure jenkins is using it-- will installation via upload work for a "core" plugin or do i have to replace it on the filesystem and restart jenkins?

          John Bito added a comment -

          The HPI file is a subversion plugin. Maven plugin available?

          John Bito added a comment - The HPI file is a subversion plugin. Maven plugin available?

          Alexander Bertram added a comment - - edited

          Here's a build of the maven-plugin.hpi from the kutzi's pull request.
          (see attachment at top of page)

          Alexander Bertram added a comment - - edited Here's a build of the maven-plugin.hpi from the kutzi's pull request. (see attachment at top of page)

          Alexander Bertram added a comment - Built from https://github.com/jenkinsci/jenkins/pull/668

          kutzi added a comment -

          Sorry, I was confusing this with another issue I'm working on.

          kutzi added a comment - Sorry, I was confusing this with another issue I'm working on.

          Code changed in jenkins
          User: Christoph Kutzinski
          Path:
          maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java
          maven-plugin/src/test/java/hudson/maven/MojoInfoBuilder.java
          maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java
          maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java
          http://jenkins-ci.org/commit/jenkins/900341f4912e3d055c30cc12ea5ed05310bf97df
          Log:
          Add regression test for SurefireArchiver test mojo detection JENKINS-8334

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java maven-plugin/src/test/java/hudson/maven/MojoInfoBuilder.java maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java http://jenkins-ci.org/commit/jenkins/900341f4912e3d055c30cc12ea5ed05310bf97df Log: Add regression test for SurefireArchiver test mojo detection JENKINS-8334

          Code changed in jenkins
          User: Christoph Kutzinski
          Path:
          maven-plugin/src/main/java/hudson/maven/MojoInfo.java
          maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java
          http://jenkins-ci.org/commit/jenkins/2c0b577aabbd7aa1518eaf2f8111f80f7c549f20
          Log:
          [FIXED JENKINS-8334] use heuristics to detect test-mojos instead of relying on hard-coded plugin names

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: maven-plugin/src/main/java/hudson/maven/MojoInfo.java maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java http://jenkins-ci.org/commit/jenkins/2c0b577aabbd7aa1518eaf2f8111f80f7c549f20 Log: [FIXED JENKINS-8334] use heuristics to detect test-mojos instead of relying on hard-coded plugin names

          Code changed in jenkins
          User: Christoph Kutzinski
          Path:
          core/src/main/java/hudson/tasks/junit/TestResult.java
          maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java
          maven-plugin/src/main/java/hudson/maven/reporters/TestMojo.java
          maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java
          maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java
          http://jenkins-ci.org/commit/jenkins/b68b84eaa04e521ac99c02257d91802d1ab9f3ff
          Log:
          Refactored detection of test mojos to simplify SurefireArchiver class JENKINS-8334

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: core/src/main/java/hudson/tasks/junit/TestResult.java maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java maven-plugin/src/main/java/hudson/maven/reporters/TestMojo.java maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java http://jenkins-ci.org/commit/jenkins/b68b84eaa04e521ac99c02257d91802d1ab9f3ff Log: Refactored detection of test mojos to simplify SurefireArchiver class JENKINS-8334

          dogfood added a comment -

          Integrated in jenkins_main_trunk #2200
          Add regression test for SurefireArchiver test mojo detection JENKINS-8334 (Revision 900341f4912e3d055c30cc12ea5ed05310bf97df)
          [FIXED JENKINS-8334] use heuristics to detect test-mojos instead of relying on hard-coded plugin names (Revision 2c0b577aabbd7aa1518eaf2f8111f80f7c549f20)
          Refactored detection of test mojos to simplify SurefireArchiver class JENKINS-8334 (Revision b68b84eaa04e521ac99c02257d91802d1ab9f3ff)

          Result = SUCCESS
          Christoph Kutzinski : 900341f4912e3d055c30cc12ea5ed05310bf97df
          Files :

          • maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java
          • maven-plugin/src/test/java/hudson/maven/MojoInfoBuilder.java
          • maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java
          • maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java

          Christoph Kutzinski : 2c0b577aabbd7aa1518eaf2f8111f80f7c549f20
          Files :

          • maven-plugin/src/main/java/hudson/maven/MojoInfo.java
          • maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java

          Christoph Kutzinski : b68b84eaa04e521ac99c02257d91802d1ab9f3ff
          Files :

          • maven-plugin/src/main/java/hudson/maven/reporters/TestMojo.java
          • maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java
          • maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java
          • maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java
          • core/src/main/java/hudson/tasks/junit/TestResult.java

          dogfood added a comment - Integrated in jenkins_main_trunk #2200 Add regression test for SurefireArchiver test mojo detection JENKINS-8334 (Revision 900341f4912e3d055c30cc12ea5ed05310bf97df) [FIXED JENKINS-8334] use heuristics to detect test-mojos instead of relying on hard-coded plugin names (Revision 2c0b577aabbd7aa1518eaf2f8111f80f7c549f20) Refactored detection of test mojos to simplify SurefireArchiver class JENKINS-8334 (Revision b68b84eaa04e521ac99c02257d91802d1ab9f3ff) Result = SUCCESS Christoph Kutzinski : 900341f4912e3d055c30cc12ea5ed05310bf97df Files : maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java maven-plugin/src/test/java/hudson/maven/MojoInfoBuilder.java maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java Christoph Kutzinski : 2c0b577aabbd7aa1518eaf2f8111f80f7c549f20 Files : maven-plugin/src/main/java/hudson/maven/MojoInfo.java maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java Christoph Kutzinski : b68b84eaa04e521ac99c02257d91802d1ab9f3ff Files : maven-plugin/src/main/java/hudson/maven/reporters/TestMojo.java maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java core/src/main/java/hudson/tasks/junit/TestResult.java

          Code changed in jenkins
          User: Christoph Kutzinski
          Path:
          changelog.html
          core/src/main/java/hudson/tasks/junit/TestResult.java
          maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java
          http://jenkins-ci.org/commit/jenkins/6b1cc47024172238a65d45f28db84f13495d8780
          Log:
          Changelog and cleanup for JENKINS-8334

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: changelog.html core/src/main/java/hudson/tasks/junit/TestResult.java maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java http://jenkins-ci.org/commit/jenkins/6b1cc47024172238a65d45f28db84f13495d8780 Log: Changelog and cleanup for JENKINS-8334

          dogfood added a comment -

          Integrated in jenkins_main_trunk #2201
          Changelog and cleanup for JENKINS-8334 (Revision 6b1cc47024172238a65d45f28db84f13495d8780)

          Result = SUCCESS
          Christoph Kutzinski : 6b1cc47024172238a65d45f28db84f13495d8780
          Files :

          • maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java
          • changelog.html
          • core/src/main/java/hudson/tasks/junit/TestResult.java

          dogfood added a comment - Integrated in jenkins_main_trunk #2201 Changelog and cleanup for JENKINS-8334 (Revision 6b1cc47024172238a65d45f28db84f13495d8780) Result = SUCCESS Christoph Kutzinski : 6b1cc47024172238a65d45f28db84f13495d8780 Files : maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java changelog.html core/src/main/java/hudson/tasks/junit/TestResult.java

          Code changed in jenkins
          User: Christoph Kutzinski
          Path:
          src/main/java/hudson/maven/reporters/SurefireArchiver.java
          src/test/java/hudson/maven/MojoInfoBuilder.java
          src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java
          src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java
          http://jenkins-ci.org/commit/maven-plugin/61f6325a51ca37b4663cc096a79056755c764b94
          Log:
          Add regression test for SurefireArchiver test mojo detection JENKINS-8334
          Originally-Committed-As: 900341f4912e3d055c30cc12ea5ed05310bf97df

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: src/main/java/hudson/maven/reporters/SurefireArchiver.java src/test/java/hudson/maven/MojoInfoBuilder.java src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java http://jenkins-ci.org/commit/maven-plugin/61f6325a51ca37b4663cc096a79056755c764b94 Log: Add regression test for SurefireArchiver test mojo detection JENKINS-8334 Originally-Committed-As: 900341f4912e3d055c30cc12ea5ed05310bf97df

          Code changed in jenkins
          User: Christoph Kutzinski
          Path:
          src/main/java/hudson/maven/MojoInfo.java
          src/main/java/hudson/maven/reporters/SurefireArchiver.java
          http://jenkins-ci.org/commit/maven-plugin/05d64e51bb883e2d684980ad8a8f52ce89761226
          Log:
          [FIXED JENKINS-8334] use heuristics to detect test-mojos instead of relying on hard-coded plugin names
          Originally-Committed-As: 2c0b577aabbd7aa1518eaf2f8111f80f7c549f20

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: src/main/java/hudson/maven/MojoInfo.java src/main/java/hudson/maven/reporters/SurefireArchiver.java http://jenkins-ci.org/commit/maven-plugin/05d64e51bb883e2d684980ad8a8f52ce89761226 Log: [FIXED JENKINS-8334] use heuristics to detect test-mojos instead of relying on hard-coded plugin names Originally-Committed-As: 2c0b577aabbd7aa1518eaf2f8111f80f7c549f20

          Code changed in jenkins
          User: Christoph Kutzinski
          Path:
          src/main/java/hudson/maven/reporters/SurefireArchiver.java
          src/main/java/hudson/maven/reporters/TestMojo.java
          src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java
          src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java
          http://jenkins-ci.org/commit/maven-plugin/4d603c4c96f155821f15b0298928a512eea44599
          Log:
          Refactored detection of test mojos to simplify SurefireArchiver class JENKINS-8334
          Originally-Committed-As: b68b84eaa04e521ac99c02257d91802d1ab9f3ff

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: src/main/java/hudson/maven/reporters/SurefireArchiver.java src/main/java/hudson/maven/reporters/TestMojo.java src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java http://jenkins-ci.org/commit/maven-plugin/4d603c4c96f155821f15b0298928a512eea44599 Log: Refactored detection of test mojos to simplify SurefireArchiver class JENKINS-8334 Originally-Committed-As: b68b84eaa04e521ac99c02257d91802d1ab9f3ff

          Code changed in jenkins
          User: Christoph Kutzinski
          Path:
          src/main/java/hudson/maven/reporters/SurefireArchiver.java
          http://jenkins-ci.org/commit/maven-plugin/487579aacc73afca228d4533598b4f44877da108
          Log:
          Changelog and cleanup for JENKINS-8334
          Originally-Committed-As: 6b1cc47024172238a65d45f28db84f13495d8780

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: src/main/java/hudson/maven/reporters/SurefireArchiver.java http://jenkins-ci.org/commit/maven-plugin/487579aacc73afca228d4533598b4f44877da108 Log: Changelog and cleanup for JENKINS-8334 Originally-Committed-As: 6b1cc47024172238a65d45f28db84f13495d8780

            kutzi kutzi
            jvoegele jvoegele
            Votes:
            2 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: