Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-31258

Jenkins Maven plug-in ignores JUnit-format test results from unknown Maven plug-ins

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • maven-plugin
    • None
    • Windows 8.1. JavaSE-1.8. Jenkins 1.635.

      The Jenkins Maven plug-in ignores JUnit format files in the test results directory (${project.build.directory}/surefire-reports by default). Debugging revealed the cause to be that maven-plugin knows about certain* test-capable MOJOs and attempts to support unknown MOJOs by making assumptions which may or may not be justified, depending on the MOJO in question:

      1. that the MOJO will target the 'test' goal or similar
      2. that the MOJO configuration will have a 'reportsDirectory' property

      * maven testing plug-ins known to the Jenkins Maven plug-in / project type as of 2014-12-04 13:40:29:

      • org.apache.maven.plugins:maven-surefire-plugin
      • net.kennychua:phantomjs-qunit-runner
      • org.apache.maven.plugins:maven-failsafe-plugin
      • com.sun.maven:maven-junit-plugin
      • org.sonatype.flexmojos:flexmojos-maven-plugin
      • org.sonatype.tycho:maven-osgi-test-plugin
      • org.eclipse.tycho:tycho-surefire-plugin
      • com.jayway.maven.plugins.android.generation2:maven-android-plugin
      • com.jayway.maven.plugins.android.generation2:android-maven-plugin
      • org.codehaus.mojo:gwt-maven-plugin
      • com.smartbear.soapui:soapui-maven-plugin
      • com.smartbear.soapui:soapui-pro-maven-plugin
      • com.github.redfish4ktc.soapui:maven-soapui-extension-plugin
      • com.github.searls:jasmine-maven-plugin
      • org.terracotta.maven.plugins:toolkit-resolver-plugin
      • org.scalatest:scalatest-maven-plugin
      • org.nanoko.playframework:play2-maven-plugin

      Examples of unknown/unsupported Maven test-capable plug-ins:

      • com.github.eirslett:frontend-maven-plugin:karma
      • com.kelveden:maven-karma-plugin:start
      • com.github.greengerong:maven-ng-protractor:run
      • org.codehaus.mojo:exec-maven-plugin:exec (can launch any arbitrary testing process)
      • others, doubtless...

      In the general 'unknown Maven plug-in' case neither of these assumptions are likely to be valid e.g., com.github.eirslett:frontend-maven-plugin running Karma tests, the goal is 'karma' (phase = 'test' or 'integration-test', typically) and the test output directory is configured in a separate karma.conf.js file.

      Overall, this means there is no general solution for reporting standard JUnit-format results that have been placed in the Maven-standard location for such results, other than to use only plug-ins that the Jenkins Maven plugin actually knows about! This is inflexible and an unreasonable limitation for a general-purpose CI tool and a highly extensible build tool.

      I see three possible solutions:

      1. The Jenkins Job configuration for Maven projects provides some declarative means to identify additional test reports.
        • PROs:
          • straightforward to configure
          • no Jenkins-specific POM changes required
          • only one set of changes required
        • CONs:
          • requires explicit configuration in every Jenkins Maven job
      2. Develop a Maven plug-in that could be incorporated into a POM's pluginManagement section purely as the means to indicate that a given plug-in execution (specified as the groupdId, artifactId, version, goal and phase of the otherwise unknown-to-Jenkins Maven test plug-in) produces test results in the specified directory).
        • PROs:
          • should work OOTB in any up-to-date Jenkins installation
          • the Maven plug-in could be extended to cover additional capabilities where Jenkins' built-in Maven integration is inadquate
        • CONs:
          • additional POM complexity purely to support Jenkins integration
          • requires changes to the Jenkins-Maven integration plus a new Maven plug-in
      3. Support a Maven property name format that can supply the equivalent information. Something like:
        • jenkins.<goal>/<group-id>:<artifact-id>[:<version>].reportsDirectory = <test-reports-path>
          • e.g. jenkins.integration-test/org.codehaus.mojo:exec-maven-plugin:1.4.0.reportsDirectory = target/surefire-reports (inexpressible in XML)
        • jenkins.<plugin-execution-id>.reportsDirectory = <test-reports-path>
          • e.g. jenkins.run-protractor-tests.reportsDirectory = target/surefire-reports
        • PROs:
          • minimal changes required to implement
          • should work OOTB in any up-to-date Jenkins installation
        • CONs:
          • somewhat arcane syntax, not readily discoverable
          • sets a sub-optimal precedent for other such extension metadata

      I have prepared a simple implementation of Option 3 for which I am awaiting legal approval to submit a Pull Request.

          [JENKINS-31258] Jenkins Maven plug-in ignores JUnit-format test results from unknown Maven plug-ins

          Adrian Price added a comment -

          Fixed in Pull Request #56.

          Adrian Price added a comment - Fixed in Pull Request #56 .

          Code changed in jenkins
          User: demonfiddler
          Path:
          src/main/java/hudson/maven/reporters/TestMojo.java
          src/main/resources/hudson/maven/Messages.properties
          src/test/java/hudson/maven/MojoInfoBuilder.java
          src/test/java/hudson/maven/reporters/TestMojoTest.java
          http://jenkins-ci.org/commit/maven-plugin/bb25b8524cbaa2f5c11226140617c994672b3ce5
          Log:
          [FIXED JENKINS-31258]

          • Unknown test plug-ins can now specify the path to their reports directory via the Maven project property jenkins.<execution-id>.reportsDirectory. Relative paths are resolved against the project's base directory.
          • Unit test for the above capability.
          • Fixed a compilation failure that was apparently caused an extraneous (and inconsistent!) Unicode right-apostrophe character instead of a single quote in Messages.properties

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: demonfiddler Path: src/main/java/hudson/maven/reporters/TestMojo.java src/main/resources/hudson/maven/Messages.properties src/test/java/hudson/maven/MojoInfoBuilder.java src/test/java/hudson/maven/reporters/TestMojoTest.java http://jenkins-ci.org/commit/maven-plugin/bb25b8524cbaa2f5c11226140617c994672b3ce5 Log: [FIXED JENKINS-31258] Unknown test plug-ins can now specify the path to their reports directory via the Maven project property jenkins.<execution-id>.reportsDirectory. Relative paths are resolved against the project's base directory. Unit test for the above capability. Fixed a compilation failure that was apparently caused an extraneous (and inconsistent!) Unicode right-apostrophe character instead of a single quote in Messages.properties

          Code changed in jenkins
          User: Adrian Price
          Path:
          src/main/java/hudson/maven/reporters/SurefireArchiver.java
          src/main/java/hudson/maven/reporters/TestMojo.java
          src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java
          src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java
          http://jenkins-ci.org/commit/maven-plugin/45712ad8938580c1ae1a8c76095917f119d3f0be
          Log:
          Implement Oleg's code review suggestions:

          • replace extraneous tab characters with spaces
          • follow Oracle coding standards
          • remove unnecessary null check
          • add test for JENKINS-31258 'detect test Mojo identified by Jenkins Maven Property'
          • add test for JENKINS-31524 'ensure updated existing results are counted'

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Adrian Price Path: src/main/java/hudson/maven/reporters/SurefireArchiver.java src/main/java/hudson/maven/reporters/TestMojo.java src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java http://jenkins-ci.org/commit/maven-plugin/45712ad8938580c1ae1a8c76095917f119d3f0be Log: Implement Oleg's code review suggestions: replace extraneous tab characters with spaces follow Oracle coding standards remove unnecessary null check add test for JENKINS-31258 'detect test Mojo identified by Jenkins Maven Property' add test for JENKINS-31524 'ensure updated existing results are counted'

          Code changed in jenkins
          User: Olivier Lamy
          Path:
          src/main/java/hudson/maven/reporters/SurefireArchiver.java
          src/main/java/hudson/maven/reporters/TestMojo.java
          src/main/resources/hudson/maven/Messages.properties
          src/test/java/hudson/maven/MojoInfoBuilder.java
          src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java
          src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java
          src/test/java/hudson/maven/reporters/TestMojoTest.java
          http://jenkins-ci.org/commit/maven-plugin/3ba74bd9564ce7af7300825341e4d392a2fa61bc
          Log:
          Merge pull request #56 from demonfiddler/master

          Merge fixes for JENKINS-31258 and JENKINS-31524.
          As it's merged now @demonfiddler please document the new expression in the wiki page.

          Compare: https://github.com/jenkinsci/maven-plugin/compare/0223184e413c...3ba74bd9564c

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Olivier Lamy Path: src/main/java/hudson/maven/reporters/SurefireArchiver.java src/main/java/hudson/maven/reporters/TestMojo.java src/main/resources/hudson/maven/Messages.properties src/test/java/hudson/maven/MojoInfoBuilder.java src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java src/test/java/hudson/maven/reporters/TestMojoTest.java http://jenkins-ci.org/commit/maven-plugin/3ba74bd9564ce7af7300825341e4d392a2fa61bc Log: Merge pull request #56 from demonfiddler/master Merge fixes for JENKINS-31258 and JENKINS-31524 . As it's merged now @demonfiddler please document the new expression in the wiki page. Compare: https://github.com/jenkinsci/maven-plugin/compare/0223184e413c...3ba74bd9564c

          adrianp could update the documentation please (wiki) ?

          Arnaud Héritier added a comment - adrianp could update the documentation please (wiki) ?

          Adrian Price added a comment -

          Arnaud, I documented the new Maven property and related matters (see the new paragraph headed 'Maven Surefire Test Results') on Mar 02, 2016 10:12.

          Adrian Price added a comment - Arnaud, I documented the new Maven property and related matters (see the new paragraph headed 'Maven Surefire Test Results') on Mar 02, 2016 10:12.

          thanks adrianprice
          Fixed in 2.13

          Arnaud Héritier added a comment - thanks adrianprice Fixed in 2.13

            Unassigned Unassigned
            adrianp Adrian Price
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: