• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • maven-plugin
    • None
    • Platform: All, OS: All

      We're building a project with Hudson using Maven2. We're also using the Maven
      Cobertura plugin to generate coverage reports. This means the tests will be
      executed twice, once with instrumentation and once without.

      Hudson is reporting the results from both runs although there is only a single
      set of results in target/surefire-reports/*.xml. Hudson should only report a
      single set of results, preferably those within target/surefire-reports.

          [JENKINS-1557] Duplicate test results with Maven2 projects

          paulamitchell created issue -

          aebbert added a comment -

          I'm having the same issue. I'm wondering if you can tweak the pom to only
          execute the tests once, which should be done anyway, in order to reduce the
          build time.

          Currently I'm using the maven goals "install findbugs:findbugs
          cobertura:cobertura" in my hudson project, but would like to reduce that to
          "install" only. If this would be documented somewhere, then this bug is also
          obsolete, imho.

          aebbert added a comment - I'm having the same issue. I'm wondering if you can tweak the pom to only execute the tests once, which should be done anyway, in order to reduce the build time. Currently I'm using the maven goals "install findbugs:findbugs cobertura:cobertura" in my hudson project, but would like to reduce that to "install" only. If this would be documented somewhere, then this bug is also obsolete, imho.

          It would be great if someone can provide a test case.

          Kohsuke Kawaguchi added a comment - It would be great if someone can provide a test case.

          javanet_ac added a comment -

          The test case is very well described by the initial reporter

          I.e. when building a Maven2 Project in Hudson and also utilizing cobertura
          (Goals e.g. "-U clean deploy findbugs:findbugs checkstyle:checkstyle pmd:pmd
          pmd:cpd cobertura:cobertura javadoc:javadoc") the tests are executed twice (once
          due to deploy/install and once due to cobertura) and Hudson is reporting the
          results from both runs - although there is only a single
          set of results in target/surefire-reports/*.xml

          Interestingly when using the Hudson FreeStyle project the tests are reported
          only once.

          Hints/Speculation:

          • When processing the Maven2 Hudson Project it seems that the test results are
            somewhow stored after the first test run and then again aftre the second
            (Instrumenting) test run.
          • With the FreeStyle Hudson Project the test results are probably processed at
            the end, i.e. after cobertura rewrites the tests on target/surefire-reports?
            This is the preferred behaviour.

          javanet_ac added a comment - The test case is very well described by the initial reporter I.e. when building a Maven2 Project in Hudson and also utilizing cobertura (Goals e.g. "-U clean deploy findbugs:findbugs checkstyle:checkstyle pmd:pmd pmd:cpd cobertura:cobertura javadoc:javadoc") the tests are executed twice (once due to deploy/install and once due to cobertura) and Hudson is reporting the results from both runs - although there is only a single set of results in target/surefire-reports/*.xml Interestingly when using the Hudson FreeStyle project the tests are reported only once. Hints/Speculation: When processing the Maven2 Hudson Project it seems that the test results are somewhow stored after the first test run and then again aftre the second (Instrumenting) test run. With the FreeStyle Hudson Project the test results are probably processed at the end, i.e. after cobertura rewrites the tests on target/surefire-reports? This is the preferred behaviour.

          jweden added a comment -

          I run the cobertura:cobertura, cobertura:check and package goals, so I actually
          get three results reported.

          jweden added a comment - I run the cobertura:cobertura, cobertura:check and package goals, so I actually get three results reported.

          javanet_ac added a comment -

          I wonder if there is a chance that somobody will look into this. Clearly, this
          is not a show-stopper, but after one year waiting it would be nice at least to
          know if anything is going to happen

          javanet_ac added a comment - I wonder if there is a chance that somobody will look into this. Clearly, this is not a show-stopper, but after one year waiting it would be nice at least to know if anything is going to happen

          marcingalazka added a comment -

          Same issue when using emma-maven-plugin and running 'package site'. According
          to FAQ (http://mojo.codehaus.org/emma-maven-plugin/faq.html) tests should be
          run twice.

          I suppose that adding an ability to restrict phases/goals when test results are
          recorded would be solution, albeit inelegant.

          marcingalazka added a comment - Same issue when using emma-maven-plugin and running 'package site'. According to FAQ ( http://mojo.codehaus.org/emma-maven-plugin/faq.html ) tests should be run twice. I suppose that adding an ability to restrict phases/goals when test results are recorded would be solution, albeit inelegant.

          There I was going to go and write a big blog entry and somebody quotes the FAQ
          entry I wrote back to me (saving me the bother of writing the blog entry)

          In any case... this is a CI server, run the damn tests twice it's best for you...

          and since you are running them twice, you should get both sets of results.

          Also FYI, the lifecycle is forked to run the instrumented tests so that the
          result is the tests are still run in the same phase... filtering by phase will
          not help you.

          SERIOUSLY I have seen too many tests where they pass with coverage and hide a
          real bug than only shows when run without coverage....

          unless ALL your developers are much smarter than me (and I'm not saying I'm
          smart) this is what you should do... if you have at least one developer as smart
          as me or less smart than me you must run the tests twice...

          Read "Java Concurrency in practice" the one with the three high-speed trains
          (TGV or japanese bullet trains I cannot remember) there is an example on page
          33. If all your developers can correctly identify what that simple program is
          allowed to do under the JVM spec, then and only then will I accept that you
          might be ok in only running the tests with coverage and not without.

          Stephen Connolly added a comment - There I was going to go and write a big blog entry and somebody quotes the FAQ entry I wrote back to me (saving me the bother of writing the blog entry) In any case... this is a CI server, run the damn tests twice it's best for you... and since you are running them twice, you should get both sets of results. Also FYI, the lifecycle is forked to run the instrumented tests so that the result is the tests are still run in the same phase... filtering by phase will not help you. SERIOUSLY I have seen too many tests where they pass with coverage and hide a real bug than only shows when run without coverage.... unless ALL your developers are much smarter than me (and I'm not saying I'm smart) this is what you should do... if you have at least one developer as smart as me or less smart than me you must run the tests twice... Read "Java Concurrency in practice" the one with the three high-speed trains (TGV or japanese bullet trains I cannot remember) there is an example on page 33. If all your developers can correctly identify what that simple program is allowed to do under the JVM spec, then and only then will I accept that you might be ok in only running the tests with coverage and not without.

          marcingalazka added a comment -

          Ha! I didn't notice, that you are emma-maven-plugin author, Stephen.

          From my point of view - the second (instrumented) pass should give me a code
          coverage info. I assume that number of succeded/failed tests should be the same
          as in first pass. If not, this Emma/Cobertura/Clover bug, not mine. Yep, warn
          me, if they are different, but doubling tests count is misleading.

          Since I'm not into M2 internals I do not know if fixing this bug is possible/
          feasible or not. Not a big deal anyway, I'm just little pedantic .

          marcingalazka added a comment - Ha! I didn't notice, that you are emma-maven-plugin author, Stephen. From my point of view - the second (instrumented) pass should give me a code coverage info. I assume that number of succeded/failed tests should be the same as in first pass. If not, this Emma/Cobertura/Clover bug, not mine. Yep, warn me, if they are different, but doubling tests count is misleading. Since I'm not into M2 internals I do not know if fixing this bug is possible/ feasible or not. Not a big deal anyway, I'm just little pedantic .

          I did not write the emma-maven-plugin, I just ran the release process for alpha-1 which involved tidy
          up and writing docs

          the point is that if a test fails when run with coverage,90% of the time you need to sit up and take
          notice because it is most likely your fault too

          the modifications made by instrumentation are all legal under the jvm spec. if your code fails in the
          presence of such modifications, that is a bug in your code. just as much as if it won't pass without
          them

          Stephen Connolly added a comment - I did not write the emma-maven-plugin, I just ran the release process for alpha-1 which involved tidy up and writing docs the point is that if a test fails when run with coverage,90% of the time you need to sit up and take notice because it is most likely your fault too the modifications made by instrumentation are all legal under the jvm spec. if your code fails in the presence of such modifications, that is a bug in your code. just as much as if it won't pass without them

            olamy Olivier Lamy
            paulamitchell paulamitchell
            Votes:
            38 Vote for this issue
            Watchers:
            22 Start watching this issue

              Created:
              Updated:
              Resolved: