Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-15570

Coverage report includes classes that have been excluded from Jacoco analysis

      In my Maven config there are a number of classes that are excluded from Jacoco analysis. The report generated by Jacoco does not include these classes. The report generated by the Jenkins Jacoco plugin does includes these classes which gives a misleading picture and makes it difficult to set any coverage thresholds. I think this may be related to JENKINS-14975.

          [JENKINS-15570] Coverage report includes classes that have been excluded from Jacoco analysis

          Ognjen Bubalo added a comment -

          Hi,

          Please give us more information.

          Which plugin version do you use?
          What is your current configuration on the configuration page of JaCoCo plugin?

          Cheers,
          Ogi

          Ognjen Bubalo added a comment - Hi, Please give us more information. Which plugin version do you use? What is your current configuration on the configuration page of JaCoCo plugin? Cheers, Ogi

          Rune Flobakk added a comment -

          I am guessing this is related to classes which are generated, e.g. from WSDLs. A common use case is to generate Java source files which end up in target/generated-sources/**. This folder then becomes part of the source folder of the project, and compiled as usual into target/classes and packaged as usual. But these classes should not be part of the coverage report.

          In my case, I have a Maven module which only purpose is to generate such classes and package them into a jar. The module does not even have a src-folder because everything is generated from WSDLs/XSDs. The jacoco-maven-plugin correctly does not produce any jacoco.exec file for this module, but it seems like the Jenkins plugin sees the output classes, and no jacoco.exec, and interprets that as zero coverage.

          Rune Flobakk added a comment - I am guessing this is related to classes which are generated, e.g. from WSDLs. A common use case is to generate Java source files which end up in target/generated-sources/**. This folder then becomes part of the source folder of the project, and compiled as usual into target/classes and packaged as usual. But these classes should not be part of the coverage report. In my case, I have a Maven module which only purpose is to generate such classes and package them into a jar. The module does not even have a src-folder because everything is generated from WSDLs/XSDs. The jacoco-maven-plugin correctly does not produce any jacoco.exec file for this module, but it seems like the Jenkins plugin sees the output classes, and no jacoco.exec, and interprets that as zero coverage.

          David Barri added a comment -

          I'm having this issue too. The relevant part of my POM looks like this:

          <plugin>
          	<groupId>org.jacoco</groupId>
          	<artifactId>jacoco-maven-plugin</artifactId>
          	<version>0.6.0.201210061924</version>
          	<executions>
          		<execution>
          			<id>coverage-agent</id>
          			<phase>process-test-classes</phase>
          			<goals>
          				<goal>prepare-agent</goal>
          			</goals>
          		</execution>
          		<execution>
          			<id>coverage-report</id>
          			<phase>test</phase>
          			<goals>
          				<goal>report</goal>
          			</goals>
          		</execution>
          	</executions>
          	<configuration>
          		<excludes>
          			<!-- Exclude classes generated by AndroidAnnotations -->
          			<exclude>**/*_.*</exclude>
          			<exclude>**/*_$*</exclude>
          		</excludes>
          	</configuration>
          </plugin>
          

          When I generate a Jacoco HTML report, classes that end in an underscore like ExampleActivity_ do not appear because of the exclude settings in my POM. However from Jenkins all of those classes that the Maven build excludes are visible again.

          My Jenkins settings for this plugin are the defaults.

          Thanks!
          David

          David Barri added a comment - I'm having this issue too. The relevant part of my POM looks like this: <plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>0.6.0.201210061924</version> <executions> <execution> <id>coverage-agent</id> <phase>process-test-classes</phase> <goals> <goal>prepare-agent</goal> </goals> </execution> <execution> <id>coverage-report</id> <phase>test</phase> <goals> <goal>report</goal> </goals> </execution> </executions> <configuration> <excludes> <!-- Exclude classes generated by AndroidAnnotations --> <exclude>**/*_.*</exclude> <exclude>**/*_$*</exclude> </excludes> </configuration> </plugin> When I generate a Jacoco HTML report, classes that end in an underscore like ExampleActivity_ do not appear because of the exclude settings in my POM. However from Jenkins all of those classes that the Maven build excludes are visible again. My Jenkins settings for this plugin are the defaults. Thanks! David

          Ognjen Bubalo added a comment -

          Is this issue still actual?

          If yes, please show your configuration (given paths) of JaCoCo.

          Thanks,
          ogi

          Ognjen Bubalo added a comment - Is this issue still actual? If yes, please show your configuration (given paths) of JaCoCo. Thanks, ogi

          Thomas Zlika added a comment -

          Yes, the problem is still present with the latest 1.0.9 version.
          I have some packages excluded from code coverage using the "excludes" parameter of the jacoco-maven-plugin. The HTML page generated by this plugin are correct (the excludes packages are not taken into account). However, the Jenkins plugin take them into account and display a code coverage of 0%.
          The configuration for the Jenkins plugin is the standard one:

          • Path to exec files: **/target/jacoco.exec
          • Path to class dir: **/target/classes
            Path to source dir: **/src/main/java

          Thomas Zlika added a comment - Yes, the problem is still present with the latest 1.0.9 version. I have some packages excluded from code coverage using the "excludes" parameter of the jacoco-maven-plugin. The HTML page generated by this plugin are correct (the excludes packages are not taken into account). However, the Jenkins plugin take them into account and display a code coverage of 0%. The configuration for the Jenkins plugin is the standard one: Path to exec files: **/target/jacoco.exec Path to class dir: **/target/classes Path to source dir: **/src/main/java

          Mirko Friedenhagen added a comment - - edited

          Same here, I exclude the generated HelpMojo:

          Maven only needs the exclusion for the report mojo, though. jacoco's version is: 0.6.1.201212231917.
          Maybe the Jenkins-Plugin should just drop all .class files where no .java files are found?

          Mirko Friedenhagen added a comment - - edited Same here, I exclude the generated HelpMojo : in build https://github.com/1and1/bill-of-materials-maven-plugin/blob/a7c20afe67620df3c3be0458ddd23a596398af33/pom.xml#L134 and reporting https://github.com/1and1/bill-of-materials-maven-plugin/blob/a7c20afe67620df3c3be0458ddd23a596398af33/pom.xml#L166 Maven only needs the exclusion for the report mojo, though. jacoco 's version is: 0.6.1.201212231917. Maybe the Jenkins-Plugin should just drop all .class files where no .java files are found?

          Ognjen Bubalo added a comment -

          Hi,

          Hopefully I'll implement this feature and release a new version in the next 2 days.

          Cheers,
          Ogi

          Ognjen Bubalo added a comment - Hi, Hopefully I'll implement this feature and release a new version in the next 2 days. Cheers, Ogi

          Ognjen Bubalo added a comment -

          It will be available in the next release.

          Ognjen Bubalo added a comment - It will be available in the next release.

          Ognjen Bubalo added a comment -

          Ognjen Bubalo added a comment - Check this: https://groups.google.com/d/topic/jenkins-jacoco-plugin-mailing-list/LCWg0ukJ_wg/discussion

          In version of plugin 10.0.16 and Jenkins 1.583 it is not working again.
          I configured pom for exclusion one class from report, but Jenkins still compute with this class.

          Libor Ondrusek added a comment - In version of plugin 10.0.16 and Jenkins 1.583 it is not working again. I configured pom for exclusion one class from report, but Jenkins still compute with this class.

          Akos Kozak added a comment - - edited

          Still not working 100%-ly with plugin version 1.0.16 and Jenkins 1.563.

          I made a smaller maven project where I got api and be modules.

          I set:

          pom.xml
                                      <execution>
                                          <id>check-coverage</id>
                                          <goals>
                                              <goal>check</goal>
                                          </goals>
                                          <configuration>
                                              <rules>
                                                  <rule>
                                                      <element>BUNDLE</element>
                                                      <excludes>
                                                          <exclude>*api*</exclude>
                                                      </excludes>
                                                      <limits>
                                                          <limit>
                                                              <counter>LINE</counter>
                                                              <value>COVEREDRATIO</value>
                                                              <minimum>0.25</minimum>
                                                          </limit>
                                                      </limits>
                                                  </rule>
                                              </rules>
                                          </configuration>
                                      </execution>
          

          And the plugin works well. No api is visible in the coverage.

          But if I do the same on our large project, if from one module Test a call will be done on an api, and therefor in the be classes this will be visible. However I would like to exclude api.

          Module A
          A-api: Interface to A service
          A-be : Service A implementation

          Module B
          B-api: Interface to B service
          B-be: Service B implementation

          From Service B implementation we call a method over A-api Interface to A service implementation. Therefore, the B-api will be excluded, but the A-api not! This is a bit disturbing, because the A-api should be also excluded.

          I hope I have right, and it is understandable what I wrote!

          Akos Kozak added a comment - - edited Still not working 100%-ly with plugin version 1.0.16 and Jenkins 1.563. I made a smaller maven project where I got api and be modules. I set: pom.xml <execution> <id>check-coverage</id> <goals> <goal>check</goal> </goals> <configuration> <rules> <rule> <element>BUNDLE</element> <excludes> <exclude>*api*</exclude> </excludes> <limits> <limit> <counter>LINE</counter> <value>COVEREDRATIO</value> <minimum>0.25</minimum> </limit> </limits> </rule> </rules> </configuration> </execution> And the plugin works well. No api is visible in the coverage. But if I do the same on our large project, if from one module Test a call will be done on an api, and therefor in the be classes this will be visible. However I would like to exclude api. Module A A-api: Interface to A service A-be : Service A implementation Module B B-api: Interface to B service B-be: Service B implementation From Service B implementation we call a method over A-api Interface to A service implementation. Therefore, the B-api will be excluded, but the A-api not! This is a bit disturbing, because the A-api should be also excluded. I hope I have right, and it is understandable what I wrote!

          C. S. added a comment -

          We ran into the same issue. It is important for us to get it working, because we have a lot of generated classes, which should be excluded and it would be a mess to set up each Jenkins job according the new classes.

          So we decided to exclude the classes from the report via pom. But it seems that the jacoco plugin does its own match from classes to exec file.

          C. S. added a comment - We ran into the same issue. It is important for us to get it working, because we have a lot of generated classes, which should be excluded and it would be a mess to set up each Jenkins job according the new classes. So we decided to exclude the classes from the report via pom. But it seems that the jacoco plugin does its own match from classes to exec file.

          Raj Bhaskar added a comment -

          I think I'm having the same issue. I package the Oracle JDBC driver as a local jar in my application (as it's not on maven) but it looks like although I'm excluding it in my pom, the plugin is still including it when doing its analysis.

          Raj Bhaskar added a comment - I think I'm having the same issue. I package the Oracle JDBC driver as a local jar in my application (as it's not on maven) but it looks like although I'm excluding it in my pom, the plugin is still including it when doing its analysis.

          Steve Powell added a comment -

          This is an issue for us also. In a maven build with the jacoco plugin we exclude all the classes in a package; and jacoco reports in the console log that these have been excluded.

          In the jacoco check phase, no further inclusions/exclusions are supplied but thresholds are provided.

          The coverage report given by Jenkins seems to include those classes, the coverage percentages (package, line and so on) remaining the same as without exclusion. The threshold checks (in particular the line thresholds) still apply and make the build fail.

          The classes we exclude are generated into generated-sources by a third party (immutables.org) which we do not wish to test.

          Steve Powell added a comment - This is an issue for us also. In a maven build with the jacoco plugin we exclude all the classes in a package; and jacoco reports in the console log that these have been excluded. In the jacoco check phase, no further inclusions/exclusions are supplied but thresholds are provided. The coverage report given by Jenkins seems to include those classes, the coverage percentages (package, line and so on) remaining the same as without exclusion. The threshold checks (in particular the line thresholds) still apply and make the build fail. The classes we exclude are generated into generated-sources by a third party (immutables.org) which we do not wish to test.

          Antoine Tran added a comment -

          We have this issue too. Right now, we need to duplicate exclusion information from pom.xml into this Jenkins Jacoco plugin. This issue is not solved yet.

          Antoine Tran added a comment - We have this issue too. Right now, we need to duplicate exclusion information from pom.xml into this Jenkins Jacoco plugin. This issue is not solved yet.

          twolfart added a comment -

          Confirmed on Jenkins 2.64, hudson.plugins.jacoco.JacocoPublisher 2.2.0.

          The code is generated to the target directory and further used in the build.
          The pom.xml contains this configuration:

                <plugin>
                  <groupId>org.jacoco</groupId>
                  <artifactId>jacoco-maven-plugin</artifactId>
                  <configuration>
                    <excludes>
                      <!-- exclude generated code from coverage report -->
                      <exclude>**/com/foo/bar/*</exclude>
                    </excludes>
                  </configuration>
                </plugin>
          

          The maven-generated JaCoCo HTML report doesn't include the classes, Jenkins includes them. Adding the ignores to the jenkins-plugin removes them as expected, but is not very convenient (two places required for configuration).

          twolfart added a comment - Confirmed on Jenkins 2.64, hudson.plugins.jacoco.JacocoPublisher 2.2.0. The code is generated to the target directory and further used in the build. The pom.xml contains this configuration: <plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <configuration> <excludes> <!-- exclude generated code from coverage report --> <exclude>**/com/foo/bar/*</exclude> </excludes> </configuration> </plugin> The maven-generated JaCoCo HTML report doesn't include the classes, Jenkins includes them. Adding the ignores to the jenkins-plugin removes them as expected, but is not very convenient (two places required for configuration).

          Jae Gangemi added a comment -

          is there any progress on resolving this? i still see this behavior w/ jenkins 2.70 and version 2.2.1 of the plugin.

          Jae Gangemi added a comment - is there any progress on resolving this? i still see this behavior w/ jenkins 2.70 and version 2.2.1 of the plugin.

          René Scheibe added a comment -

          A workaround is to configure the exclusions in the Jenkins Jacoco-Plugin too.

          René Scheibe added a comment - A workaround is to configure the exclusions in the Jenkins Jacoco-Plugin too.

          Jae Gangemi added a comment -

          is there any futher movement on this issue? configuring exclusions in maven and jenkins is not a viable work around.

          Jae Gangemi added a comment - is there any futher movement on this issue? configuring exclusions in maven and jenkins is not a viable work around.

          Alex Taylor added a comment -

          I was looking into this issue and it appears that the Jacoco plugin fixed this issue for ENV variables with https://github.com/jenkinsci/jacoco-plugin/pull/74 but it does not actually parse the pom.xml for the `excludes`(or `includes` for that matter) configuration. Does that sound like the actual issue here rather than the Excludes not working at all?

           

          Did I miss anything here? I was going to work on getting this fixed with a PR hopefully within the next week

          Alex Taylor added a comment - I was looking into this issue and it appears that the Jacoco plugin fixed this issue for ENV variables with https://github.com/jenkinsci/jacoco-plugin/pull/74  but it does not actually parse the pom.xml for the `excludes`(or `includes` for that matter) configuration. Does that sound like the actual issue here rather than the Excludes not working at all?   Did I miss anything here? I was going to work on getting this fixed with a PR hopefully within the next week

          Jae Gangemi added a comment -

          i'm confused as to why this information just isn't part of the generated results file in the first place as a block that indicates what to exclude. perhaps even better would just be excluding the classes from the result file outright. disclaimer: i have not looked into how the maven plugin works when i choose to generate a report, but it does do the proper exclusions.

          if the plugin needs to parse the pom to figure this out, then that is the fix but that doesn't really seem like the correct solution here b/c what if i'm a nodejs project that is able to generate coverage reports in jacoco format? there's not going to be any pom to parse for exclusions in that case.

          Jae Gangemi added a comment - i'm confused as to why this information just isn't part of the generated results file in the first place as a block that indicates what to exclude. perhaps even better would just be excluding the classes from the result file outright. disclaimer: i have not looked into how the maven plugin works when i choose to generate a report, but it does do the proper exclusions. if the plugin needs to parse the pom to figure this out, then that is the fix but that doesn't really seem like the correct solution here b/c what if i'm a nodejs project that is able to generate coverage reports in jacoco format? there's not going to be any pom to parse for exclusions in that case.

          ataylor , what you've described, regarding the exclude block in the maven plugin not taking affect in Jenkins, is the issue I've seen and am interested in a fix for, and seems to align with the initial description (which is specific to maven based projects).

          Gabriel Stanek added a comment - ataylor , what you've described, regarding the exclude block in the maven plugin not taking affect in Jenkins, is the issue I've seen and am interested in a fix for, and seems to align with the initial description (which is specific to maven based projects).

          Alex Taylor added a comment -

          Thanks for the update gstanek just wanted make sure I was understanding everyone's issue and how best I can fix it

          Alex Taylor added a comment - Thanks for the update gstanek just wanted make sure I was understanding everyone's issue and how best I can fix it

          really looking forward to this fix! lots of anxiety around this bug

          Alan Czajkowski added a comment - really looking forward to this fix! lots of anxiety around this bug

          Rajiv Jandial added a comment -

          I am still seeing this as an open issue. Even I exclude the package from the pom.xml as well as from the Jenkins "exclusions" fields, those packages are still being included in the JaCoCo report.

          Rajiv Jandial added a comment - I am still seeing this as an open issue. Even I exclude the package from the pom.xml as well as from the Jenkins "exclusions" fields, those packages are still being included in the JaCoCo report.

          Pablo Grisafi added a comment -

          my workaround: in the pom.xml I save all patterns  as properties:

          <coverage-exclusion-01>*/com/company/product//config//.</coverage-exclusion-01><coverage-exclusion-02>/com/company/cfa//CallDTO.</coverage-exclusion-02><coverage-exclusion-03>*/com/company/cfa/*/DAO.</coverage-exclusion-03>

          And I use them to configure the maven jacoco plugin

          <configuration> <excludes> <exclude>${coverage-exclusion-01}</exclude> <exclude>${coverage-exclusion-02}</exclude>

          etc...

           

          And then, in the pipeline file y read again the properties

           

          script{      props = readMavenPom().getProperties()     exclusionPattern = props.entrySet().findAll\{entry -> entry.key.startsWith('coverage-exclusion-')}.collect{it.value}.join(',')
              echo "exclusionPattern = ${exclusionPattern}"
          }
          jacoco( execPattern: '**/target/jacoco.exec', exclusionPattern: exclusionPattern)

           

          It works, but is ugly

           

           

          Pablo Grisafi added a comment - my workaround: in the pom.xml I save all patterns  as properties: < coverage-exclusion-01 > * /com/company/product/ /config/ / . </ coverage-exclusion-01 > < coverage-exclusion-02 > /com/company/cfa/ /CallDTO. </ coverage-exclusion-02 > < coverage-exclusion-03 > * /com/company/cfa/ */ DAO. </ coverage-exclusion-03 > And I use them to configure the maven jacoco plugin < configuration > < excludes > < exclude > ${coverage-exclusion-01} </ exclude > < exclude > ${coverage-exclusion-02} </ exclude > etc...   And then, in the pipeline file y read again the properties   script{      props = readMavenPom().getProperties()     exclusionPattern = props.entrySet().findAll\{entry -> entry.key.startsWith('coverage-exclusion-')}.collect{it.value}.join(',')     echo "exclusionPattern = ${exclusionPattern}" } jacoco( execPattern: '**/target/jacoco.exec', exclusionPattern: exclusionPattern)   It works, but is ugly    

          Ariel M added a comment -

          We are also facing the same issue. it's weird coz' it's working for one MS, but not for the other. is there any update for it?

          Ariel M added a comment - We are also facing the same issue. it's weird coz' it's working for one MS, but not for the other. is there any update for it?

            ognjenb Ognjen Bubalo
            daknin Darrell King
            Votes:
            23 Vote for this issue
            Watchers:
            29 Start watching this issue

              Created:
              Updated: