Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22018

Support for ratcheting of minimum thresholds in Jacoco coverage

    • Icon: New Feature New Feature
    • Resolution: Unresolved
    • Icon: Major Major
    • jacoco-plugin
    • None

      The addition of an option to "ratchet" or update the minimum thresholds for build failure would be very useful. This request is inspired by and heavily related to JENKINS-8326 for the Cobertura coverage plugin.

      Ideally we'd be able to set the publisher to automatically update the minimum thresholds to higher values when a project's observed value is higher than a previously set minimum. This enables a project to help enforce that code coverage cannot decrease over time as a code quality standard.

          [JENKINS-22018] Support for ratcheting of minimum thresholds in Jacoco coverage

          Because I've had to work around this myself in the meantime, I use the following groovy postbuild script to achieve this:

          // Grab the API for Jacoco
          def b = manager.build;
          
          def action = b.getActions().find { it.getUrlName() == "jacoco" }
          if (action == null) {
          	// no Jacoco coverage, so nothing to do!
          	manager.listener.logger.println "Unable to get Jacoco Build Action on build, so bailing out"
          	return;
          }
          
          def publishers = b.getProject().publishersList;
          def jacocoPublisher = publishers.find { it.getDescriptor().getId() == "hudson.plugins.jacoco.JacocoPublisher" }
          
          // Collect the current "observed" values for coverage
          def line = action.getLineCoverage();
          def clazz = action.getClassCoverage();
          def method = action.getMethodCoverage();
          def instruction = action.getInstructionCoverage();
          def branch = action.getBranchCoverage();
          //def complexity = action.getComplexityScore();
          
          // Now let's set the new threshold minimums to last value!
          // Only set these values if the observed percent is _higher_ than the existing minimum threshold!
          def thresholds = action.getThresholds();
          
          // Classes
          int percent = clazz.getPercentage();
          manager.listener.logger.println "Observed class coverage: ${percent}%"
          if (percent > thresholds.getMinClass()) {
          	manager.listener.logger.println "Increasing minimum threshold to observed value for class coverage"
          	thresholds.setMinClass(percent);
          }
          // methods
          percent = method.getPercentage();
          manager.listener.logger.println "Observed methods coverage: ${percent}%"
          if (percent > thresholds.getMinMethod()) {
          	manager.listener.logger.println "Increasing minimum threshold to observed value for methods coverage"
          	thresholds.setMinMethod(percent);
          }
          // Branches
          percent = branch.getPercentage();
          manager.listener.logger.println "Observed branches coverage: ${percent}%"
          if (percent > thresholds.getMinBranch()) {
          	manager.listener.logger.println "Increasing minimum threshold to observed value for branches coverage"
          	thresholds.setMinBranch(percent);
          }
          // Lines
          percent = line.getPercentage();
          manager.listener.logger.println "Observed lines coverage: ${percent}%"
          if (percent > thresholds.getMinLine()) {
          	manager.listener.logger.println "Increasing minimum threshold to observed value for lines coverage"
          	thresholds.setMinLine(percent);
          }
          // Instructions
          percent = instruction.getPercentage();
          manager.listener.logger.println "Observed instruction coverage: ${percent}%"
          if (percent > thresholds.getMinInstruction()) {
          	manager.listener.logger.println "Increasing minimum threshold to observed value for instruction coverage"
          	thresholds.setMinInstruction(percent);
          }
          // Complexity
          //percent = complexity.getPercentage();
          //if (percent > thresholds.getMinComplexity()) {
          //	thresholds.setMinComplexity(percent);
          //}
          
          // We need to replace the publisher with a new instance that has updated minimums
          // HACK to work around bug in plugin where we don't have access to plugin classes.
          def c = Class.forName("hudson.plugins.jacoco.JacocoPublisher", true, manager.hudson.getPluginManager().uberClassLoader)
          def constructor = c.getConstructor( [ String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, boolean ] as Class[] )
          // make a new version of the coverage publisher with updated minimums
          def newPub = constructor.newInstance(jacocoPublisher.getExecPattern(), jacocoPublisher.getClassPattern(), jacocoPublisher.getSourcePattern(),
           jacocoPublisher.getInclusionPattern(), jacocoPublisher.getExclusionPattern(), jacocoPublisher.getMaximumInstructionCoverage(),
           jacocoPublisher.getMaximumBranchCoverage(), jacocoPublisher.getMaximumComplexityCoverage(), jacocoPublisher.getMaximumLineCoverage(),
           jacocoPublisher.getMaximumMethodCoverage(), jacocoPublisher.getMaximumClassCoverage(), Integer.toString(thresholds.getMinInstruction()), Integer.toString(thresholds.getMinBranch()),
           Integer.toString(thresholds.getMinComplexity()), Integer.toString(thresholds.getMinLine()), Integer.toString(thresholds.getMinMethod()), Integer.toString(thresholds.getMinClass()),
           jacocoPublisher.isChangeBuildStatus())
          // Now remove the old version and replace with the new publisher with updated minimums
          publishers.replace(jacocoPublisher, newPub); // remove old and replace with new
          b.getProject().save(); // persist the change
          

          Chris Williams added a comment - Because I've had to work around this myself in the meantime, I use the following groovy postbuild script to achieve this: // Grab the API for Jacoco def b = manager.build; def action = b.getActions().find { it.getUrlName() == "jacoco" } if (action == null ) { // no Jacoco coverage, so nothing to do ! manager.listener.logger.println "Unable to get Jacoco Build Action on build, so bailing out" return ; } def publishers = b.getProject().publishersList; def jacocoPublisher = publishers.find { it.getDescriptor().getId() == "hudson.plugins.jacoco.JacocoPublisher" } // Collect the current "observed" values for coverage def line = action.getLineCoverage(); def clazz = action.getClassCoverage(); def method = action.getMethodCoverage(); def instruction = action.getInstructionCoverage(); def branch = action.getBranchCoverage(); // def complexity = action.getComplexityScore(); // Now let's set the new threshold minimums to last value! // Only set these values if the observed percent is _higher_ than the existing minimum threshold! def thresholds = action.getThresholds(); // Classes int percent = clazz.getPercentage(); manager.listener.logger.println "Observed class coverage: ${percent}%" if (percent > thresholds.getMinClass()) { manager.listener.logger.println "Increasing minimum threshold to observed value for class coverage" thresholds.setMinClass(percent); } // methods percent = method.getPercentage(); manager.listener.logger.println "Observed methods coverage: ${percent}%" if (percent > thresholds.getMinMethod()) { manager.listener.logger.println "Increasing minimum threshold to observed value for methods coverage" thresholds.setMinMethod(percent); } // Branches percent = branch.getPercentage(); manager.listener.logger.println "Observed branches coverage: ${percent}%" if (percent > thresholds.getMinBranch()) { manager.listener.logger.println "Increasing minimum threshold to observed value for branches coverage" thresholds.setMinBranch(percent); } // Lines percent = line.getPercentage(); manager.listener.logger.println "Observed lines coverage: ${percent}%" if (percent > thresholds.getMinLine()) { manager.listener.logger.println "Increasing minimum threshold to observed value for lines coverage" thresholds.setMinLine(percent); } // Instructions percent = instruction.getPercentage(); manager.listener.logger.println "Observed instruction coverage: ${percent}%" if (percent > thresholds.getMinInstruction()) { manager.listener.logger.println "Increasing minimum threshold to observed value for instruction coverage" thresholds.setMinInstruction(percent); } // Complexity //percent = complexity.getPercentage(); // if (percent > thresholds.getMinComplexity()) { // thresholds.setMinComplexity(percent); //} // We need to replace the publisher with a new instance that has updated minimums // HACK to work around bug in plugin where we don't have access to plugin classes. def c = Class .forName( "hudson.plugins.jacoco.JacocoPublisher" , true , manager.hudson.getPluginManager().uberClassLoader) def constructor = c.getConstructor( [ String , String , String , String , String , String , String , String , String , String , String , String , String , String , String , String , String , boolean ] as Class [] ) // make a new version of the coverage publisher with updated minimums def newPub = constructor.newInstance(jacocoPublisher.getExecPattern(), jacocoPublisher.getClassPattern(), jacocoPublisher.getSourcePattern(), jacocoPublisher.getInclusionPattern(), jacocoPublisher.getExclusionPattern(), jacocoPublisher.getMaximumInstructionCoverage(), jacocoPublisher.getMaximumBranchCoverage(), jacocoPublisher.getMaximumComplexityCoverage(), jacocoPublisher.getMaximumLineCoverage(), jacocoPublisher.getMaximumMethodCoverage(), jacocoPublisher.getMaximumClassCoverage(), Integer .toString(thresholds.getMinInstruction()), Integer .toString(thresholds.getMinBranch()), Integer .toString(thresholds.getMinComplexity()), Integer .toString(thresholds.getMinLine()), Integer .toString(thresholds.getMinMethod()), Integer .toString(thresholds.getMinClass()), jacocoPublisher.isChangeBuildStatus()) // Now remove the old version and replace with the new publisher with updated minimums publishers.replace(jacocoPublisher, newPub); // remove old and replace with new b.getProject().save(); // persist the change

          Seán Dunne added a comment -

          I'd like to add my voice to this issue.
          IMO being able to enforce that unit test coverage does not regress is a key feature of a coverage plugin.

          Would love to see it implemented!

          Seán Dunne added a comment - I'd like to add my voice to this issue. IMO being able to enforce that unit test coverage does not regress is a key feature of a coverage plugin. Would love to see it implemented!

            ognjenb Ognjen Bubalo
            seadub Chris Williams
            Votes:
            6 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated: