Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64487

BFA Prometheus Metrics

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Hi,

      I am currently setting up the BFA plugin in my Jenkins instance and I really want to get some nice visualizations of how often certain build failures occur. I understand that you do not want to maintain the built-in graph visualizations.

      So I have another proposal: How about supporting the Prometheus plugin by providing a Collector implementation?
      Then people can generate graphs out of the metrics however they please (for example with Grafana) and even trigger alerts based on the data.

      I intend to implement this functionality myself including tests. My initial research shows that I might have to reintroduce the getStatistics method that was removed here.
      Would that be ok and would such a feature have a realistic chance of being accepted into the plugin?

        Attachments

          Activity

          Hide
          organised_chaos James Robson added a comment -

          I've created a basic integration with the metrics plugin, you can see it at https://github.com/jenkinsci/build-failure-analyzer-plugin/pull/141
          These will be exported by the prometheus plugin and you can use the following config to get metrics and labels the way prometheus expects :

          metric_relabel_configs:
            - source_labels: [__name__]
              regex: 'jenkins_bfa_category_(.*)'
              target_label: 'category'
            - source_labels: [__name__]
              regex: 'jenkins_bfa_cause_(.*)'
              target_label: 'cause'
            - source_labels: [__name__]
              regex: 'jenkins_bfa_(.*)_(.*)'
              replacement: 'jenkins_bfa'
              target_label: __name__ 
          
          

          This results in a metrics like:

          jenkins_bfa{category="my_category_name"}
          jenkins_bfa{cause="my_cause_name"} 
          Show
          organised_chaos James Robson added a comment - I've created a basic integration with the metrics plugin, you can see it at https://github.com/jenkinsci/build-failure-analyzer-plugin/pull/141 These will be exported by the prometheus plugin and you can use the following config to get metrics and labels the way prometheus expects : metric_relabel_configs: - source_labels: [__name__] regex: 'jenkins_bfa_category_(.*)' target_label: 'category' - source_labels: [__name__] regex: 'jenkins_bfa_cause_(.*)' target_label: 'cause' - source_labels: [__name__] regex: 'jenkins_bfa_(.*)_(.*)' replacement: 'jenkins_bfa' target_label: __name__ This results in a metrics like: jenkins_bfa{category= "my_category_name" } jenkins_bfa{cause= "my_cause_name" }
          Hide
          paybas Pay Bas added a comment -

          Philipp I'm very interested in this as well. Do you have anything yet? We could start with creating a separate mini-plugin. I did this for the Lockable Resources plugin as well. That might make the road to an eventual integration into BFA easier.

          This is the entire plugin.

          pom.xml

          <?xml version="1.0" encoding="UTF-8"?>
          <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
            <modelVersion>4.0.0</modelVersion>
            <parent>
              <groupId>org.jenkins-ci.plugins</groupId>
              <artifactId>plugin</artifactId>
              <version>4.18</version>
              <relativePath />
            </parent>
          
            <groupId>com.mycorp.jenkins</groupId>
            <artifactId>prometheus-lockable-resources</artifactId>
            <version>1.0.0</version>
            <packaging>hpi</packaging>
            <name>Prometheus Lockable Resources plugin</name>
            <description>Expose Lockable Resources metrics in prometheus format.</description>
          
            <properties>
              <jenkins.version>2.277.3</jenkins.version>
              <java.level>8</java.level>
            </properties>
          
            <dependencies>
              <dependency>
                <groupId>org.jenkins-ci.plugins</groupId>
                <artifactId>prometheus</artifactId>
                <version>2.0.10</version>
              </dependency>
              <dependency>
                <groupId>org.6wind.jenkins</groupId>
                <artifactId>lockable-resources</artifactId>
                <version>2.10</version>
              </dependency>
            </dependencies>
          
          </project>

          LockableResourcesCollector.java

          package com.mycorp.jenkins;
          
          import hudson.Extension;
          import io.prometheus.client.Collector;
          import io.prometheus.client.Gauge;
          import jenkins.model.Jenkins;
          import org.jenkins.plugins.lockableresources.LockableResource;
          import org.jenkins.plugins.lockableresources.LockableResourcesManager;
          import org.jenkinsci.plugins.prometheus.util.ConfigurationUtils;
          import org.slf4j.Logger;
          import org.slf4j.LoggerFactory;
          
          import javax.annotation.Nonnull;
          import java.util.ArrayList;
          import java.util.List;
          import java.util.Set;
          
          @Extension
          public class LockableResourcesCollector extends Collector {
          
              private static final Logger logger = LoggerFactory.getLogger(LockableResourcesCollector.class);
          
              @Override
              @Nonnull
              public List<MetricFamilySamples> collect() {
                  List<MetricFamilySamples> samples = new ArrayList<>();
          
                  logger.debug("Collecting lockable resource metrics for prometheus");
                  String namespace = ConfigurationUtils.getNamespace();
                  String subsystem = ConfigurationUtils.getSubSystem();
                  String prefix = "lockable_resources";
                  String[] labelNameArray = {"label"};
          
                  Gauge resourcesDefined = Gauge.build()
                          .namespace(namespace)
                          .subsystem(subsystem)
                          .name(prefix + "_defined")
                          .labelNames(labelNameArray)
                          .help("Lockable resources defined")
                          .create();
                  Gauge resourcesFree = Gauge.build()
                          .namespace(namespace)
                          .subsystem(subsystem)
                          .name(prefix + "_free")
                          .labelNames(labelNameArray)
                          .help("Lockable resources free")
                          .create();
                  Gauge resourcesLocked = Gauge.build()
                          .namespace(namespace)
                          .subsystem(subsystem)
                          .name(prefix + "_locked")
                          .labelNames(labelNameArray)
                          .help("Lockable resources locked")
                          .create();
                  Gauge resourcesReserved = Gauge.build()
                          .namespace(namespace)
                          .subsystem(subsystem)
                          .name(prefix + "_reserved")
                          .labelNames(labelNameArray)
                          .help("Lockable resources reserved")
                          .create();
                  Gauge resourcesQueued = Gauge.build()
                          .namespace(namespace)
                          .subsystem(subsystem)
                          .name(prefix + "_queued")
                          .labelNames(labelNameArray)
                          .help("Lockable resources queued")
                          .create();
          
                  LockableResourcesManager lockableResourcesManager;
                  try {
                      lockableResourcesManager = Jenkins.get().getExtensionList(LockableResourcesManager.class).getInstance(LockableResourcesManager.class);
                  } catch (IllegalStateException e) {
                      logger.warn("Cannot get lockable resource data for Prometheus!");
                      return samples;
                  }
          
                  Set<String> labels = lockableResourcesManager.getAllLabels();
                  labels.add("");
                  for (String label : labels) {
                      List<LockableResource> lockableResources = lockableResourcesManager.getResourcesWithLabel(label, null);
                      if (lockableResources.isEmpty()) {
                          continue;
                      }
          
                      int defined = lockableResources.size();
                      int free = lockableResourcesManager.getFreeResourceAmount(label);
                      int locked = 0;
                      int reserved = 0;
                      int queued = 0;
          
                      for (LockableResource r : lockableResources) {
                          if (r.isReserved()) {
                              reserved++;
                          }
                          if (r.isLocked()) {
                              locked++;
                          }
                          if (r.isQueued()) {
                              queued++;
                          }
                      }
          
                      String[] labelValueArray = {label};
                      resourcesDefined.labels(labelValueArray).set(defined);
                      resourcesFree.labels(labelValueArray).set(free);
                      resourcesLocked.labels(labelValueArray).set(locked);
                      resourcesReserved.labels(labelValueArray).set(reserved);
                      resourcesQueued.labels(labelValueArray).set(queued);
                  }
          
                  samples.addAll(resourcesDefined.collect());
                  samples.addAll(resourcesFree.collect());
                  samples.addAll(resourcesLocked.collect());
                  samples.addAll(resourcesReserved.collect());
                  samples.addAll(resourcesQueued.collect());
          
                  return samples;
              }
          }
          
          Show
          paybas Pay Bas added a comment - Philipp I'm very interested in this as well. Do you have anything yet? We could start with creating a separate mini-plugin. I did this for the Lockable Resources plugin as well. That might make the road to an eventual integration into BFA easier. This is the entire plugin. pom.xml <?xml version= "1.0" encoding= "UTF-8" ?> <project xmlns= "http: //maven.apache.org/POM/4.0.0" xmlns:xsi= "http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation= "http: //maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" > <modelVersion>4.0.0</modelVersion> <parent> <groupId>org.jenkins-ci.plugins</groupId> <artifactId>plugin</artifactId> <version>4.18</version> <relativePath /> </parent> <groupId>com.mycorp.jenkins</groupId> <artifactId>prometheus-lockable-resources</artifactId> <version>1.0.0</version> <packaging>hpi</packaging> <name>Prometheus Lockable Resources plugin</name> <description>Expose Lockable Resources metrics in prometheus format.</description> <properties> <jenkins.version>2.277.3</jenkins.version> <java.level>8</java.level> </properties> <dependencies> <dependency> <groupId>org.jenkins-ci.plugins</groupId> <artifactId>prometheus</artifactId> <version>2.0.10</version> </dependency> <dependency> <groupId>org.6wind.jenkins</groupId> <artifactId>lockable-resources</artifactId> <version>2.10</version> </dependency> </dependencies> </project> LockableResourcesCollector.java package com.mycorp.jenkins; import hudson.Extension; import io.prometheus.client.Collector; import io.prometheus.client.Gauge; import jenkins.model.Jenkins; import org.jenkins.plugins.lockableresources.LockableResource; import org.jenkins.plugins.lockableresources.LockableResourcesManager; import org.jenkinsci.plugins.prometheus.util.ConfigurationUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import javax.annotation.Nonnull; import java.util.ArrayList; import java.util.List; import java.util.Set; @Extension public class LockableResourcesCollector extends Collector { private static final Logger logger = LoggerFactory.getLogger(LockableResourcesCollector.class); @Override @Nonnull public List<MetricFamilySamples> collect() { List<MetricFamilySamples> samples = new ArrayList<>(); logger.debug( "Collecting lockable resource metrics for prometheus" ); String namespace = ConfigurationUtils.getNamespace(); String subsystem = ConfigurationUtils.getSubSystem(); String prefix = "lockable_resources" ; String [] labelNameArray = { "label" }; Gauge resourcesDefined = Gauge.build() .namespace(namespace) .subsystem(subsystem) .name(prefix + "_defined" ) .labelNames(labelNameArray) .help( "Lockable resources defined" ) .create(); Gauge resourcesFree = Gauge.build() .namespace(namespace) .subsystem(subsystem) .name(prefix + "_free" ) .labelNames(labelNameArray) .help( "Lockable resources free" ) .create(); Gauge resourcesLocked = Gauge.build() .namespace(namespace) .subsystem(subsystem) .name(prefix + "_locked" ) .labelNames(labelNameArray) .help( "Lockable resources locked" ) .create(); Gauge resourcesReserved = Gauge.build() .namespace(namespace) .subsystem(subsystem) .name(prefix + "_reserved" ) .labelNames(labelNameArray) .help( "Lockable resources reserved" ) .create(); Gauge resourcesQueued = Gauge.build() .namespace(namespace) .subsystem(subsystem) .name(prefix + "_queued" ) .labelNames(labelNameArray) .help( "Lockable resources queued" ) .create(); LockableResourcesManager lockableResourcesManager; try { lockableResourcesManager = Jenkins.get().getExtensionList(LockableResourcesManager.class).getInstance(LockableResourcesManager.class); } catch (IllegalStateException e) { logger.warn( "Cannot get lockable resource data for Prometheus!" ); return samples; } Set< String > labels = lockableResourcesManager.getAllLabels(); labels.add(""); for ( String label : labels) { List<LockableResource> lockableResources = lockableResourcesManager.getResourcesWithLabel(label, null ); if (lockableResources.isEmpty()) { continue ; } int defined = lockableResources.size(); int free = lockableResourcesManager.getFreeResourceAmount(label); int locked = 0; int reserved = 0; int queued = 0; for (LockableResource r : lockableResources) { if (r.isReserved()) { reserved++; } if (r.isLocked()) { locked++; } if (r.isQueued()) { queued++; } } String [] labelValueArray = {label}; resourcesDefined.labels(labelValueArray).set(defined); resourcesFree.labels(labelValueArray).set(free); resourcesLocked.labels(labelValueArray).set(locked); resourcesReserved.labels(labelValueArray).set(reserved); resourcesQueued.labels(labelValueArray).set(queued); } samples.addAll(resourcesDefined.collect()); samples.addAll(resourcesFree.collect()); samples.addAll(resourcesLocked.collect()); samples.addAll(resourcesReserved.collect()); samples.addAll(resourcesQueued.collect()); return samples; } }
          Hide
          t_westling Tomas Westling added a comment -

          Hi Philipp: I would say that yes, go ahead. I don't see any issues in adding getStatistics back again,
          it was maintaining all the graph support that was the issue, and being uncertain if it worked the way it should
          after big upgrades.
          This sounds like a much better way for people to set up statistics than the old one.

          Show
          t_westling Tomas Westling added a comment - Hi Philipp : I would say that yes, go ahead. I don't see any issues in adding getStatistics back again, it was maintaining all the graph support that was the issue, and being uncertain if it worked the way it should after big upgrades. This sounds like a much better way for people to set up statistics than the old one.

            People

            Assignee:
            phihos Philipp
            Reporter:
            phihos Philipp
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated: