There appears to be an unhappy interaction between:

• https://plugins.jenkins.io/metrics/
• https://plugins.jenkins.io/prometheus/
• https://plugins.jenkins.io/kubernetes/

Our https://jenkins.big.corp/prometheus/ Prometheus scrape endpoint is absolutely flooded with metrics like:

# HELP kubernetes_cloud_myproj_master_7116_zvf9t_provision_request_total Generated from Dropwizard metric import (metric=kubernetes.cloud.myproj_master_7116-zvf9t.provision.request, type=com.codahale.metrics.Meter)
# TYPE kubernetes_cloud_myproj_master_7116_zvf9t_provision_request_total counter
kubernetes_cloud_myproj_master_7116_zvf9t_provision_request_total 1.0

Around 50k lines in totaal. Which brings the scraping to a crawl.

But more importantly: I have a strong feeling that it is actually slowing Jenkins down as well, since the data is passed to the Metrics plugin, which in turn feeds the Prometheus plugin. All this data is processed every generation interval.

The culprit appears to be https://github.com/jenkinsci/kubernetes-plugin/blob/4287.v73451380b_576/src/main/java/org/csanchez/jenkins/plugins/kubernetes/MetricNames.java#L27

Which is triggered by: https://github.com/jenkinsci/kubernetes-plugin/blob/4287.v73451380b_576/src/main/java/org/csanchez/jenkins/plugins/kubernetes/KubernetesCloud.java#L602

I feel this should simply be removed unless there is a clear reason to keep it. It has the potential to balloon the active Metrics to astronomical proportions, which could lead to stability issues.

The other metrics I have no problem with:

kubernetes_cloud_pods_created 13475.0
kubernetes_cloud_pods_creation_failed 3430.0
kubernetes_cloud_pods_launched 9742.0
kubernetes_cloud_pods_terminated 16909.0
kubernetes_cloud_provision_nodes 16905.0 

But those kubernetes_cloud_###_provision_request_total entries are killing.

There doesn't appear to be a mechanism to prevent the Kubernetes plugin from generating these metrics.