Prometheus + Grafana

KubeHero keeps Prometheus as your metrics TSDB and rides your existing Prometheus + Grafana stack. The chart ships three resource types that plug into kube-prometheus-stack:

• ServiceMonitor — tells Prometheus where to scrape our /metrics
• PrometheusRule — chargeback recording rules + alerts
• ConfigMap dashboards — picked up by Grafana's sidecar via the grafana_dashboard=1 label

PrerequisitesCopy Anchor Link

You need the prometheus-operator CRDs in the cluster — installing the kube-prometheus-stack chart covers this. The KubeHero chart detects monitoring.coreos.com/v1 and only emits the resources when those CRDs exist.

Values that wire itCopy Anchor Link

# values.yaml for kubehero
prometheus:
  enabled: true
  scrapeInterval: "30s"
  scrapeTimeout: "10s"
  # Match whatever label your Prometheus CR uses for ServiceMonitor selection.
  # kube-prometheus-stack's default is the release name of its own install.
  release: "kube-prometheus-stack"
  extraLabels: {}

grafana:
  enabled: true
  sidecarLabel: "grafana_dashboard"
  sidecarLabelValue: "1"
  folder: "KubeHero"
  dashboards:
    chargeback: true
    fleet: true
    gpu: true

What gets installedCopy Anchor Link

Inspect after helm install:

kubectl -n kubehero-system get servicemonitor,prometheusrule
kubectl -n monitoring get configmap -l grafana_dashboard=1 | grep kubehero

You should see:

• servicemonitor/kubehero-collector
• servicemonitor/kubehero-control-plane
• prometheusrule/kubehero-chargeback
• 3 ConfigMaps (chargeback / fleet / gpu dashboards)

Troubleshooting discoveryCopy Anchor Link

If Prometheus isn't scraping us:

1. Confirm the release: label in values.yaml matches your Prometheus CR's spec.serviceMonitorSelector.matchLabels.release. Mismatch means Prometheus ignores our ServiceMonitor.

2. Or disable label-based selection in your Prometheus:

   Copy Text

   # kube-prometheus-stack values
   prometheus:
     prometheusSpec:
       serviceMonitorSelectorNilUsesHelmValues: false
       ruleSelectorNilUsesHelmValues: false

3. Check the Prometheus web UI → Status → Targets — kubehero-collector should be UP.

Remote-write to Grafana CloudCopy Anchor Link

If you terminate Prometheus at Grafana Cloud rather than self-hosted:

# kube-prometheus-stack values.yaml
prometheus:
  prometheusSpec:
    remoteWrite:
      - url: https://prometheus-blocks-prod-us-central1.grafana.net/api/prom/push
        basicAuth:
          username: { name: grafana-cloud-creds, key: username }
          password: { name: grafana-cloud-creds, key: password }

Our recording rules evaluate in your self-hosted Prometheus and arrive at Grafana Cloud pre-aggregated, so dashboard queries are fast even on the hosted side.

Shipped dashboardsCopy Anchor Link

Chargeback by teamCopy Anchor Link

Five panels: hourly rate time series per team, 30-day projected spend bar gauge, nodepool breakdown, GPU idle cost per team, top-20 workloads table.

FleetCopy Anchor Link

Headline stats: hourly burn, recoverable / month, per-cluster time series.

GPU panelCopy Anchor Link

Utilization heatmap + per-GPU idle-hour cost ranking.

Disable any of them:

grafana:
  dashboards:
    chargeback: true
    fleet: true
    gpu: false

Logs in GrafanaCopy Anchor Link

KubeHero also implements Loki's query API, so Grafana can read KubeHero's logs through a regular Loki data source — Explore, log panels and Grafana-managed alerts all work. See Compatibility for the provisioning snippet.

Custom PromQL for your own dashboardsCopy Anchor Link

The chargeback series are gauges in USD per second, so aggregate them with sum, not rate:

# $/hour per team, right now
sum by (team) (kubehero_pod_cost_usd_per_second) * 3600

# projected monthly spend per team (recording rule)
kubehero:team_cost_usd:rate30d

# the 20 pods with the most recoverable $/hour
topk(20, sum by (namespace, pod) (kubehero_pod_recoverable_usd_per_second) * 3600)

See Metrics reference for the complete list, including the collector's own health series.