Metrics reference
The Prometheus series KubeHero exports, the recording rules and alerts the chart installs, and the collector's own health metrics.
KubeHero keeps Prometheus as your metrics TSDB. The collector exposes a small, stable set of chargeback series on /metrics (port 8081); the chart installs a ServiceMonitor, recording rules and alerts for kube-prometheus-stack. Everything richer — logs, profiles, flows, usage history, allocation over any window — lives in ClickHouse and is served by the APIs.
Chargeback series
Every series carries the chargeback labels below.
| Metric | Type | Unit |
|---|---|---|
kubehero_pod_cost_usd_per_second | gauge | USD per second attributed to the pod |
kubehero_pod_recoverable_usd_per_second | gauge | USD per second rightsizing could recover |
kubehero_pod_cpu_millicores | gauge | CPU, millicores |
kubehero_pod_gpu_util_ratio | gauge | GPU utilization 0–1 (GPU pods only) |
| Label | From | Example |
|---|---|---|
namespace, pod | the pod | payments, payments-worker-5c7b9d8f6-9tj2l |
team | kubehero.io/team pod label (chargeback.teamLabel), namespace map, else unattributed | payments |
cost_center | kubehero.io/cost-center pod label | fin-platform |
nodepool | the node's pool label (auto-detected for AKS / GKE / EKS, or chargeback.nodepoolLabel) | gpu-a100 |
cloud, region | the node's provider and topology.kubernetes.io/region | aws, us-east-1 |
gpu_kind | GPU pods only | A100 80GB |
# $/hour per team, averaged over 5 minutes
sum by (team) (kubehero_pod_cost_usd_per_second) * 3600Recording rules
Installed by the chart's PrometheusRule (prometheus.enabled: true, requires prometheus-operator CRDs):
| Rule | Meaning |
|---|---|
kubehero:pod_cost_usd:rate1m | per-pod $/minute |
kubehero:team_cost_usd:rate1h / rate24h / rate30d | per-team spend projected per hour / day / 30 days |
kubehero:nodepool_cost_usd:rate1h | per-nodepool $/hour |
kubehero:cost_center_cost_usd:rate1h | per-cost-center $/hour |
kubehero:team_gpu_idle_cost_usd:rate1h | $/hour teams spend on idle GPU |
kubehero:team_recoverable_usd:rate1h | $/hour recoverable by rightsizing, per team |
Alerts
| Alert | Fires when |
|---|---|
KubeHeroTeamOverBudgetProjected | a team's 24h spend, projected over 30 days, exceeds its budget series for 15 minutes |
KubeHeroGPUIdleExcessive | a team spends more than $500/hour on idle GPU for an hour |
These are Prometheus alerts for teams that alert from Prometheus. KubeHero's own alert engine covers logs, spend, budgets, anomalies, network and events without Prometheus.
Collector health
The collector also exports its own health on /metrics:
| Metric | What it tells you |
|---|---|
kubehero_collector_pods | pods on this node the collector is tracking |
kubehero_collector_leader | 1 on the elected leader (cluster-scoped scans) |
kubehero_collector_scan_duration_seconds, kubehero_collector_scan_errors_total | cost scan timing and failures |
kubehero_collector_items_sent_total, kubehero_collector_items_dropped_total | items shipped / dropped |
kubehero_collector_send_errors_total, kubehero_collector_payload_bytes_total | shipping errors and bytes |
kubehero_collector_queue_items, kubehero_collector_queue_bytes | in-memory queue depth (bounded) |
kubehero_collector_last_success_timestamp_seconds | last successful ship — alert on staleness |
kubehero_collector_log_files, kubehero_collector_log_lines_read_total, kubehero_collector_log_bytes_read_total | log tailing |
kubehero_collector_log_lines_dropped_total, kubehero_collector_log_lines_truncated_total | rate-limited and over-long lines |
kubehero_collector_log_file_events_total | rotations, truncations, new files |
kubehero_collector_profile_scrapes_total | pprof scrapes |
kubehero_collector_events_emitted_total | cluster events emitted |
kubehero_collector_ebpf_enabled, kubehero_collector_ebpf_attached | eBPF requested / actually attached |
kubehero_collector_ebpf_total | eBPF subsystem counters (flows and profiles emitted, drain errors) |
# a collector that hasn't shipped for 5 minutes
time() - kubehero_collector_last_success_timestamp_seconds > 300
# nodes where eBPF was requested but couldn't attach
kubehero_collector_ebpf_enabled == 1 and kubehero_collector_ebpf_attached == 0Cardinality
The chargeback series are per pod: expect roughly one series per running pod per metric. Per-container, per-flow and per-log-line data never becomes Prometheus series — query it through the APIs.