Architecture

KubeHero is a collector on every node, a control plane that stores and serves everything, an operator that is the only thing allowed to change workloads, and an advisor that can read everything and change nothing.

┌──────────────────── every cluster (K8s ≥ 1.28 · AKS · GKE · EKS) ────────────────────┐
│ collector (DaemonSet — reports only its own node)          operator (Deployment)      │
│  cost + usage · logs · pprof scrape · eBPF flows + CPU     RightsizingPolicy          │
│  profiles · events (leader: unschedulable, Warning events)  CeilingPolicy · Budget     │
└──────────────┬───────────────────────────────────────────────────────▲────────────────┘
               │ Connect-RPC (protobuf, gzip, per-cluster token)          │ burn rate, recommendations
               ▼                                                         │
┌──────────────────────────────────── control plane ─────────────────────┴──────────────┐
│ ingest: IngestPodCost + TelemetryService · compat: Loki push/query · OTLP · Pyroscope  │
│ engines: LogQL · Drain · allocation · FOCUS · rightsizing · flamegraphs · service map  │
│          anomalies · alert evaluator          auth: API keys, cluster tokens, OIDC     │
│   ClickHouse — every time series (TTLs + rollups)      Postgres — clusters, policies,  │
│                                                        alerts, silences, audit         │
└──────▲─────────────────▲────────────────────▲────────────────────────▲────────────────┘
       │ read RPCs        │                    │                        │ Loki / OpenCost APIs
   dashboard         advisor (read-only)   CLI · MCP server         Grafana · OpenCost clients
                          │ optional, your key
                          ▼
                      Claude API

ComponentsCopy Anchor Link

The collectorCopy Anchor Link

Each collector reports only the node it runs on (NODE_NAME from the downward API). Cluster-scoped scans — unschedulable pods and Kubernetes Warning events — run on one leader elected through a coordination.k8s.io Lease named kubehero-collector.

• Cost — pods are priced from their node: max(requests, measured usage) share of allocatable × the node's hourly price, split into CPU / memory / GPU. Each sample carries its interval, so spend is rate × interval. See Cost allocation.
• Usage history — per-container CPU and working-set memory from the kubelet Summary API with requests, limits, restarts and the last termination reason — the input to rightsizing.
• Logs — tails /var/log/pods (CRI and json-file), checkpointed and rate-limited. See Logs.
• Profiles — pprof scraping of annotated pods, and eBPF perf_event CPU sampling of every container. See Profiling.
• Flows — eBPF cgroup_skb byte accounting and TCP retransmits. See Network.
• Events — OOM kills, crash loops, image pull failures, restarts, evictions, node pressure, unschedulable pods.

Its Kubernetes RBAC is read-only. eBPF requires a privileged container — see Security.

The collector ships to the control plane with generated Connect clients (protobuf + gzip), per-call timeouts, jittered retries and a bounded queue: a control-plane outage drops the oldest items (counted) rather than growing memory.

The control planeCopy Anchor Link

• Transport — Connect-RPC: the same .proto contracts serve Go, TypeScript, the CLI and plain curl. The services: ControlPlaneService, TelemetryService, CostService, LogsService, ProfilesService, NetworkService, AlertsService (and AdvisorService on the advisor). See the API reference.
• Ingest — batched writers per table with backpressure (ResourceExhausted when a queue is full) and validation limits: at most 10,000 log entries, 5,000 flows, 2,000 profiles per request; log lines truncated at 64 KiB; timestamps more than 10 minutes in the future rejected.
• Stores — ClickHouse for every time series, Postgres for metadata. The ClickHouse schema is versioned (kubehero_schema_migrations); Postgres uses golang-migrate. control-plane migrate applies both; on start the control plane waits for configured stores (KUBEHERO_STORE_WAIT, default 2m) instead of silently falling back to demo data.
• Demo mode — RPCs whose store isn't configured serve clearly-labelled fixtures (source: "demo"). KUBEHERO_DEMO_MODE=false (controlPlane.demoMode: "false") turns them off; those RPCs then fail with FailedPrecondition.

RetentionCopy Anchor Link

The operator and the guarded loopCopy Anchor Link

The operator watches three CRDs — see the CRD reference:

1. Propose — an agent (or you) drafts a policy manifest. It is inert.
2. Arm — a human sets the arming annotation from the dashboard or kubehero cap --arm.
3. Execute — the operator acts in bounded steps: rightsizing patches requests only when every guard passes; ceilings escalate alert → hpa.cap → pod.evict → nodepool.cordon, dry-run by default.
4. Audit — every change writes an HMAC-signed, append-only row.
5. Reverse — kubehero undo restores the previous spec.

The operator reads burn rates and rightsizing recommendations from the control plane. If it can't reach the control plane it runs observe-only: ceilings never trip and rightsizing never applies.

The advisorCopy Anchor Link

Read-only by construction: its only inputs are the control plane's read RPCs, it holds no Kubernetes credentials, and every proposal is validated against a CRD whitelist. Claude (your key) or a deterministic rules brain. See Agents & MCP.

Deployment topologiesCopy Anchor Link

• Single cluster — everything in one namespace, embedded stores. Quickstart.
• Federated — one control plane in a hub; collector + operator in each workload cluster. Deploy.
• Air-gapped — images mirrored, rules brain only. Deploy.