Production
Managed stores, availability, sizing starting points, retention, backups, hardening and monitoring KubeHero itself.
The default install is built to show real data in five minutes, not to run a fleet for years. This page covers what to change before KubeHero becomes something your team relies on.
Start from the production overlay
The chart ships values.production.yaml:
helm upgrade --install kubehero oci://ghcr.io/kubehero-io/charts/kubehero -n kubehero-system \
-f values.production.yaml -f my-values.yamlIt sets three control-plane replicas with autoscaling (3–12), two dashboard replicas, two operator replicas (leader-elected), external Postgres and ClickHouse via DSN Secrets, demoMode: "false", auth required, network policies, PodDisruptionBudgets, a PriorityClass and restricted security contexts (non-root, read-only root filesystem, all capabilities dropped, RuntimeDefault seccomp).
Stores
| Store | Holds | Production choice |
|---|---|---|
| ClickHouse | every time series: cost, logs, profiles, flows, usage, events | a managed or operator-run ClickHouse (replicated) |
| Postgres | clusters, policies, alert rules and state, silences, the audit log | a managed Postgres with point-in-time recovery |
ClickHouse is the hot path — logs dominate its volume. Size it for your log ingest first. The embedded single-replica stores are fine for evaluation and small clusters; they are not highly available.
Availability
| Component | How it fails over |
|---|---|
| control plane | stateless replicas behind a Service; scale horizontally |
| operator | leader election — one active replica, the other waits |
| collector | DaemonSet; a restarting collector resumes logs from its checkpoint, and its shipping queue is bounded so an outage never grows memory without limit |
| collector leader | a Lease; another collector takes over cluster-scoped scans |
| advisor, dashboard, pricing engine | stateless |
If the control plane is unreachable, collectors queue (and eventually drop, counted) and the operator runs observe-only — ceilings don't trip and rightsizing doesn't apply. Nothing escalates on stale data.
Sizing starting points
Measure your own workload — these are starting points, not benchmarks.
| Component | Requests (chart default) | Scale with |
|---|---|---|
| collector | 100m CPU / 128 Mi per node (limit 1 CPU / 512 Mi) | log lines per second per node; profiling frequency |
| control plane | 200m / 256 Mi per replica (production overlay: 500m / 512 Mi) | ingest rate and query load |
| operator | 50m / 64 Mi | number of policies and workloads in scope |
| advisor | 50m / 64 Mi | investigations per minute |
| ClickHouse | 500m / 1 Gi (embedded) | log volume above all; retention |
Protect nodes from a noisy log producer with collector.logs.rateLimit (lines/s per container) and exclude chatty namespaces with collector.logs.excludeNamespaces.
Retention
ClickHouse tables carry TTLs in the schema:
| Data | Kept |
|---|---|
| log lines | 14 days (volume rollups 90 days) |
| profiles | 14 days (stacks 30 days) |
| flows | 30 days raw, 400 days hourly |
| container usage | 35 days raw, 180 days in 5-minute digests |
| cluster events | 90 days |
| spend rollups | 400 days |
To keep logs longer, change the TTL on your ClickHouse (ALTER TABLE logs MODIFY TTL …) and size storage accordingly.
Backups
- Postgres holds the state you can't recreate — policies, alert rules, the audit log. Back it up with your provider's PITR or your Postgres operator.
- ClickHouse holds telemetry. Back up spend rollups if you need long-term cost history (
BACKUP TABLE workload_cost_1h TO S3(…)); raw logs and profiles are usually not worth backing up. - Policies are CRDs — keep them in Git and let GitOps restore them.
Hardening
- Keep
controlPlane.auth.requireAuth: trueand use per-person tokens or OIDC (Identity); give MCP clients and Grafanaviewertokens. - Bring your own secret material with
controlPlane.existingSecret(keysaudit-hmac-key,admin-token,service-token,cluster-token,session-secret) if you manage secrets outside the cluster. - Enable
networkPolicies.enabled(on in the production overlay). - Decide on eBPF deliberately: it requires a privileged collector. See Security.
- Verify images: they're cosign-signed (keyless) with SPDX SBOM attestations.
Monitoring KubeHero
- Prometheus scrapes the collector and control plane via the chart's ServiceMonitors.
- Alert on collector staleness and eBPF attach failures — see collector health.
- Watch
droppedcounts on ingest andkubehero_collector_items_dropped_total: sustained drops mean the control plane or ClickHouse is undersized.
Upgrading
helm upgrade with the new chart version. The control plane applies new ClickHouse and Postgres migrations on start — they're versioned and recorded, and applied once. Read the changelog's upgrade notes for each release.