Topic starter
19/08/2026 3:32 pm
TL;DR: Prometheus cardinality incidents usually start with one new high-value label, not a traffic surge, and Sawmills’ analysis shows how active series growth, histogram fan-out, and misplaced identifiers drive memory, query, and billing pain. The real issue is governance of where request-level identity belongs, not whether labels are useful.
Editorial analysis by NHI Mgmt Group, based on content published by Sawmills: “Prometheus Cardinality: Why Your Active Series Keep Growing”.
Key questions
Q: What breaks when a Prometheus label starts carrying request-level identity?
A: Series count multiplies across every existing label combination, so memory, query latency, and cost all rise together.
Q: Why do high-cardinality labels create cost and reliability problems?
A: High-cardinality labels multiply the number of time series that must be stored, indexed, queried, and billed.
Q: What are the signs that a metrics stack is heading toward a cardinality incident?
A: A sudden jump in active series, rising head-block memory, slower queries, or cost increases without a matching traffic change are the clearest signs.
Practitioner guidance
- Bound label value sets before deployment Require review for any new metric label whose values can grow with users, sessions, requests, or objects.
- Use traces and logs for forensic identity Move request-level identity into trace attributes or structured logs so incident investigation still has the needed detail without multiplying metric series across every observation.
- Monitor active series as a control signal Track prometheus_tsdb_head_series alongside deploys and scrape health, and alert on step changes that are not explained by traffic growth or planned instrumentation changes.
Bottom line: Prometheus cardinality incidents usually begin with one misplaced label that turns stable metrics into an unbounded series problem.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
19/08/2026 4:14 pm
Label placement is the control plane for observability cost. Cardinality failures are rarely a Prometheus storage defect. They are a governance failure in which teams allow identity-like values into a data model that was designed to aggregate behaviour, not preserve forensic detail. Once that line is crossed, every downstream control becomes more expensive and less effective, so practitioners need explicit rules for what can become a metric dimension and what must remain trace or log context.
A question worth separating out:
Q: How should platform teams decide what belongs in metrics versus logs or traces?
A: Use metrics for stable, population-level behaviour and reserve logs or traces for request-specific identity and forensic detail. If a value can expand with every user, request, session, object, or pod restart, it is usually the wrong choice for a metric label because it converts observability into an unbounded series problem.
👉 Read our full editorial: Prometheus cardinality is a control problem, not a metrics problem