Notifications
Clear all
Topic starter
19/08/2026 3:32 pm
TL;DR: Prometheus cardinality incidents usually start with one new high-value label, not a traffic surge, and Sawmills’ analysis shows how active series growth, histogram fan-out, and misplaced identifiers drive memory, query, and billing pain. The real issue is governance of where request-level identity belongs, not whether labels are useful.
NHIMG editorial — based on content published by Sawmills: Prometheus Cardinality: Why Your Active Series Keep Growing
Questions worth separating out
Q: How should security and platform teams control high-cardinality labels in metrics?
A: Start by banning unbounded identifiers from metric labels, then enforce that rule in code review and telemetry standards.
Q: Why do request IDs and user IDs create problems in Prometheus metrics?
A: Because metrics are meant to aggregate behaviour across populations, while those identifiers describe single events or single actors.
Q: How do you know if metric cardinality is actually under control?
A: Watch active series trends, not just ingest volume.
Practitioner guidance
- Set a bounded-label policy for metrics Define which identifier types are prohibited in metric labels, especially user_id, request_id, pod.uid, session_id, and raw URL paths.
- Use TSDB status as the first diagnostic step When series count spikes, inspect seriesCountByMetricName and labelValueCountByLabelName before changing scrape settings or storage limits.
- Move forensic identity into logs and traces Keep request-level identity in structured logs and trace attributes, then use templated labels such as http.route for metrics.
What's in the full article
Sawmills' full article covers the operational detail this post intentionally leaves for the source:
- Step-by-step PromQL and TSDB status queries for isolating the exact label that drove the series spike
- Collector and scrape-pipeline examples for deleting or rewriting harmful labels before export
- Native histogram migration considerations for teams whose latency metrics dominate series count
- Practical examples showing where specific high-cardinality values should move instead of being deleted
👉 Read Sawmills' analysis of Prometheus cardinality growth and active series control →
Prometheus cardinality explosion: what is your team doing about it?
Explore further
19/08/2026 4:14 pm
Cardinality governance is now a telemetry control discipline, not a housekeeping task. High-cardinality labels are not just noisy data points. They are unmanaged growth factors that change storage cost, query latency, and incident response quality at the same time. In observability systems, request-level identity behaves like privileged data and should be governed with the same lifecycle discipline as secrets and tokens. The practitioner takeaway is that telemetry design needs policy, not just engineering taste.
A question worth separating out:
Q: What should teams do when a high-cardinality label reaches production?
A: Contain it in the pipeline first by deleting or rewriting the label before export, then fix the source instrumentation in the next code cycle. Do not rely on storage expansion as the primary response. That only delays the cost while leaving the growth mechanism untouched.
👉 Read our full editorial: Prometheus cardinality is a control problem, not a metrics problem
