Prometheus cardinality is a control problem, not a metrics problem

At a glance

What this is: This analysis explains why Prometheus active series keep growing and identifies the label patterns, histogram mechanics, and pipeline controls that usually cause the explosion.

Why it matters: It matters because platform and IAM-adjacent teams need to decide where identity-like values belong, how to keep observability cost bounded, and how to stop one label from creating operational risk.

By the numbers:

• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap.

👉 Read Sawmills' analysis of Prometheus cardinality growth and active series control

Context

Prometheus cardinality becomes a governance problem when a seemingly harmless label turns one metric into thousands or millions of time series. In practice, the failure is usually not volume alone, but a mismatch between request-level identity and the places where metrics are meant to stay bounded.

This article focuses on the mechanics of active series growth, how to find the offending label quickly, and where to remove or reshape high-cardinality values before they hit storage or billing. That pattern is typical in modern observability stacks, especially when developers mix forensic detail into metrics instead of logs or traces.

The identity bridge is real here: user IDs, request IDs, pod UIDs, and other high-entropy values behave like access identifiers in a telemetry pipeline, and they need lifecycle governance just as much as secrets do.

Key questions

Q: How should security and platform teams control high-cardinality labels in metrics?

A: Start by banning unbounded identifiers from metric labels, then enforce that rule in code review and telemetry standards. Keep request-level identity in logs or traces, and use collector-side relabelling only as a backstop. The goal is to stop series growth at the source, because every new label combination creates storage, query, and billing overhead.

Q: Why do request IDs and user IDs create problems in Prometheus metrics?

A: Because metrics are meant to aggregate behaviour across populations, while those identifiers describe single events or single actors. A request ID or user ID can create a new time series for every request or user, which turns a useful metric into an unbounded cost driver. They belong in forensic data stores, not in metric labels.

Q: How do you know if metric cardinality is actually under control?

A: Watch active series trends, not just ingest volume. If prometheus_tsdb_head_series stays stable while traffic grows, the system is probably healthy; if it jumps after a deploy or framework change, a label has likely escaped its intended boundary. You should also check whether high-cardinality metrics are being rewritten before export.

Q: What should teams do when a high-cardinality label reaches production?

A: Contain it in the pipeline first by deleting or rewriting the label before export, then fix the source instrumentation in the next code cycle. Do not rely on storage expansion as the primary response. That only delays the cost while leaving the growth mechanism untouched.

Technical breakdown

Why a single label can explode active series

Prometheus stores data as series, not just metrics. A series is the unique combination of a metric name and its label values, so every new label value combination creates another indexed object, another chunk of memory, and another query path. That is why low traffic can still produce high cost. The storage engine keeps active series in the head block and indexes them for retrieval, so cardinality growth hits memory first and billing second. Histograms amplify this because each bucket multiplies every label combination. The core lesson is simple: metric usefulness depends on bounded dimensions, not on how much data the service emits.

Practical implication: Treat any label with unbounded values as a design flaw before it reaches production.

How to find the label driving the explosion

The fastest way to diagnose a cardinality incident is to inspect the TSDB status endpoint and rank series by metric name, label cardinality, and label-value pairs. PromQL can do the same job, but it is heavier, so the diagnostic sequence matters. First identify the metric family that grew, then the label that multiplied it, then the exact value pattern that changed after deployment. This turns cardinality work from guesswork into evidence-based triage. The important operational detail is that the investigation itself can be expensive, so teams should prefer targeted calls over broad, ad hoc queries when the instance is already under memory pressure.

Practical implication: Add TSDB status inspection to incident runbooks so the first response to a memory spike is attribution, not blind tuning.

Why histograms are a multiplier, not a detail

Classic Prometheus histograms create one bucket series per label combination for every bucket configured, plus sum and count series. That means the wrong extra label can multiply the entire bucket set, not just a single counter. Native histograms reduce the series footprint by using adaptive exponential bucketing, but they require instrumentation and downstream support changes. The architecture difference matters because it shifts cost from fixed bucket fan-out to a more compact distribution model. Teams that rely heavily on latency histograms should understand that bucket count and label count interact multiplicatively, which is why histogram tuning often delivers the biggest savings.

Practical implication: Review histogram instrumentation first when series growth tracks latency metrics more than application traffic.

NHI Mgmt Group analysis

Cardinality governance is now a telemetry control discipline, not a housekeeping task. High-cardinality labels are not just noisy data points. They are unmanaged growth factors that change storage cost, query latency, and incident response quality at the same time. In observability systems, request-level identity behaves like privileged data and should be governed with the same lifecycle discipline as secrets and tokens. The practitioner takeaway is that telemetry design needs policy, not just engineering taste.

High-entropy identifiers belong in logs and traces, not in metric labels. User IDs, request IDs, session IDs, and raw URL paths are useful for forensic correlation, but they are the wrong abstraction for aggregated metrics. Putting them in metric labels turns bounded operational signals into near-unbounded series sets. The better control pattern is to keep metrics coarse, then preserve investigative detail in trace attributes and structured logs. Teams should treat the placement decision as a data-governance choice, not a convenience trade-off.

Cardinality drift is a lifecycle problem because the bad label usually arrives through normal delivery. The root issue is often not an exotic failure but a routine deploy, framework upgrade, or debug field that survives longer than intended. That makes cardinality drift similar to stale access in identity systems: the control weakens when lifecycle ownership is unclear. This is where the named concept of telemetry entropy debt: the steady accumulation of unbounded labels that outpaces cleanup, making every new metric slightly more expensive to govern. Practitioners should expect drift unless they build continuous review into the pipeline.

Pipeline controls are necessary, but they are not a substitute for source-level discipline. Dropping or relabelling at the collector is an emergency brake, not a permanent design strategy. If teams rely only on downstream cleanup, the next service, framework, or incident will recreate the same problem under a different label name. The field implication is that observability governance needs both preventive coding standards and runtime enforcement. Practitioners should align developer conventions with pipeline policy so high-cardinality values never become the default path.

What this signals

Telemetry entropy debt: once high-entropy labels enter the metrics pipeline, cleanup becomes a recurring governance task rather than a one-time optimisation. Platform teams should expect recurring pressure from new frameworks, new services, and new debug fields, so policy must sit closer to instrumentation than to billing.

This is where identity governance thinking helps. Values that behave like request-level identifiers need lifecycle rules, bounded placement, and review paths just like secrets or service accounts. Teams that already manage non-human identity sprawl should recognise the same pattern in observability data and extend control discipline into telemetry.

For practitioners, the signal is to align developer guidance, collector rules, and alerting thresholds before the next deploy adds another high-entropy field. The control objective is not perfect hygiene, but preventing one label from becoming a persistent cost and reliability problem.

For practitioners

• Set a bounded-label policy for metrics Define which identifier types are prohibited in metric labels, especially user_id, request_id, pod.uid, session_id, and raw URL paths. Put the rule in engineering standards and code review checklists so the source never emits unbounded series.
• Use TSDB status as the first diagnostic step When series count spikes, inspect seriesCountByMetricName and labelValueCountByLabelName before changing scrape settings or storage limits. That quickly tells you which metric family and label values caused the jump.
• Move forensic identity into logs and traces Keep request-level identity in structured logs and trace attributes, then use templated labels such as http.route for metrics. This preserves investigative value without multiplying active series.
• Apply collector-side relabelling for emergency containment Use metric_relabel_configs or the OpenTelemetry Collector transform processor to delete or rewrite harmful labels before export. This is the fastest containment control when a bad label reaches production.
• Review histogram fan-out before scaling storage Check whether latency histograms are multiplying series through excessive bucket counts or extra labels. If they are, migrate candidate metrics to native histograms or reduce bucket granularity before buying more capacity.

Key takeaways

• Prometheus cardinality problems usually begin with one unbounded label, not with raw traffic growth.
• High-cardinality identifiers belong in logs and traces, while metrics should stay bounded and aggregable.
• Teams that treat telemetry design as a lifecycle governance issue will catch series explosions before they become outages or billing shocks.

Key terms

• Cardinality: Cardinality is the number of distinct time series a metric can create when its labels take different values. In Prometheus, every unique combination of metric name and label values becomes a separate series, so uncontrolled cardinality increases memory use, query cost, and sometimes vendor billing.
• Active Series: Active series are the time series currently held in Prometheus memory and actively tracked for scraping, querying, and storage. They matter because head-block memory scales with the number of active series, not just with how much raw data a service emits.
• Histogram Fan-out: Histogram fan-out is the multiplication effect created when a histogram metric emits multiple bucket series for each label combination, plus sum and count series. A single extra label can therefore multiply an entire bucket set, making histograms a common source of hidden series growth.
• Telemetry Entropy Debt: Telemetry entropy debt is the accumulation of high-entropy labels, debug fields, and unstable dimensions that gradually make observability data harder and more expensive to manage. The concept describes a lifecycle problem, where each new uncontrolled value increases the cost of future cleanup.

What's in the full article

Sawmills' full article covers the operational detail this post intentionally leaves for the source:

• Step-by-step PromQL and TSDB status queries for isolating the exact label that drove the series spike
• Collector and scrape-pipeline examples for deleting or rewriting harmful labels before export
• Native histogram migration considerations for teams whose latency metrics dominate series count
• Practical examples showing where specific high-cardinality values should move instead of being deleted

👉 The full Sawmills article covers the TSDB queries, histogram mechanics, and pipeline controls in implementation detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners apply lifecycle thinking to access, credentials, and other high-entropy control problems across their programmes.