Why do high-cardinality metrics make Prometheus harder to operate at scale?

Why This Matters for Security Teams

Prometheus is efficient when series counts are predictable, but it becomes increasingly expensive when labels explode into many distinct combinations. The operational problem is not only storage growth, it is also index pressure, higher scrape and query cost, and more time spent figuring out which label dimensions are creating the load. In practice, teams usually discover the issue after dashboards slow down or alert latency starts to drift.

That is why series discipline matters early. High-cardinality patterns often come from labels that look useful in isolation, such as request IDs, user IDs, pod hashes, or other per-entity dimensions that produce near-unique time series. Once those dimensions are exposed at scale, the monitoring system itself starts to compete with the application for memory and CPU. Prometheus then shifts from being a simple telemetry backbone to a system that requires active governance of metric design and retention.

For background on broader operational risk in large observability estates, the Ultimate Guide to NHIs, Why NHI Security Matters Now is useful because it shows how quickly unmanaged scale creates visibility and control problems. In practice, many security teams encounter the cost of bad telemetry design only after storage, query, and alerting strain are already affecting incident response.

How It Works in Practice

Prometheus stores each unique metric plus label set as a separate time series. Cardinality rises whenever a metric is multiplied by labels with many possible values, and the system has to track, index, scrape, compact, and query each series independently. That affects both the ingest path and the query path. Even if each sample is small, the cumulative overhead of many series can dominate the cost profile.

The practical effect is that high-cardinality labels create more than just “more data.” They increase the number of active series in memory, enlarge the TSDB index, slow down aggregations, and make rule evaluation more expensive. Queries that filter or group by the offending labels also become harder to optimize because the engine has to scan and combine a much larger set of series. This is why a small change in instrumentation can have an outsized effect on cluster stability.

A useful operating model is to treat labels as dimensions that should explain system behaviour, not identify every individual event. Common examples of problematic label choices include:

• Per-request or per-transaction identifiers
• Highly variable user or customer identifiers
• Ephemeral pod or container hashes when they are not needed for analysis
• Paths or parameters that encode unbounded values

Teams usually get better results when they aggregate at the right boundary, keep labels bounded, and separate debugging detail from long-lived metrics. When detailed per-entity visibility is needed, it is usually better handled through logs, traces, or short-lived debug tooling rather than permanent metrics. The Guide to the Secret Sprawl Challenge is a useful companion for understanding how operational scale becomes hard to govern once uncontrolled growth is normalised. These controls tend to break down when teams allow free-form labels in shared instrumentation libraries because the blast radius grows faster than review processes can keep up.

Common Variations and Edge Cases

Tighter metric design often improves stability, but it also reduces the amount of fine-grained detail available during investigations, so teams have to balance observability depth against operational cost. That tradeoff is especially visible in multi-tenant systems, service meshes, and large Kubernetes environments where labels are tempting because they simplify troubleshooting for one team while creating system-wide cost for everyone.

One common edge case is that a label may be acceptable in a small environment but become harmful after rollout to many clusters, customers, or regions. Another is that a metric may look safe because the value space is bounded today, then turn high-cardinality later when product behaviour changes. Best practice is to review labels against expected future growth, not just current traffic.

There is also a difference between metrics used for stable operational reporting and metrics used for ad hoc debugging. The first should stay low-cardinality and reusable; the second can tolerate more detail if it is time-limited and explicitly scoped. The The NHI and Secrets Risk Report is relevant as a scale-and-governance analogue because it highlights how unmanaged proliferation makes control much harder. In practice, cardinality becomes a problem fastest when instrumentation decisions are made locally by developers without a central standard for label hygiene.

Risk and Threat Considerations

High-cardinality metrics create an operational risk surface because they can exhaust memory, slow queries, and make alerting unreliable. In large environments, the danger is not just cost, it is reduced visibility at the exact moment teams need telemetry to support incident response or performance triage.

Failure mechanism: A metric with unbounded label values multiplies the number of series Prometheus must store and evaluate. That drives higher scrape overhead, heavier compaction work, slower rule execution, and longer query latency, especially when dashboards or alerts aggregate across the busy dimensions.

Impact: Prometheus can become sluggish, expensive to run, and harder to trust. Teams may miss important signals, delay investigations, or disable useful alerts because the monitoring stack itself has become noisy and unstable.

Practitioner Guidance

What to prioritise: Review the labels on your highest-volume metrics first, not your most visible dashboards. The goal is to find unbounded or near-unbounded dimensions that are multiplying series count without adding durable analytical value.

Decision rule: If a label value can grow with every request, user, build, or ephemeral instance, treat it as a debugging attribute rather than a production metric dimension. Keep the production metric bounded and push detail into logs or traces when needed.

What good looks like: A healthy Prometheus estate has stable series growth, predictable query latency, and alert rules that remain fast enough to evaluate under load. If operators need constant manual cleanup of new labels, the instrumentation model is already too permissive.

Practitioner takeaway: The real discipline is not collecting less telemetry, it is making sure each metric dimension earns its operating cost at scale.

Related resources from NHI Mgmt Group

• Why do non-human identities make traditional DLP and access controls harder to operate at scale?
• Why do fragmented tools make zero trust harder to scale?
• Why does AI make economies of scale harder to secure?
• Why do shutdowns make identity and access management harder to operate safely?