Metric cardinality is breaking observability governance at scale

At a glance

What this is: This is an analysis of why metric cardinality explodes in observability systems and how policy guardrails can reduce memory pressure, query fanout, and cost.

Why it matters: It matters to security and identity practitioners because the same governance patterns used to control unbounded telemetry labels apply to access scope, data minimisation, and safe-by-default pipeline design across identity and security programmes.

👉 Read Sawmills' full analysis of why metric cardinality keeps exploding and how to tame it

Context

Metric cardinality is the number of distinct time series a monitoring system must store and query, and it becomes a governance problem when teams let unbounded labels turn a simple metric into millions of series. In practice, that means observability spend, data retention, and incident visibility can all degrade at the same time.

The security relevance is real even though this is not an identity article in the narrow sense. Cardinality control depends on disciplined source-side policy, constrained metadata, and pipeline enforcement, which are the same control patterns that identity, NHI, and security teams already rely on when they limit exposure and contain blast radius.

Sawmills uses Prometheus and Datadog examples, but the operational issue is broader than any one platform. Any stack that accepts arbitrary labels without guardrails will eventually let convenience outrun control, which is a typical failure mode in fast-growing engineering environments.

Key questions

Q: How should security teams control metric cardinality in shared observability platforms?

A: Security teams should treat metric labels like governed metadata, not free-form text. Start with a short allowlist of stable dimensions, block unbounded values at the collector edge, and set per-service series budgets so one noisy workload cannot degrade shared dashboards or billing. The most effective control is prevention at ingestion, not cleanup after storage.

Q: Why does metric cardinality become an availability problem during incidents?

A: High cardinality forces query engines to scan too many series, which slows dashboards, raises memory use, and can delay alert evaluation. During an incident, that means engineers lose visibility exactly when they need it most. If observability systems feed autoscaling or alerting, the impact can spread from monitoring delay to service instability.

Q: What do teams get wrong about high-cardinality labels?

A: Teams often focus on whether a label is useful in one dashboard and ignore whether it is stable across the whole fleet. User IDs, request IDs, raw URLs, pod IDs, and image hashes look harmless in isolation but create multiplication effects when combined with service, region, and host dimensions. That is why label review has to be policy-driven.

Q: How do teams know if cardinality controls are actually working?

A: Look for falling active series counts, fewer sudden series spikes, faster query response times, and lower variance between services. If an observability platform still shows repeated fanout, expensive indexing, or noisy metrics after policy changes, the controls are not reaching the source. Effective governance shows up in stable baselines, not just cleaner dashboards.

Technical breakdown

Why label combinations create time-series explosion

Metric cardinality rises because each unique metric name and label set becomes a separate time series. Add labels such as service, endpoint, status code, and host, and the storage engine must track every combination. Histograms multiply the effect because each bucket is another series per label set. Churn is different: it measures how quickly series appear and disappear, which strains write paths and compaction even when total cardinality is modest. The important point is that both problems are multiplicative, not additive, so a single unbounded label can overwhelm a healthy metric family.

Practical implication: define a label allowlist and block unbounded dimensions before they ever reach collection or storage.

How Kubernetes and autoscaling amplify metric cardinality

Modern infrastructure adds metadata automatically, which means the application team often does not control the full label set. Kubernetes, cloud providers, and service meshes inject pod, node, zone, cluster, and version data, and those fields combine with application labels into a much larger effective cardinality set. Autoscaling and ephemeral jobs make this worse because new instances create fresh values continuously. Even well-intended per-instance or per-endpoint metrics can become untenable when multiplied across regions, clusters, and rollouts.

Practical implication: standardise which runtime labels are permitted and strip unstable metadata at the collector or agent edge.

Why observability backends fail under query fanout

Cardinality is not only a storage problem. Query engines must fan out across all matching series, which drives slower dashboards, higher memory use, and poorer cache efficiency. In self-hosted Prometheus-like systems, that means larger head blocks, heavier compaction, and slower WAL replay. In SaaS platforms, the same behaviour shows up as rising billable metric counts and more expensive indexing. The system can still be technically healthy while becoming operationally unusable during an incident, which is why the governance model must treat cardinality as an availability risk, not just a cost issue.

Practical implication: monitor active series per metric and enforce per-service caps before slow queries become an incident multiplier.

NHI Mgmt Group analysis

Metric cardinality is an observability governance failure, not a telemetry tuning issue. Once teams allow arbitrary labels into hot metrics, they create a control problem that is structurally similar to credential sprawl in identity systems. The issue is not that observability is inherently expensive, but that unbounded dimensions create unmanaged state. Practitioners should treat label policy as a security-adjacent control with auditability, ownership, and enforcement.

Label sprawl creates a new kind of blast-radius problem. A single high-cardinality service can starve dashboards, inflate costs, and degrade incident response across shared platforms. That is the same failure pattern seen when one poorly governed identity or workload overwhelms a shared control plane. In observability, the remedy is not only technical filtering but also ownership boundaries, service-level caps, and documented exception handling.

Cardinality control and identity governance share the same design principle: constrain what can multiply. In identity programmes, that means controlling account proliferation, privilege scope, and lifecycle drift. In observability, it means controlling label vocabulary, metadata injection, and query fanout. The named concept here is series multiplication risk, which is the point at which convenience-driven instrumentation outpaces the system’s ability to govern it.

Pipeline enforcement matters more than backend cleanup. Reactive tools can show where the expensive series are, but they do not prevent the next bad label from shipping. That makes collection-time policy, CI checks, and agent-side filtering the durable controls. Teams should align observability governance with the same fail-closed thinking used in access control and secrets handling.

The organisations that keep observability usable will treat telemetry as governed data, not a free-for-all event stream. That means baselining, anomaly detection, reviewable allowlists, and automatic blocking or aggregation when thresholds are crossed. Practitioners should expect cardinality to become a recurring cost and resilience issue as service fleets, ephemeral workloads, and third-party metadata continue to grow.

What this signals

Metric cardinality is a preview of a broader governance pattern that security teams should expect to confront more often: systems fail when unbounded metadata is allowed to multiply faster than policy can contain it. The practical lesson is to push controls left into collection, classification, and automation rather than rely on back-end cleanup after the fact.

The named concept here is series multiplication risk: the point at which a few seemingly reasonable tags become an uncontrollable series explosion once they are combined with infrastructure metadata and autoscaling. Teams that already manage identity sprawl, secrets drift, and privilege creep should recognise the same control logic in telemetry governance.

For practitioners, the next step is to connect observability policy to existing governance routines such as change control, platform engineering standards, and service onboarding. That creates a durable control surface for labels, retention, and exceptions instead of letting each team invent its own metric hygiene.

For practitioners

• Publish a stable label vocabulary Allow only low-entropy labels such as env, region, service, and status code, and explicitly deny user_id, session, request_id, raw URL paths, and container-specific identifiers.
• Enforce filtering at the collector edge Use Prometheus relabeling or OpenTelemetry Collector processors to drop or rewrite unstable labels before data reaches storage or indexing layers.
• Set per-service series budgets Track active series counts by metric and by service, then cap or aggregate the offending metric when a service exceeds its expected baseline.
• Reduce Kubernetes metadata inheritance Audit agent tag settings so pods, nodes, image hashes, and similar runtime fields are only attached when they are operationally necessary for dashboards or alerts.
• Review histograms as multiplication points Treat every histogram bucket as a force multiplier and avoid attaching high-cardinality labels to request latency metrics, especially in autoscaled fleets.

Key takeaways

• Metric cardinality becomes dangerous when unstable labels multiply across fleets, histograms, and platform-generated metadata.
• The operational impact is not just cost inflation, but slower queries, weaker incident visibility, and greater strain on shared monitoring infrastructure.
• The durable fix is source-side governance: strict label policy, collector enforcement, and per-service limits that prevent bad metrics from entering the system.

Key terms

• Metric Cardinality: The number of unique time series created by a metric and its label combinations. Higher cardinality increases storage, indexing, and query cost because every distinct label set becomes separately managed telemetry.
• Cardinality Churn: The rate at which new time series appear and old ones disappear over time. Churn stresses ingestion, compaction, and write paths even when the total number of series is not especially large.
• Label Allowlist: A controlled list of metric labels that are permitted in a telemetry pipeline. It is a governance mechanism that keeps stable dimensions visible while preventing unbounded or sensitive values from multiplying series.
• Query Fanout: The widening of a query across many matching time series. When fanout is high, dashboards and alerts become slower because the backend must touch more series to answer the request.

What's in the full article

Sawmills' full article covers the operational detail this post intentionally leaves for the source:

• Concrete Prometheus relabeling examples for dropping risky labels before storage
• Datadog indexing guidance for keeping high-volume metrics queryable without exploding cost
• Step-by-step policy patterns for allowlists, denylists, and threshold-based blocking
• Tool-by-tool comparison of telemetry pipeline controls across self-hosted and SaaS stacks

👉 Sawmills' full article includes implementation examples for Prometheus, Datadog, and telemetry pipeline controls.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It gives identity and security practitioners a practical governance lens they can apply to other control-plane problems, including telemetry and access policy design.