TL;DR: Metric cardinality turns ordinary labels into a cost, performance, and availability problem across Prometheus-like stacks and SaaS monitoring platforms, according to Sawmills. The governance answer is not more dashboards, but label discipline, pipeline enforcement, and blast-radius controls that stop unbounded dimensions before they reach storage or billing.
At a glance
What this is: Metric cardinality is the number of distinct time-series combinations created by metric names and labels, and Sawmills argues it is the hidden driver of observability cost and reliability failure.
Why it matters: For IAM, NHI, and broader security teams, the lesson is that uncontrolled telemetry dimensions can become an availability and governance problem, especially when monitoring underpins alerting and automation.
Context
Metric cardinality is a governance problem in observability because every new label combination creates another time series, and those series consume memory, index space, query capacity, and money. When teams allow unbounded or near-unbounded labels into monitoring pipelines, the control failure is not just noisy dashboards, but a telemetry estate that becomes harder to trust during incidents.
The identity angle is indirect but real: the same discipline needed to govern labels, dimensions, and lifecycle scope in observability is similar to the discipline required for non-human identity inventories and machine-generated access patterns. When platform metadata and application labels multiply without policy, the monitoring layer starts to behave like any other unmanaged control plane.
This is not a niche Prometheus issue. The same pattern appears in self-hosted stacks and SaaS monitoring services, which makes the governance question broader than any single vendor or deployment model.
Key questions
Q: What breaks when metric cardinality is not governed in observability pipelines?
A: When metric cardinality is not governed, series growth can overwhelm memory, indexing, query fanout, and recovery. The visible symptoms are slow dashboards, missing data, delayed alerts, and higher costs. In practice, observability stops behaving like a dependable control layer and starts behaving like an unbounded data sink.
Q: Why do high-cardinality labels create cost and reliability problems?
A: High-cardinality labels multiply the number of time series that must be stored, indexed, queried, and billed. That multiplication is what turns otherwise ordinary metrics into a governance problem. The operational result is more infrastructure load, more spend, and less confidence in the telemetry during an incident.
Q: What are the signs that observability cardinality is getting out of control?
A: Common signs include sudden growth in active series, query fanout that slows dashboards, rising memory pressure, noisy alerts, and unexpected bill spikes. If the same metric becomes expensive or slow after a new label was added, that is usually the clearest signal that the label set has escaped governance.
Q: Should teams rely on platform controls or pipeline controls for metric governance?
A: Pipeline controls should do most of the work because they can stop risky dimensions before storage or indexing. Platform controls are still useful for visibility and limits, but they are usually reactive. If the label reaches the backend, the cost and operational impact have already begun.
Technical breakdown
Where self-hosted and SaaS observability fail differently
Self-hosted Prometheus-like systems absorb cardinality through memory pressure, index growth, compaction delay, and longer recovery windows. In SaaS platforms, the pain often shifts into custom metric counts, indexing overhead, and higher bills, but the underlying issue is the same: the system is forced to carry too many distinct series. In both models, reactive tooling can show the problem after the fact, but it rarely stops bad labels from being emitted. That is why the control has to live at collection, relabeling, or pipeline enforcement points. Practical implication: choose guardrails that act before telemetry enters the long-term store or billing model.
Practical implication: place enforcement at collection time, not only in downstream dashboards.
Threat narrative
Attacker objective: The objective is not a human attacker but an uncontrolled telemetry shape that overwhelms observability governance and degrades availability, cost control, and detection reliability.
- Entry occurs when instrumentation or platform automation introduces high-cardinality labels such as user_id, request_id, raw URLs, or per-instance metadata into hot metrics.
- Escalation happens as those labels combine with histograms, autoscaling, and cloud metadata to create a combinatorial explosion of time series that consumes memory and query capacity.
- Impact appears as slow dashboards, delayed alerts, stalled ingestion, higher SaaS charges, and in some environments a monitoring layer that can no longer support incident response or autoscaling with confidence.
Breaches seen in the wild
- reviewdog Action compromise 2025: A stolen maintainer token poisoned reviewdog/action-setup, leaking CI secrets including the tj-actions bot token used in the next attack.
- CI/CD pipeline exploitation case study: Credentials in an exposed .git/config let a researcher edit a Bitbucket pipeline so it planted their SSH key on the server. No victim was named.
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
Metric cardinality is an observability governance failure, not just a performance issue. Once label combinations are allowed to expand without policy, the monitoring stack loses predictability across cost, recovery, and query behaviour. The control problem is closer to identity inventory discipline than to dashboard tuning: you need rules for what may exist, where it may be attached, and how quickly it can be removed. The practitioner lesson is to treat telemetry shape as a governed asset.
Unbounded dimensions create a hidden blast radius in both self-hosted and SaaS monitoring models. Prometheus-like systems feel the damage through memory, compaction, and query fanout, while SaaS platforms surface it through indexing pressure and billing. Different architectures expose the same governance gap: once a high-cardinality label is in flight, downstream controls are usually too late. The practitioner conclusion is to block the dimension before it becomes a series.
Metric cardinality exposes a named concept we can call observability label drift. That is the slow, often unnoticed expansion of label sets beyond the stable business and service dimensions they were meant to describe. Drift matters because it turns a monitoring vocabulary into an uncontrolled data model, and that weakens both reliability and accountability. The practitioner conclusion is to manage labels with approval, review, and enforcement as if they were production access paths.
Guardrails on telemetry shape are now resilience controls. In many environments, observability supports alerting, autoscaling, and incident triage, which means excessive series growth can affect more than cost. The governance implication is that teams should design enforcement at the collector, relabel, or pipeline layer, then verify that the control applies consistently across services and tenants. The practitioner conclusion is to make cardinality limits part of operational resilience planning.
What this signals
Observability label drift is the control gap most teams underestimate. Labels start as service metadata, then expand into a loose vocabulary that no longer maps cleanly to business or operational meaning. Once that happens, cardinality becomes a governance problem because the platform can no longer distinguish useful dimensions from accidental ones.
Pipeline enforcement matters more than retrospective cleanup. If a metric reaches the TSDB or SaaS index, teams are already paying for the mistake in storage, indexing, or query fanout. That is why collector-level filtering, relabeling, and tenant-aware caps should be treated as resilience controls, not as optional optimisation.
Metric cardinality control should sit alongside identity and access governance in modern operational policy. The parallel is not perfect, but the governance pattern is familiar: define what is allowed, enforce it before proliferation, and watch for drift over time. For practitioners, the practical test is whether noisy telemetry can be stopped at source before it becomes an outage amplifier.
For practitioners
- Define a stable label allowlist Publish a short allowlist for metrics such as env, region, service, status_code, and method, then reject labels that introduce unbounded values like user_id, session, request_id, pod_uid, or raw URLs.
- Enforce label filtering before storage Use collectors, relabel rules, or pipeline controls to drop or rewrite risky dimensions before they reach the TSDB or SaaS index, because downstream dashboards cannot undo cardinality already ingested.
- Monitor series growth per service Track active series, series added per scrape, and sudden label-value spikes by service so that runaway metrics can be isolated to the owning workload instead of affecting the whole observability estate.
- Constrain high-volume metrics by tenant or workload Set caps on max series, ingest rate, and label length for noisy services, and apply those limits at the narrowest practical scope so one team’s metric mistake does not affect others.
Key takeaways
- Metric cardinality becomes a governance failure when label combinations expand faster than teams can control cost, query load, and recovery behaviour.
- The operational evidence is consistent across stacks: uncontrolled labels increase memory pressure, fan out queries, and make both self-hosted and SaaS monitoring less trustworthy.
- The most effective control is to stop risky dimensions at collection time with label policy, pipeline enforcement, and workload-scoped limits.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK addresses the attack and risk surface, while NIST CSF 2.0, CIS Controls v8 and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | PR.AA-05 — Access Permissions, Entitlements and Authorizations | Metric labels act like governed dimensions that should not expand without control. |
| PR.DS-01 — Data-at-Rest Confidentiality | Cardinality control protects the data store from uncontrolled ingestion and storage growth. | |
| Recommendation — Apply PR.AA-05-style governance to restrict which telemetry dimensions may flow into production monitoring. Treat telemetry shape as a data protection issue and limit what gets stored at rest. | ||
| CIS Controls v8 | CIS-8 — Audit Log Management | Observability pipelines need controlled collection and review of what is being emitted. |
| Recommendation — Use CIS-8 controls to monitor telemetry growth and detect label sprawl before it impacts operations. | ||
| NIST SP 800-53 Rev 5 | AU-12 — Audit Record Generation | Metrics are audit-like records that become unreliable when their structure is uncontrolled. |
| Recommendation — Apply AU-12 to ensure telemetry generation is deliberate, bounded, and reviewable. | ||
| MITRE ATT&CK | TA0040 — Impact | The outcome is operational degradation rather than stealthy adversary activity. |
| Recommendation — Map runaway telemetry growth to TA0040 and prioritise controls that reduce operational impact. | ||
Key terms
- Metric Cardinality: The number of unique time series created by a metric and its label combinations. Higher cardinality increases storage, indexing, and query cost because every distinct label set becomes separately managed telemetry.
- Cardinality Churn: The rate at which new time series appear and old ones disappear over time. Churn stresses ingestion, compaction, and write paths even when the total number of series is not especially large.
- Label Drift: The gradual expansion of metric labels beyond the stable, intended set of dimensions. It usually starts with harmless metadata and ends with uncontrolled series growth, making the telemetry model harder to govern and more expensive to operate.
- Telemetry pipeline: A telemetry pipeline is the path security data follows from collection to analysis and retention. In mature environments it must preserve context, maintain throughput, and avoid introducing blind spots as sources, formats, and volumes change over time.
Deepen your knowledge
NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and secrets management for practitioners building stronger identity controls. It helps security teams connect identity discipline to the broader control challenges that modern platforms create.
Published by the NHIMG editorial team on August 19, 2026.
Updated on October 12, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org