Why do high-cardinality metrics cause so many observability problems?

Why This Matters for Security Teams

High-cardinality metrics are not just a performance nuisance. They can hide operational signals, inflate monitoring spend, and delay response when teams need fast, trustworthy telemetry. The core issue is that observability systems are designed to aggregate patterns, not preserve every unique request attribute. When labels include user IDs, session tokens, container IDs, or other near-unique values, the metric backend must store and query far more series than intended.

That creates a security and resilience problem as well as a platform problem. During an incident, noisy metric dimensions can make it harder to see service degradation, anomaly spikes, or infrastructure saturation. Current guidance from NIST SP 800-53 Rev 5 Security and Privacy Controls supports controlling monitoring data quality and protecting the reliability of security-relevant telemetry, which is the right lens here. Metrics that are too granular often look precise but are operationally misleading.

Security teams also need to consider the identity and automation angle. Agents and scripts often emit labels automatically, and if those labels carry request-specific context, the observability pipeline becomes a hidden data-governance risk. In practice, many security teams encounter the cost and noise problem only after a dashboard has already become unusable during a live incident, rather than through intentional telemetry design.

How It Works in Practice

Metrics systems usually index data by name plus label set. Each unique combination becomes a separate time series. A small set of stable labels, such as service, region, or status code, is manageable. A label that changes on every request, such as transaction ID or raw URL path, multiplies series count rapidly and drives up storage, indexing, and query overhead.

The operational fix is to keep metrics low-cardinality and move detailed context elsewhere. Traces can preserve request paths and timing details, while logs can carry event-level context under stronger retention and access controls. That separation is aligned with how observability platforms are intended to work and with the control emphasis in NIST guidance on mitigating excessive data exposure in logging and telemetry.

• Use labels for stable dimensions only, such as service name, environment, and coarse status categories.
• Replace raw identifiers with bounded categories, hashes only where they remain operationally meaningful, or sampling where full fidelity is not required.
• Push request-level detail into traces or logs, then restrict access to those stores with role-based controls and retention rules.
• Set cardinality budgets in instrumentation code reviews so new labels are evaluated before release.
• Monitor series growth as a platform health indicator, not just application error rates.

For resilient operations, this also fits the monitoring and detection expectations in the CISA Secure by Design guidance, which favours systems that fail safely rather than generating avoidable complexity. These controls tend to break down in microservice-heavy environments with auto-generated labels because the telemetry volume grows faster than review processes can catch it.

Common Variations and Edge Cases

Tighter metric governance often increases implementation overhead, requiring organisations to balance richer diagnostic detail against backend cost and operational simplicity. That tradeoff becomes more visible in distributed systems, multi-tenant platforms, and agentic automation where many components emit telemetry independently.

There is no universal standard for the ideal label set, so best practice is evolving. Some teams allow higher-cardinality dimensions in short-lived debug environments but block them in production. Others permit controlled exceptions for a small number of business-critical dimensions. The key is to define those exceptions explicitly, because “temporary” labels often become permanent once dashboards depend on them.

Identity and access data deserve special caution. A request-specific label can accidentally expose usernames, account identifiers, or session context to anyone who can view dashboards. That makes observability design part of privacy and access governance, not just SRE hygiene. Where AI agents generate metrics, the same issue can appear through tool names, prompt fragments, or job IDs, so telemetry schemas should be reviewed as part of release gating rather than left to application defaults. In practice, the hardest failures happen when teams treat observability metadata as harmless technical detail instead of governed production data.

Related resources from NHI Mgmt Group

• How should security and platform teams control high-cardinality labels in metrics?
• Why do SaaS misconfigurations cause so many breaches?
• How can organisations avoid reporting too many cybersecurity metrics?
• Why do disconnected apps create so many audit problems?