How should security and platform teams control high-cardinality labels in metrics?

Why This Matters for Security Teams

High-cardinality labels turn observability into a cost and stability problem, not just a data hygiene issue. When metrics carry user IDs, request IDs, session tokens, pod names, or other unbounded values, the telemetry backend creates a new time series for each unique combination. That can distort dashboards, slow queries, increase retention costs, and hide real incidents under noise. The security concern is broader than billing: unstable telemetry weakens detection engineering and can make platform controls look effective until the backend saturates.

Security and platform teams should treat metric label governance as part of control design, not an afterthought. NIST Cybersecurity Framework 2.0 frames this kind of discipline under governance and continuous improvement, because telemetry quality affects visibility, response, and accountability across the stack. The practical mistake is assuming observability systems will absorb arbitrary context safely. They usually do not, especially when engineers add identifiers to simplify troubleshooting without considering cardinality. In practice, many security teams encounter the impact only after the metrics store has already been stressed by an incident, deployment spike, or runaway automation loop.

How It Works in Practice

The safest pattern is to define a narrow label contract for metrics and enforce it before data reaches the backend. Labels should describe bounded attributes such as service, route, status code, region, or environment. Anything that identifies an individual event, user, secret, or workload instance belongs in logs, traces, or security events, not in metrics. This is consistent with the operational guidance in NIST Cybersecurity Framework 2.0, where control intent depends on reliable measurement and consistent handling of data.

• Set an allowlist for metric labels at the instrumentation layer.
• Reject or strip unapproved labels in code review, CI checks, and telemetry tests.
• Use collector-side relabelling as a backstop, not the primary control.
• Track series counts, churn, and top label dimensions as operational health indicators.
• Separate request identity into logs or traces so investigations still have context.

In practice, teams also need ownership. Platform engineering should define the schema, while application teams must follow it. If a label may vary without a hard upper bound, it should be treated as a design defect. This matters for security telemetry too: cardinality explosions can bury alerting signals and create blind spots during attacks, especially when automation or multi-tenant systems generate bursts of unique label values. Guidance from Prometheus metric naming best practices remains useful here, but current guidance suggests enforcing the policy at source because downstream cleanup cannot fully recover lost performance or prevent unnecessary ingestion. These controls tend to break down when teams instrument ephemeral identifiers in shared libraries because those values spread faster than backend rules can catch them.

Common Variations and Edge Cases

Tighter label controls often reduce troubleshooting convenience, requiring organisations to balance developer speed against observability cost and reliability. That tradeoff is real, especially during incident response or in environments where teams want rich per-request context for debugging. The compromise is to preserve detail in logs and traces, then join those signals during investigation instead of encoding everything into metrics.

There is no universal standard for this yet across every telemetry stack, so best practice is evolving. Some environments allow limited dynamic labels in short-lived test namespaces, but production systems should keep the policy strict. Multi-tenant platforms, service meshes, and serverless workloads need extra care because workload identifiers and routing metadata can change rapidly and inflate series counts. The same applies to AI services and agentic workflows, where prompt IDs, conversation IDs, or tool-call identifiers should not become metric labels if they can vary without limit.

For regulated environments, treat label governance as part of resilience engineering and change control. If a team cannot explain why a label is bounded, why it is needed for dashboards, and what happens when its values grow, the label should not be promoted. That approach is compatible with monitoring guidance in the CISA Secure by Design material, which favours preventive controls over compensating cleanup. In practice, high-cardinality failures usually surface first as noisy spend reports or slow queries, then later as missing alerts.

Related resources from NHI Mgmt Group

• How should security teams handle NHI risk when visibility is high but control is weak?
• How should security teams implement dual control for high-risk identity actions?
• How should security teams handle remote access platform end-of-life without weakening control?
• How should security teams choose between a dedicated certificate platform and a unified NHI control plane?