Why do high-cardinality metric tags create such a cost problem?

Why This Matters for Security Teams

High-cardinality metric tags are a cost problem because they turn observability into a storage and query scaling problem, not just a telemetry design choice. A tag such as user ID, request ID, container ID, or full URL can explode the number of unique time series and force monitoring platforms to retain, index, and scan far more data than planned. That drives both direct spend and operational drag when dashboards, alerts, and investigations slow down. The same pattern also makes it harder to define stable security signals, because noisy labels fragment what should have been one measurable control.

Security teams should treat tag design as part of telemetry governance. The practical question is not whether a label is convenient in code, but whether it is stable enough to support incident response, trend analysis, and alerting at scale. This is consistent with the control discipline encouraged by the NIST Cybersecurity Framework 2.0, where measurement only helps if it is usable and repeatable. In practice, many teams discover the real cost only after a dashboard or billing review shows that one “helpful” tag created millions of series and masked the signal they needed most.

How It Works in Practice

Metric backends usually store each unique combination of metric name plus tag values as a separate time series. When the tag set includes values that change for every request, session, host, or object, the platform no longer sees one metric with dimensions. It sees a large population of near-unique streams that must be ingested, indexed, compacted, retained, and queried independently. That affects compute, memory, storage, and network traffic. The result is often higher cost per unit of visibility, which is the opposite of what observability is meant to achieve.

Operationally, the issue shows up in three places:

• Ingestion: more series means more metadata and more write overhead.

• Querying: dashboards and searches fan out across many series, increasing latency.

• Alerting: rules that should watch a population instead evaluate too many distinct streams, creating noise and extra evaluation cost.

Good practice is to keep tags bounded, stable, and useful for aggregation. Security and platform teams should prefer service, environment, region, and workload class over per-request or per-user identifiers unless there is a very specific investigative need. Where fine-grained attribution is required, current guidance suggests keeping that data in logs or traces rather than in metrics, because metrics are optimized for aggregation, not identity-level detail. For implementation guidance on signal design and telemetry hygiene, the CISA Secure by Design guidance is a useful reminder that defaults should reduce operational burden rather than amplify it. These controls tend to break down when engineering teams auto-tag every event with request-scoped identifiers because the metric pipeline then loses its ability to aggregate efficiently.

Common Variations and Edge Cases

Tighter tag governance often improves cost control and query performance, but it can reduce the speed of root-cause analysis if teams remove labels that investigators actually need. That tradeoff is real, and best practice is evolving rather than universal. Some organisations can safely support moderately rich tags if they enforce strong cardinality budgets, short retention, and clear naming standards. Others, especially those with rapid microservices growth or tenant-heavy SaaS workloads, need much stricter limits because each new customer, pod, or endpoint can multiply series counts quickly.

The edge cases are usually about intent. A tag that is harmless in a low-volume dev environment can become expensive in production once it is attached to high-frequency metrics. High-cardinality labels are also more dangerous when they appear in automated alert rules, because every unique value can become a separate evaluation path. Where identity or tenant attributes must be retained, consider whether the data belongs in logs, traces, or security events instead of metrics. This distinction matters because metrics answer “how much” and “how often,” while investigation workflows often need “who” and “which one.” The high-cardinality guidance widely discussed by observability practitioners aligns with that principle, even though there is no universal standard for tag budgets yet.

Related resources from NHI Mgmt Group

• Why do package publishing credentials create such a high-risk identity problem?
• Why do exposed JWTs and API tokens create such high risk?
• Why do shared accounts create such a large security problem in higher education?
• Why do plaintext secrets create such a large AWS security problem?