Security teams should treat metric labels like governed metadata, not free-form text. Start with a short allowlist of stable dimensions, block unbounded values at the collector edge, and set per-service series budgets so one noisy workload cannot degrade shared dashboards or billing. The most effective control is prevention at ingestion, not cleanup after storage.
Why This Matters for Security Teams
Metric cardinality becomes a security and reliability issue the moment shared observability platforms turn into cost, performance, or blind-spot amplifiers. High-cardinality labels can overwhelm ingestion pipelines, distort alerting, and hide useful signals behind noise. That makes the problem more than a monitoring concern: it is a control problem tied to governance, service ownership, and abuse resistance. The NIST Cybersecurity Framework 2.0 is useful here because it emphasizes governance, monitoring, and operational resilience rather than treating telemetry as a purely technical afterthought.
Teams often misread cardinality growth as a tooling issue and respond by adding more storage or more retention, which delays the real fix. The better question is which label values are genuinely bounded, which are derived from controlled identifiers, and which are effectively user input. If that distinction is unclear, then alert fatigue and billing surprises are usually only the visible symptoms.
In practice, many security teams discover cardinality abuse only after dashboards slow down, costs spike, or a single service overwhelms shared telemetry budgets rather than through intentional platform governance.
How It Works in Practice
Controlling metric cardinality works best when teams manage telemetry at the source, at the collector, and at the platform boundary. The practical model is to define a small, stable set of approved dimensions, then reject or normalize everything else before it reaches long-term storage. This is consistent with current observability guidance from OpenTelemetry metrics guidance, which stresses dimensional discipline because every new label combination creates a new time series.
- Use allowlists for metric labels such as service name, region, and environment.
- Block or hash unbounded values like request IDs, session tokens, usernames, and full URLs.
- Set per-service series budgets so one workload cannot consume platform-wide capacity.
- Apply relabeling, drop rules, or metric transforms at the collector edge before storage.
- Review telemetry ownership the same way security teams review access to secrets or privileged accounts.
Operationally, the strongest control is ingestion prevention, because cleanup after storage still pays the cost of collection, indexing, and alert evaluation. Teams should also distinguish between debugging metrics and shared production metrics. Ad hoc metrics can exist in short-lived namespaces or isolated sandboxes, but they should not inherit the same retention, routing, or alerting rules as enterprise telemetry. Where shared observability is tied to incident response, the risk is not only expense; it is delayed detection when a platform becomes too noisy to trust. Guidance is evolving for how aggressively to cap cardinality in fast-moving environments, but current practice strongly favors explicit budgets over informal reviewer judgment. These controls tend to break down when application teams can emit arbitrary labels directly to the backend because enforcement then happens too late to protect cost or performance.
Common Variations and Edge Cases
Tighter cardinality control often increases operational overhead, requiring organisations to balance signal richness against ingestion cost and analyst convenience. That tradeoff becomes sharper in environments with ephemeral infrastructure, multi-tenant platforms, or aggressive service mesh instrumentation, where legitimate dimension growth can look like abuse. Best practice is evolving, and there is no universal standard for how many series a service should be allowed to generate.
Some edge cases need special handling. Kubernetes pod names, container IDs, and deploy hashes can create legitimate churn, but they should usually be aggregated into bounded service identifiers rather than emitted as raw labels. Similarly, security teams should be careful with labels that carry identity context, because pseudonymous values can still become high-cardinality identifiers and expose sensitive usage patterns. In those cases, governance should treat telemetry labels as controlled metadata with explicit business purpose, not as convenient debug fields.
For regulated or shared environments, it is also worth defining exceptions for incident forensics, but those exceptions should be time-bound and visible to platform owners. Policy-based enforcement patterns can help express these exceptions consistently, although they do not replace good schema design. Current guidance suggests that the safest model is to permit exceptions only in isolated pipelines, not in the default shared path, because once cardinality escapes into the main backend it is difficult to distinguish malicious misuse from ordinary application growth.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK address the attack and risk surface, while NIST CSF 2.0 and CIS Controls set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | GV.OC-01 | Shared telemetry needs ownership and risk decisions, not ad hoc metric sprawl. |
| MITRE ATT&CK | T1005 | Overly rich telemetry can leak data and create security exposure through observability paths. |
| CIS Controls | 8.2 | Asset and software inventory discipline supports bounded, accountable telemetry sources. |
Define telemetry owners and acceptable label schemas before allowing metrics into shared platforms.
Related resources from NHI Mgmt Group
- How should security teams include observability platforms in disaster recovery planning?
- How can security teams handle shared accounts without losing control?
- How should security teams recover observability platforms after a configuration loss?
- How should security teams centralize access control for analytics platforms?
