Why do custom metrics often become the biggest observability cost?

Why This Matters for Security Teams

Custom metrics are attractive because they promise sharper detection, faster triage, and better service-level visibility. The problem is that observability platforms usually charge by time series volume, not by the conceptual value of a metric. When teams add labels for pod, region, customer, request type, device class, or tenant without governance, the metric footprint can expand quietly and persist long after the original use case has faded. That turns telemetry into a budget risk, not just an engineering detail.

For security teams, this matters because observability data often supports incident response, threat hunting, and control validation. If costs rise unpredictably, organisations may respond by sampling more aggressively, shortening retention, or turning off the very signals that help confirm whether a control is working. Current guidance from the NIST Cybersecurity Framework 2.0 reinforces that measurement should support risk management outcomes, not create avoidable operational friction. In practice, many security teams discover observability waste only after a bill spike or a post-incident review reveals that dashboards were built without any label ownership or expiry discipline.

How It Works in Practice

The cost mechanism is straightforward: every unique combination of metric name and tag set becomes a separate series. A counter with three stable dimensions may be cheap, while the same counter with an unconstrained user ID, request path, and container identifier can explode into thousands or millions of active series. That increases ingestion, storage, query latency, and alert noise. The operational issue is not the metric itself; it is the cardinality of the labels attached to it.

Effective control usually starts with a metric catalogue and a naming standard. Teams define which labels are approved, which are conditional, and which are prohibited because they are too granular or too volatile. Good practice is evolving, but most mature programmes also add review gates for new dashboards and instrumentation changes. In security-sensitive environments, the same discipline should apply to logs and traces so telemetry governance is consistent across signals.

• Use low-cardinality dimensions such as service, region, environment, and coarse status codes.
• Avoid identifiers that are effectively unique per event, such as session IDs, request IDs, and full URLs.
• Set ownership for each custom metric so someone is accountable when cost grows.
• Apply retention and sampling rules based on investigative value, not convenience.
• Review alerts that reference highly dimensional metrics, because they often amplify cost and noise together.

Security teams should also align observability choices with control objectives. For example, a detection rule that depends on a high-cardinality label may be cheaper to implement in logs or SIEM than as a continuously emitted metric. Where telemetry is used for identity or access monitoring, label design should preserve useful context without exposing sensitive personal data unnecessarily, especially under privacy and retention constraints. These controls tend to break down when developers can add new labels directly in production because cardinality growth becomes invisible until query performance and spend both degrade.

Common Variations and Edge Cases

Tighter metric governance often increases engineering overhead, requiring organisations to balance cost control against developer autonomy and diagnostic depth. That tradeoff is real, especially in fast-moving product environments where teams want to instrument first and optimise later. The practical answer is not to ban custom metrics, but to distinguish between stable operational dimensions and exploratory debug data that belongs in short-lived traces or logs.

There is no universal standard for what label set is acceptable, because the right threshold depends on platform pricing, workload volatility, and query patterns. For regulated environments, telemetry choices may also need to reflect audit retention, incident reconstruction, and privacy requirements. In some cases, a metric that looks expensive is still justified if it materially improves detection or service assurance. In others, a cheap metric becomes costly once thousands of tenants, devices, or microservices inherit the same pattern.

Frameworks such as NIST Cybersecurity Framework 2.0 help teams treat observability as part of governance and resilience rather than an isolated tooling choice. For organisations with cloud-heavy estates, the same logic extends to release controls, infrastructure-as-code reviews, and alert hygiene. The key question is whether each custom metric has a named business or security purpose. If not, it is usually a candidate for reduction, aggregation, or removal before it becomes the largest line item in the monitoring budget.

Related resources from NHI Mgmt Group

• Why do unused licenses often become a security issue as well as a cost issue?
• Why do npm supply chain attacks often become NHI governance failures?
• Why do onboarding workflows often become an IAM control problem?
• When do custom IAM roles become a governance problem?