Observability cost keeps rising because telemetry growth is unmanaged

At a glance

What this is: This is an analysis of why observability bills keep climbing, with the central finding that unmanaged telemetry growth, not vendor pricing alone, is the real cost driver.

Why it matters: It matters to IAM and security practitioners because the same governance failure that lets telemetry sprawl persist also shows up in identity, secrets, and machine access programmes: volume grows faster than control.

By the numbers:

• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap.

👉 Read Sawmills' analysis of why observability bills keep rising

Context

Observability cost is a telemetry governance problem, not just a pricing problem. Modern systems generate logs, metrics, and traces as a byproduct of normal operation, and every new service or deployment increases the amount of data that could be collected, retained, indexed, and queried.

For security and identity practitioners, the broader lesson is familiar: uncontrolled growth becomes expensive long before it becomes visible. In NHI, IAM, and secrets management, the same pattern appears when teams add controls or telemetry without lifecycle governance, then inherit the operational cost later.

The article argues that switching vendors does not solve the underlying issue because the telemetry stream itself is expanding. That starting point is typical in cloud-native environments, where teams optimize for debugging speed first and governance after the fact.

Key questions

Q: What breaks when telemetry governance is not in place?

A: Without telemetry governance, logs, metrics, and traces expand faster than teams can review or pay for them. The result is higher ingestion, indexing, and retention cost, plus lower signal quality because noisy data crowds out useful data. Over time, the organisation pays more while understanding less.

Q: Why do high-cardinality metric tags create such a cost problem?

A: High-cardinality tags multiply one metric into many distinct time series, which increases storage, query load, and alerting overhead. The issue is not the metric itself but the uniqueness of the labels attached to it. Identifiers that look useful in code often become expensive at scale.

Q: How do teams know if endpoint telemetry control is actually working?

A: Look for fewer duplicate events, lower collection overhead, faster routing to the right destination, and a clear audit trail for policy changes. If the pipeline still depends on manual filtering, or if teams cannot explain why certain data was captured or suppressed, the control is not mature enough for modern detection and compliance needs.

Q: Who should own observability cost control across engineering and platform teams?

A: Ownership should sit with a shared control function, because developers create the telemetry and platform teams absorb the cost. The practical model is joint accountability: engineering owns instrumentation quality, while platform or SRE owns policy, review, and enforcement. Cost control fails when neither side can change source behavior.

Technical breakdown

Why telemetry volume compounds faster than teams expect

Telemetry is not a single data stream. A single request can generate logs, metrics, and traces across gateways, services, queues, databases, and third-party calls, and each layer multiplies the volume. The problem is structural: instrumentation is cheap at creation time, while the cost of ingest, indexing, retention, transport, and query support accumulates later. That means observability systems are excellent at collecting data but poor at deciding whether the data deserved to exist in the first place.

Practical implication: move control upstream so only value-bearing telemetry reaches expensive storage and search layers.

How metric cardinality turns small tags into major cost

Cardinality is the number of unique tag combinations attached to a metric. A few stable labels are manageable, but user IDs, request IDs, pod names, and dynamic URLs can explode one metric into thousands or millions of time series. Kubernetes and other ephemeral environments make this worse because tag values churn constantly. The failure mode is subtle: developers add detail to make a metric more useful, but in production that detail creates persistent billing and performance pressure.

Practical implication: enforce cardinality limits centrally and keep high-uniqueness identifiers out of metric labels.

Why APM sampling decisions are hard to keep aligned with reality

APM traces are trees, not single events. One request can fan out into dozens of spans, and full capture quickly becomes expensive in high-traffic services. Head-based sampling is easy but blunt, because it decides too early and can miss the slow or anomalous paths teams care about. Tail-based sampling is better because it preserves errors and outliers, but it is more operationally complex. The key issue is not trace collection itself, but whether sampling follows value instead of fear.

Practical implication: use adaptive sampling policies that preserve errors and latency outliers without retaining every routine trace.

NHI Mgmt Group analysis

Telemetry sprawl is the observability equivalent of unmanaged identity sprawl. The article is right to frame the cost problem as a governance problem, not a vendor problem. When every team can emit more logs, tags, and traces without lifecycle controls, the platform becomes a dumping ground for low-value data. The practitioner lesson is the same one identity teams learned with standing access and duplicate accounts: creation is easy, cleanup is where control fails.

Metric cardinality is a named governance failure, not a tuning detail. Once identifiers such as user_id or pod_uid enter metric labels, the control surface shifts from cost management to uncontrolled dimensional growth. In practice, that means finance sees the bill before platform teams see the cause. Practitioners should treat cardinality limits as policy, not preference.

Telemetry management should sit before the observability vendor, not inside it. The strongest point in the article is that downstream tools cannot undo upstream waste. That mirrors NHI governance, where vaulting or monitoring after the fact cannot compensate for weak issuance, retention, or offboarding. The control boundary matters more than the destination.

Agentic telemetry control is a natural next step for complex estates. Static filters decay as services change, just as static access reviews miss dynamic machine behavior. A system that can detect waste in context and adapt rules over time is more consistent with modern cloud operations. Practitioners should think in terms of continuous telemetry governance, not one-time cleanups.

What this signals

Telemetry governance is becoming a control-plane discipline, not a tooling preference. Teams that still treat observability as a vendor selection problem will keep paying for uncontrolled data growth. The better model is policy-led collection, where the pipeline decides what arrives, what is sampled, and what is discarded before cost is incurred.

Metric cardinality deserves the same governance attention as secret sprawl. Once high-uniqueness identifiers enter shared telemetry, costs rise in ways that are hard to reverse quickly. For identity-led programmes, that is a useful analogy because machine identity estates also grow through small additions that become expensive when left unmanaged.

The most practical response is to connect telemetry policy to platform change control, incident retrospectives, and release engineering. That keeps noisy logging, over-detailed tags, and routine trace capture from becoming permanent defaults while preserving the signals needed for troubleshooting.

For practitioners

• Set telemetry budgets by service and signal type Define per-service limits for logs, metrics, and traces, then make owners accountable for staying within budget. Tie budget exceptions to incidents or explicit debugging windows rather than leaving high-volume telemetry on indefinitely.
• Remove high-cardinality fields from metrics Block identifiers such as user_id, session_id, request_id, pod_uid, and raw URLs from metric labels unless there is a documented aggregation need. Keep those values in logs or traces where they can be sampled and controlled more safely.
• Apply adaptive sampling to APM traces Keep full-fidelity capture for errors, latency outliers, and rare conditions, but reduce collection of routine traffic. Revisit sampling rules when traffic patterns, service topology, or retry behavior changes.
• Create an upstream telemetry review workflow Review new instrumentation before it reaches production billing paths, especially during feature launches, incident retros, and Kubernetes changes. A short approval path for new tags and debug logging prevents small additions from becoming permanent cost multipliers.

Key takeaways

• Observability bills rise because telemetry growth is unmanaged, not because one vendor charges too much.
• Metric cardinality, trace volume, and retention choices create compounding cost pressure long before teams notice the source.
• Upstream telemetry governance is the durable fix, because downstream platforms cannot undo data that has already been collected.

Key terms

• Telemetry sprawl: Telemetry sprawl is the accumulation of too many logs, metrics, and events across too many systems for security teams to use effectively. It usually happens when collection decisions are driven by fear of missing something rather than by a clear detection or investigation purpose.
• Cardinality: The number of unique attribute combinations in telemetry data. High cardinality increases the number of metric series and can inflate cost, overwhelm backends, and make analysis noisy unless attributes are normalised or removed before export.
• Tail-Based Sampling: A trace sampling method that decides whether to keep a trace after the request completes. It preserves error and latency outliers more reliably than early sampling, which makes it useful when teams need value-based trace retention instead of blind volume reduction.

What's in the full article

Sawmills' full article covers the operational detail this post intentionally leaves for the source:

• A practical breakdown of how telemetry growth shows up across logs, metrics, and traces in modern engineering environments
• Examples of where cost pressure accumulates in ingestion, indexing, retention, and query paths rather than in storage alone
• The vendor's operating model for keeping telemetry useful while reducing waste before it reaches the observability platform
• Why upstream control matters more than a platform swap when engineering teams are already embedded in existing workflows

👉 The full Sawmills article covers telemetry sprawl, cardinality pressure, and upstream control in more operational detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and identity lifecycle control. It helps security and identity practitioners build the governance discipline that keeps growth from outrunning control.