Observability data waste is raising costs and slowing incident response

At a glance

What this is: Sawmills says observability programmes are generating too much low-value telemetry, with most collected data never used and incident response getting slower as a result.

Why it matters: For SRE and security teams, telemetry sprawl is now a governance problem because excess data raises cost, obscures anomalies, and makes triage and containment harder.

By the numbers:

• Only 13% of collected telemetry is used.
• 84% of companies use less than a quarter of the data they ingest.
• 86% of companies run two or more observability platforms.

Context

Observability data waste is the gap between what teams collect and what they can actually use during troubleshooting. When telemetry volume grows faster than governance, engineers inherit noisy dashboards, rising storage and processing costs, and slower incident resolution. The core issue is not simply more data, but unmanaged telemetry pipelines that blur the signals required for operations.

This matters to SRE and security teams because incident response depends on fast signal extraction, not indiscriminate retention. In identity-heavy environments, excessive logs can also obscure authentication failures, privilege changes, and anomalous access patterns, making it harder to trace security events alongside service degradation.

Key questions

Q: What breaks when observability teams collect everything?

A: Collecting everything turns observability into a storage and search problem, not a diagnostic one. Noise increases, signal quality drops, and engineers spend more time filtering irrelevant telemetry than resolving the incident. The result is slower triage, higher spend, and less confidence in whether the data set is actually telling the truth.

Q: Why do fragmented telemetry sources slow down incident response?

A: Because no single source tells the full story. Identity logs, endpoint telemetry, cloud events, and network data often only become meaningful when they are correlated together, and manual correlation takes time. If the data is split across silos, analysts spend more time gathering evidence than making decisions, which increases dwell time and inconsistency.

Q: What are the signs that observability data is no longer useful?

A: Common signs include large volumes of unused telemetry, rising ingestion and storage costs, duplicated platforms, and repeated complaints that dashboards are crowded but incidents still take too long to diagnose. If teams cannot point to specific decisions supported by each telemetry source, the pipeline is probably over-collected.

Q: How should teams reduce observability costs without losing useful telemetry?

A: Start at the pipeline, not the backend. Filter obvious noise, remove high-cardinality fields, and tail-sample traces so routine traffic is reduced after full context is available. That preserves error and latency signal while cutting ingest, storage, and query costs before they become locked into the observability bill.

Technical breakdown

Why telemetry sprawl degrades incident response

Telemetry sprawl occurs when logs, traces, and metrics accumulate without a clear decision rule for what to keep, enrich, or discard. High-cardinality metrics and duplicate log streams inflate cost while making it harder to isolate the few events that explain a failure. The practical result is that observability systems become search problems instead of detection systems, and engineers spend more time filtering noise than validating root cause.

Practical implication: define telemetry retention and routing rules so incident-critical signals remain searchable while low-value noise is dropped or sampled.

Smart telemetry as a control layer, not just a dashboard

A smart telemetry layer sits between producers and analysis tools, classifying data in real time and deciding whether a record is useful for debugging, compliance, or trend analysis. That is a control function, not a visualisation feature. In mature environments, this layer can reduce redundant ingestion, preserve high-value events, and support faster triage without forcing teams to buy more storage for every new data source.

Practical implication: treat telemetry filtering and enrichment as a governed pipeline with explicit ownership, not an ad hoc platform setting.

Where AI fits in telemetry operations

AI can help cluster anomalies, identify repetitive noise, and prioritise events that correlate with incidents, but it only works when the underlying telemetry is clean enough to learn from. Poor data quality turns automation into another layer of confusion. For SRE and security operations, the better use of AI is selective reduction and correlation, not blind accumulation of more signals for the model to sort out later.

Practical implication: use AI to reduce and correlate telemetry only after you have defined minimum signal quality and routing standards.

NHI Mgmt Group analysis

Telemetry waste has become an operational governance failure, not a tooling inconvenience. When most collected data is never used, observability stops serving incident response and starts taxing it. The real issue is that teams often optimise for collection capacity while neglecting signal quality, ownership, and retention policy. Practitioners should treat telemetry governance as a control plane for operational reliability.

Observability sprawl creates a blind spot by overwhelming the very controls meant to improve visibility. Excess logs and traces can hide the events that matter, especially during outages or suspicious access patterns. That is why the useful unit of management is not raw volume but signal-to-noise ratio. Teams should align data collection to diagnostic value, not platform appetite.

AI-assisted telemetry management will only be credible if data discipline comes first. Automation can reduce noise, but it cannot compensate for undifferentiated data estates or duplicated pipelines. The named concept here is telemetry governance debt: the accumulated cost of collecting data faster than teams can classify, route, and discard it. Practitioners should address that debt before layering on more automation.

For identity-adjacent operations, observability quality is part of access and anomaly detection. Authentication failures, unusual privilege changes, and workload-to-workload access events become harder to detect when the telemetry pipeline is flooded. Security teams should see telemetry control as a prerequisite for reliable detection engineering, not a separate performance topic.

Cost control and incident response are now the same conversation. The organisations that keep collecting indiscriminately will continue to spend more while resolving incidents more slowly. That trade-off is no longer sustainable for engineering or security programmes. Practitioners should make telemetry reduction a reliability and governance objective, not just a finance exercise.

What this signals

Telemetry governance debt: the cost of collecting more data than the organisation can classify, route, and justify. Teams that ignore it end up paying for storage and processing that do not improve detection or troubleshooting, which is why reduction and routing deserve the same control attention as collection.

Observability programmes should be measured by how quickly they surface the few events that matter, not by how much data they retain. For security and SRE teams, the practical test is whether telemetry helps separate service failure, suspicious access, and routine noise before response decisions are made.

For practitioners

• Audit telemetry value by source and use case Map logs, traces, and metrics to the incidents and workflows they actually support, then remove or sample data that has no clear operational consumer.
• Set retention and routing policies for high-volume signals Apply explicit rules for duplicate logs, high-cardinality metrics, and low-value debug output so expensive data does not flow into long-term storage by default.
• Create ownership for telemetry pipelines Assign a team to govern ingestion, enrichment, and discard rules so observability sprawl is managed as a lifecycle problem rather than a platform setting.
• Use AI to reduce noise, not to hide poor data discipline Pilot anomaly clustering and correlation only after you have defined minimum signal quality thresholds and validated that the resulting outputs improve triage speed.

Key takeaways

• Most observability environments now contain a large amount of low-value data that adds cost without improving troubleshooting outcomes.
• When telemetry volume outpaces governance, the main operational risk is slower incident resolution because critical signals are harder to find.
• The right response is not to collect less by default, but to manage telemetry as a governed pipeline with explicit value, retention, and routing rules.

Key terms

• Telemetry-driven governance: Telemetry-driven governance is a control approach that relies on runtime signals rather than periodic paperwork. For AI, that means watching drift, leakage, prompt anomalies, and other live indicators so governance decisions reflect current system behaviour instead of stale review findings.
• Signal-to-Noise Ratio: The balance between meaningful security events and routine activity in detection tooling. A weak ratio makes analysts spend more time filtering alerts and less time identifying real attacks, which is why architecture quality strongly affects SOC effectiveness.
• High-cardinality Data: High-cardinality data contains many unique combinations of tags or labels, which makes querying and storage more expensive. In observability, this can reduce the usefulness of telemetry. For identity teams, it also makes access evidence harder to search and correlate at incident speed.
• Telemetry sprawl: Telemetry sprawl is the accumulation of too many logs, metrics, and events across too many systems for security teams to use effectively. It usually happens when collection decisions are driven by fear of missing something rather than by a clear detection or investigation purpose.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. Explore nhimg.org for resources that connect identity governance to the broader security disciplines your programme depends on.