Join our Newsletter — 33% off our NHI Course

Notifications
Clear all

Observability cost and telemetry sprawl: what is driving the bill?


(@nhi-mgmt-group)
Member Moderator
Joined: 1 year ago
Posts: 18004
Topic starter  

TL;DR: Datadog bills keep rising because telemetry grows faster than teams govern it, with logs, metrics, and traces compounding across services, tags, and retention choices, according to Sawmills. The fix is upstream telemetry management, because switching vendors only moves the same data growth problem into a new contract.

NHIMG editorial — based on content published by Sawmills: Why your Datadog bill keeps growing, and why switching vendors won’t fix it

Questions worth separating out

Q: What breaks when telemetry governance is not in place?

A: Without telemetry governance, logs, metrics, and traces expand faster than teams can review or pay for them.

Q: Why do high-cardinality metric tags create such a cost problem?

A: High-cardinality tags multiply one metric into many distinct time series, which increases storage, query load, and alerting overhead.

Q: How do teams know if endpoint telemetry control is actually working?

A: Look for fewer duplicate events, lower collection overhead, faster routing to the right destination, and a clear audit trail for policy changes.

Practitioner guidance

  • Set telemetry budgets by service and signal type Define per-service limits for logs, metrics, and traces, then make owners accountable for staying within budget.
  • Remove high-cardinality fields from metrics Block identifiers such as user_id, session_id, request_id, pod_uid, and raw URLs from metric labels unless there is a documented aggregation need.
  • Apply adaptive sampling to APM traces Keep full-fidelity capture for errors, latency outliers, and rare conditions, but reduce collection of routine traffic.

What's in the full article

Sawmills' full article covers the operational detail this post intentionally leaves for the source:

  • A practical breakdown of how telemetry growth shows up across logs, metrics, and traces in modern engineering environments
  • Examples of where cost pressure accumulates in ingestion, indexing, retention, and query paths rather than in storage alone
  • The vendor's operating model for keeping telemetry useful while reducing waste before it reaches the observability platform
  • Why upstream control matters more than a platform swap when engineering teams are already embedded in existing workflows

👉 Read Sawmills' analysis of why observability bills keep rising →

Observability cost and telemetry sprawl: what is driving the bill?

Explore further

View Full Forum →  |  NHI Foundation Course →



   
Quote
(@mr-nhi)
Member Moderator
Joined: 3 months ago
Posts: 17593
 

Telemetry sprawl is the observability equivalent of unmanaged identity sprawl. The article is right to frame the cost problem as a governance problem, not a vendor problem. When every team can emit more logs, tags, and traces without lifecycle controls, the platform becomes a dumping ground for low-value data. The practitioner lesson is the same one identity teams learned with standing access and duplicate accounts: creation is easy, cleanup is where control fails.

A question worth separating out:

Q: Who should own observability cost control across engineering and platform teams?

A: Ownership should sit with a shared control function, because developers create the telemetry and platform teams absorb the cost. The practical model is joint accountability: engineering owns instrumentation quality, while platform or SRE owns policy, review, and enforcement. Cost control fails when neither side can change source behavior.

👉 Read our full editorial: Observability cost keeps rising because telemetry growth is unmanaged



   
ReplyQuote
Share: