Join our Newsletter — 33% off our NHI Course

Datadog cost sprawl: what telemetry governance teams need to change

 

(@nhi-mgmt-group)
Member Moderator
Joined: 1 year ago
Posts: 16521
Topic starter  

TL;DR: Datadog bills balloon when high-cardinality tags, noisy logs, and over-indexed traces turn telemetry into a continuously expanding cost surface, according to Sawmills. The deeper issue is governance: human review loops cannot keep pace with developer- and agent-generated telemetry, so cost control has to move into the pipeline.

Editorial analysis by NHI Mgmt Group, based on content published by Sawmills: “Datadog Cost Optimization: One Tag Can 1000x Your Bill. Here's How to Find It and Stop the Bleed”.

Key questions

Q: How should teams stop high-cardinality telemetry from creating runaway Datadog cost?

A: Start by treating cardinality as a policy issue, not a cleanup issue.

Q: Why do manual Datadog cost reviews keep failing over time?

A: Because the telemetry source keeps changing faster than a human audit cadence can track.

Q: What are the signs that APM sampling is being misapplied?

A: The usual sign is that ingestion costs stay high even after teams think they reduced trace volume, or that the wrong traces are retained for search.

Practitioner guidance

  • Implement cardinality controls for custom metrics Inventory metrics by estimated timeseries count, then remove or aggregate tags with unbounded values such as request_id, pod_name, or user_id before they reach billing.
  • Move sampling policy into a shared collector Apply tail-based or service-specific sampling in a central control point so retention decisions are made once, consistently, and with full-trace context.
  • Route logs by query value Separate logs into indexed, infrequent-access, archive, and drop-at-ingestion paths so only streams with real retrieval value are indexed.

Bottom line: Datadog cost sprawl is caused by uncontrolled telemetry shape, not just excessive volume, and high-cardinality tags are the most expensive failure mode.

Explore further

View Full Forum →  |  NHI Foundation Course →  |  Our Services →  |  Read the full analysis →



   
Quote
(@mr-nhi)
Member Moderator
Joined: 5 months ago
Posts: 17012
 

Telemetry governance debt is the right name for Datadog cost sprawl: the problem is not a one-time cleanup failure, it is a control failure at the point telemetry is created. When developers and agents can emit new tags, spans, and logs without an upstream policy gate, cost becomes structurally unbounded. Practitioners should treat telemetry shaping as a standing governance control, not a periodic optimisation exercise.

A question worth separating out:

Q: When should organisations route logs away from indexed storage?

A: When a log stream is rarely searched, exists mainly for retention, or can be rehydrated from cheaper storage if needed. Health checks, stable access logs, and low-value debug noise are better dropped or archived than indexed. The decision should follow query frequency and operational value, not a blanket retention rule.

👉 Read our full editorial: Datadog cost sprawl is a telemetry governance problem, not a cleanup task



   
ReplyQuote
Share:

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.