Datadog cost sprawl is a telemetry governance problem, not a cleanup task

At a glance

What this is: This is an analysis of why Datadog spend keeps rebounding after cleanup efforts and why telemetry cost needs continuous governance, not periodic trimming.

Why it matters: It matters because platform, IAM, and security teams increasingly rely on telemetry pipelines that are shaped by developers and coding agents, so unchecked metadata and access patterns can create both financial and control risk.

By the numbers:

• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap.

👉 Read Sawmills' analysis of Datadog cost governance and telemetry sprawl

Context

Datadog cost control fails when teams treat telemetry as a one-time cleanup exercise rather than a governed flow. The underlying problem is that logs, metrics, and traces are continuously produced by services, developers, and increasingly coding agents, while the financial impact is only visible after the meter has already run.

For identity and access practitioners, the important parallel is that telemetry is behaving like an uncontrolled credential estate: tags, sampling rules, and log streams accumulate outside a lifecycle model. When access and data flow are not governed at the source, the platform team inherits the bill and the risk after the fact.

Key questions

Q: What breaks when high-cardinality telemetry tags are not controlled?

A: Billable timeseries expand multiplicatively, so a single metric can become thousands or millions of unique series. That drives cost far faster than most teams expect and makes manual cleanup ineffective because the issue is embedded in the data shape, not the number of metrics alone.

Q: When should organisations move telemetry cost control into the pipeline?

A: They should do it as soon as multiple teams or agents can create instrumentation independently. At that point, periodic audits no longer keep pace with the rate of change, and cost governance has to happen where telemetry is emitted and shaped.

Q: What do security and platform teams get wrong about telemetry spend?

A: They often treat cost as a retention problem or a quarterly cleanup task. In reality, the main issue is governance over what gets emitted, indexed, and retained, especially when agent-generated code can replicate expensive patterns across many services.

Q: How should teams govern telemetry when coding agents are creating instrumentation?

A: Use policy checks for new tags, metric dimensions, and log fields before code is merged or deployed. Agents can generate plausible defaults at machine speed, so the control must shift from human review alone to automated enforcement at the pipeline boundary.

Technical breakdown

Why high-cardinality tags explode custom metric cost

Datadog charges custom metrics by unique timeseries, which means each tag combination becomes part of the billable surface. A metric tagged by endpoint, pod name, status, and service can multiply into hundreds of thousands or millions of distinct series. The real issue is not volume alone but unbounded cardinality, where a single field such as request_id or pod_name becomes a cost amplifier. This is especially dangerous in ephemeral environments because autoscaling continuously creates new values.

Practical implication: remove or aggregate the highest-cardinality tags before they reach the billing layer.

How APM sampling and log indexing create hidden spend

APM cost is often split between ingested spans and indexed spans, so teams may reduce one while the other continues to accumulate cost. Head-based sampling decides too early, usually before an error or latency spike is known, while tail-based sampling preserves the traces that matter for troubleshooting. Logs follow a similar pattern: indexing every stream is expensive, but not every stream needs searchability. Routing decisions should distinguish active incident data from archival or noise.

Practical implication: set sampling and indexing policy by business value, not by default agent settings.

Why agent-generated telemetry makes manual controls fail

Coding agents change the scale and speed of telemetry creation. They can scaffold services, add instrumentation, and copy example patterns that look observability-friendly but create expensive tags. That breaks the old assumption that a human reviewer can catch bad telemetry before it lands. The control gap is lifecycle governance, because the cost decision is made at emission time, not at audit time. Once the pattern is copied across services, the bill compounds faster than quarterly reviews can respond.

Practical implication: enforce telemetry policy in the pipeline so generated code cannot bypass cost controls.

Threat narrative

Attacker objective: The objective is not theft but uncontrolled resource consumption, where telemetry growth outpaces governance and turns observability into a budget drain.

1. Entry occurs when developers or coding agents introduce high-cardinality tags, excessive spans, or noisy log streams into production telemetry.
2. Escalation follows as each new service and tag value multiplies billable timeseries, indexed spans, or log volume without any central control point.
3. Impact is a sustained cost blowout that persists until telemetry shaping is enforced upstream rather than cleaned up after the invoice arrives.

NHI Mgmt Group analysis

Telemetry governance debt is now a cost-control problem. The article shows that spend does not rebound because teams fail once, but because the source of cost keeps changing faster than review processes can respond. That is the same structural pattern NHIMG sees in NHI estates where lifecycle drift, not a single misconfiguration, creates persistent exposure. Practitioners should treat telemetry shaping as a governed control layer, not a cleanup task.

Agentic code generation creates a new cost-amplification pattern. When coding agents add instrumentation, they can reproduce expensive defaults at machine speed and across many services. That means the control point shifts from code review to policy enforcement at emission time, which is closer to how NHI governance has to work for machine-generated identities and secrets. Teams should assume that unreviewed defaults will scale faster than human audit cycles.

Cardinality is the named concept that explains why observability bills spiral. High-cardinality metadata turns one metric into a large number of billable series, and the effect is multiplicative rather than linear. The operational lesson is that governance must target the field values that expand the series count, not just the number of metrics. Practitioners should identify which tags behave like unbounded identifiers and suppress them before they become recurring cost.

Lifecycle control matters more than periodic cleanup. Manual exclusion filters and ad hoc audits can reduce spend temporarily, but they do not change the source of the problem. That is the same governance failure seen in identity programmes that rely on after-the-fact review instead of continuous control. Teams should move cost policy into the pipeline so shaping happens as telemetry is created.

Identity teams should recognise the access-control analogy here. Who can emit telemetry, what tags they can attach, and which streams are allowed to be indexed are governance questions, not just platform tuning questions. In practice, that maps to access boundaries, policy enforcement, and evidence of control. Practitioners should align telemetry governance with their broader control model rather than leaving it as an engineering preference.

What this signals

Cardinality pressure is a governance signal, not just a cost signal. When teams allow unbounded tags to proliferate, they are effectively permitting a form of uncontrolled metadata access that changes the economics of observability. The practical response is to treat telemetry policy as part of the control plane, with explicit approval boundaries for new dimensions and streams.

As coding agents become more common, telemetry governance needs to look more like identity governance for machine-generated activity. The organisation that can enforce policy at the point of emission will control both spend and blast radius better than the organisation that relies on after-the-fact reviews. For teams already working from Top 10 NHI Issues, this is the same lifecycle logic applied to operational data.

For practitioners

• Implement cardinality limits at ingestion Block or aggregate tags such as pod_name, request_id, user_id, and container_id before they create billable series. Use automated checks on new instrumentation so unbounded dimensions cannot enter production without approval.
• Separate sampling policy from agent defaults Define head-based and tail-based sampling rules by service criticality and error sensitivity. Preserve searchability for incident-relevant traces while reducing routine successful traffic before it reaches the backend meter.
• Classify log streams by query value Index only streams that are actively searched during incidents or investigations. Route rarely queried data to cheaper storage, and drop pure noise at ingestion rather than paying to retain it.
• Add telemetry governance to CI and code review Treat new metrics, tags, and log fields as policy-controlled changes. Require review for any instrumentation that introduces new dimensions or expands existing cardinality beyond defined thresholds.
• Move cost controls into a continuous pipeline Use an enforcement layer that shapes telemetry continuously as services change, so the organisation does not depend on quarterly audits to find the next expensive tag or log stream.

Key takeaways

• Datadog overspend is driven by telemetry shape, not just telemetry volume.
• High-cardinality tags and default sampling choices can multiply cost faster than human review cycles can correct.
• Continuous pipeline enforcement is the durable control because it governs telemetry at the point of creation.

Key terms

• Cardinality: Cardinality is the number of distinct values a metric tag or dimension can take. In observability systems, high cardinality multiplies unique timeseries and therefore cost, because each unique combination can become billable data rather than a simple label.
• Tail-Based Sampling: Tail-based sampling decides whether to keep a trace after it completes, based on what actually happened. That allows teams to retain errors, slow requests, and unusual paths while discarding routine traces that add storage and indexing cost without improving diagnosis.
• Telemetry-driven governance: Telemetry-driven governance is a control approach that relies on runtime signals rather than periodic paperwork. For AI, that means watching drift, leakage, prompt anomalies, and other live indicators so governance decisions reflect current system behaviour instead of stale review findings.
• Agent-Generated Instrumentation: Agent-generated instrumentation is telemetry code produced or modified by coding agents rather than humans. It can scale quickly, replicate expensive defaults, and bypass the informal cost checks that usually happen during human code review, so it requires automated policy controls.

What's in the full article

Sawmills' full analysis covers the operational detail this post intentionally leaves for the source:

• API-based cardinality auditing workflow for ranking the most expensive metrics
• Sample Datadog filtering and sampling patterns for logs, custom metrics, and APM traces
• Pipeline-level shaping logic for reducing billable telemetry before it reaches Datadog
• Implementation guidance for teams moving from quarterly cleanup to continuous control

👉 The full Sawmills post covers the cardinality workflow, sampling decisions, and pipeline controls in more operational detail.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, IAM, and secrets management. It helps practitioners build lifecycle controls that scale across identity programmes and the systems they support.