Datadog cost grows in telemetry pipelines, not rate cards

At a glance

What this is: This article argues that Datadog cost is driven by telemetry growth across four meters, and that the real control point is upstream in the pipeline before data is billed.

Why it matters: It matters to IAM practitioners because the same governance pattern applies to identities, secrets, and access telemetry: if you do not control what enters the system, downstream cost and risk both compound.

By the numbers:

• Splunk's State of Observability 2025 surveyed 1,855 ITOps and engineering professionals.
• Datadog's log billing can charge for ingestion and indexing separately, with retention tiers of 3, 7, 15, or 30 days multiplying the indexed-event cost.

👉 Read Sawmills' analysis of what actually drives Datadog cost

Context

Datadog cost is not a static pricing problem. It is a telemetry governance problem in which log volume, metric cardinality, span counts, and host churn expand independently unless something upstream decides what should be collected, retained, and billed.

That matters because the same pattern shows up across IAM and NHI programmes: uncontrolled inputs create uncontrolled cost and uncontrolled exposure. When the control point sits only at the destination platform, teams are always reacting after the system has already expanded.

Sawmills frames the issue as a pipeline control problem, and that framing is typical for modern observability estates rather than an edge case.

Key questions

Q: How should teams control Datadog cost without cutting useful telemetry?

A: Start by governing the telemetry inputs rather than the bill itself. Filter noisy logs, strip unapproved metric labels, use tail-based sampling for traces, and route high-value records to the right retention tier. The goal is to preserve errors, audit data, and latency outliers while removing the volume that adds cost but not operational value.

Q: Why do custom metrics often become the biggest observability cost?

A: Because each distinct metric-and-tag combination is billable, so high-cardinality labels multiply cost far faster than traffic alone. A few extra dimensions can turn one metric into thousands of series. Teams need label governance, not just better dashboards, if they want cost to stay aligned with actual operational need.

Q: What breaks when logs and spans are sampled too late?

A: If sampling happens only after data has already crossed the ingestion layer, the expensive part is already paid for. Late controls can reduce retention load, but they cannot undo ingest cost. That is why effective observability governance places filtering and sampling in the pipeline before the billing meter sees the data.

Q: How do you know if telemetry cost controls are actually working?

A: Look for lower billable volume without losing incident value. Good signals include stable error coverage, reduced high-cardinality series, fewer retained routine logs, and trace sampling that still preserves outliers. If the bill falls but troubleshooting becomes harder, the control is too blunt and needs adjustment.

Technical breakdown

Log ingest and indexing create two separate cost paths

Datadog logs are expensive for two different reasons: ingestion and indexing. Ingestion charges apply to every gigabyte that arrives, while indexing charges apply only to logs kept searchable, with retention tier acting as a multiplier. That means a team can overspend either by sending too much low-value noise or by retaining too much of what it does index. The practical architecture question is not just what reaches Datadog, but what survives filtering, redaction, and routing in the collector tier before billing starts.

Practical implication: enforce log filtering and retention routing in the pipeline, not after the logs have already crossed the billing boundary.

Metric cardinality is the hidden multiplier in custom metrics

Custom metrics are billed by unique combinations of metric names and tag values, so cardinality is what turns a stable metric into a cost explosion. A useful metric with a few tags can become thousands of billable series when developers add user IDs, pod names, endpoint labels, or other high-variance dimensions. The control surface is upstream label governance. If the pipeline strips unapproved labels or allowlists only query-worthy tags, the billing surface stays aligned with actual operational need rather than raw instrumentation enthusiasm.

Practical implication: treat metric labels as governed data, and block unapproved high-cardinality tags before they become billable series.

Tail-based sampling controls span volume more precisely than head-based sampling

APM cost grows with span volume, and span volume scales with request fan-out across service graphs. Head-based sampling makes the keep-or-drop decision too early, which means you either lose the traces you needed or keep too many of the traces nobody inspects. Tail-based sampling evaluates the completed trace, so it can preserve errors and latency outliers while dropping routine successes. In practice, that is the difference between paying for forensic value and paying for indiscriminate fidelity.

Practical implication: move sampling decisions downstream in the trace pipeline so high-value spans are retained and routine traffic is sampled away.

NHI Mgmt Group analysis

Telemetry governance is now a control problem, not a procurement problem. The article shows that observability cost grows from data shape, not just vendor pricing. That is a familiar pattern in identity security too: when inputs are not governed, the downstream platform inherits the blast radius. For practitioners, the lesson is to move control upstream where data is created and tagged.

Cardinality drift is the named concept teams should watch. In this context, cardinality drift means the slow expansion of tag combinations, log variants, and trace dimensions until a once-manageable telemetry stream becomes a billing and operational liability. This is structurally similar to identity sprawl, where every new service, tag, or environment multiplies the management burden. The practitioner conclusion is simple: govern dimensions before they become permanent.

Pipeline controls matter more than destination platform features. The article correctly shifts emphasis from the observability vendor to the collector and routing layer. That is the same governance logic behind NHI and secrets management, where controls are most effective before credentials, logs, or spans reach a system of record. Teams should evaluate whether their control plane can actually constrain inputs continuously.

FinOps for telemetry needs an Operate phase, not just one-time cleanup. The article's strongest point is that static filters decay as services, tags, and incidents change. That is why cost governance must be continuous and policy-driven rather than dependent on periodic cleanup. For practitioners, the implication is that telemetry controls need ownership, review, and automation just like access controls do.

Identity teams should recognise the same pattern in machine and service accounts. Where telemetry growth is unmanaged, access and secret growth usually is too. The same operational discipline that caps logs and spans should also cap credential exposure, entitlement drift, and stale non-human identities. The conclusion for IAM leaders is to align governance of data flows with governance of identities, because both fail the same way when inputs are left unchecked.

What this signals

Cardinality drift will increasingly show up as both a cost problem and an access-governance problem, because telemetry, secrets, and entitlement data all grow fastest where developers have the least friction. That is why pipeline control should be treated as a standing operational control, not a cleanup task.

Teams that already run secrets governance should apply the same discipline to observability data. The control pattern is the same: define approved inputs, enforce them continuously, and review exceptions before they become normal.

If your programme already uses NIST SP 800-53 Rev 5 Security and Privacy Controls, the relevant lesson is to extend access, audit, and configuration controls into the telemetry path rather than assuming the vendor boundary is enough.

For practitioners

• Map each Datadog meter to an upstream control point Create a control map for logs, custom metrics, spans, and hosts so every billing driver has a specific pipeline decision owner. Use that map to decide where filtering, allowlisting, sampling, and routing should occur before data reaches the billing platform.
• Govern metric labels before they become billable series Define an approved label set for application metrics and remove or transform high-cardinality tags such as user_id, pod_name, and endpoint where they are not essential. Apply the rule in collectors or instrumentation libraries so the count never reaches the vendor unbounded.
• Use value-based routing for logs and spans Route audit-grade records to long-retention searchable storage, but sample or down-tier routine debug, health-check, and success-path telemetry. Keep errors and latency outliers at full fidelity so cost reduction does not destroy incident response value.
• Treat sampling policy as a living control Review sampling thresholds whenever service topology, traffic shape, or incident practice changes. A rule that worked for a monolith or a small microservice set will often fail once request fan-out and debug usage increase.

Key takeaways

• Datadog overspend is usually created upstream, where telemetry is emitted and tagged, not at the pricing page.
• High-cardinality labels, late sampling, and over-retention are the three most common reasons observability bills inflate faster than traffic.
• The durable fix is continuous pipeline governance, which preserves operational value while constraining what becomes billable.

Key terms

• Cardinality Drift: The gradual increase in metric tag combinations, log variants, or trace dimensions that turns a manageable telemetry stream into a cost and operations problem. It usually happens slowly, through ordinary developer changes, until the bill or the query surface becomes unbounded.
• Tail-Based Sampling: A trace sampling method that decides whether to keep a trace after the request completes. It preserves error and latency outliers more reliably than early sampling, which makes it useful when teams need value-based trace retention instead of blind volume reduction.
• Telemetry-driven governance: Telemetry-driven governance is a control approach that relies on runtime signals rather than periodic paperwork. For AI, that means watching drift, leakage, prompt anomalies, and other live indicators so governance decisions reflect current system behaviour instead of stale review findings.
• Billable Cardinality: The number of distinct metric series created by combinations of a metric name and its tags. In observability platforms that charge by series, this is often the most important driver of custom metric spend because every new tag combination can create another billable unit.

What's in the full article

Sawmills' full article covers the operational detail this post intentionally leaves for the source:

• The billing mechanics behind log ingest versus indexing, including how retention tiers change the final cost curve.
• The meter-by-meter breakdown for custom metrics, APM spans, and infrastructure hosts with concrete examples of where volume expands.
• The pipeline control examples for filtering, sampling, redaction, and routing that are needed to implement the article's cost model.
• The practical comparison of upstream telemetry control options for teams building a permanent governance process.

👉 The full Sawmills article explains the meter-by-meter control surface and the pipeline decisions that reduce spend.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It is designed for practitioners who need to connect identity controls to broader security and operational programmes.