Telemetry pipelines are the missing control layer for observability costs

At a glance

What this is: A telemetry pipeline sits between emitting services and observability backends, and the article argues that filtering, sampling, transformation, and routing decisions belong there before costs hit the meter.

Why it matters: For IAM and security practitioners, the same control logic that governs telemetry waste is relevant to identity telemetry, privileged activity logging, and workload signal governance across NHI and human identity programmes.

By the numbers:

• 74% of respondents now treat cost as a top priority when selecting observability tools.
• 41% increased their observability spend in the past year.

👉 Read Sawmills' analysis of telemetry pipelines as the observability control layer

Context

Telemetry pipelines are the control layer that decides which observability data survives, what shape it takes, and how much teams pay to keep it. In the article's framing, the problem is not backend storage alone but upstream governance, because cost, noise, and reliability are already being locked in before data reaches Datadog, Grafana Cloud, or Splunk.

The technical issue is familiar to identity and platform teams alike: control belongs where the signal is still editable. That is the same governance pattern seen in secrets management and workload identity, where the upstream decision is usually cheaper and safer than trying to correct the outcome after ingestion or exposure.

Key questions

Q: How should teams reduce observability costs without losing useful telemetry?

A: Start at the pipeline, not the backend. Filter obvious noise, remove high-cardinality fields, and tail-sample traces so routine traffic is reduced after full context is available. That preserves error and latency signal while cutting ingest, storage, and query costs before they become locked into the observability bill.

Q: Why do telemetry pipelines matter for governance rather than just engineering efficiency?

A: Because they decide what data is allowed to exist in the observability system at all. That makes them a policy layer for signal quality, cost control, and reliability. If the pipeline is poorly governed, teams pay for noise, lose incident context, or create brittle operational dependencies.

Q: What breaks when telemetry pipelines are not treated as sensitive data flows?

A: Secrets, tokens, and identity context can replicate across brokers, caches, and SIEM systems before redaction occurs, which means attackers may harvest usable credentials long before defenders notice. The failure is not just leakage. It is persistence of exposed identity material across multiple hops, which expands the blast radius and complicates containment.

Q: How should security teams think about telemetry pipeline reliability during incidents?

A: Treat the pipeline as critical infrastructure. Backpressure, memory limits, and buffering need to be tuned so severe logs and audit evidence survive backend slowness, while low-value streams can degrade gracefully. Otherwise, the organisation either loses evidence or destabilises the systems producing it.

Technical breakdown

What a telemetry pipeline does between ingest and export

A telemetry pipeline is not a passive forwarder. It is the processing layer that receives logs, metrics, and traces, then filters, samples, transforms, enriches, and exports them according to policy. In OpenTelemetry terms, receivers ingest data, processors manipulate it in memory, and exporters ship it onward. That matters because the cost and quality decisions are made before backend storage, indexing, and retention charges begin. Once telemetry reaches the backend, the expensive parts are already committed.

Practical implication: place governance rules upstream, where data can still be dropped, reshaped, or routed before billing and retention apply.

Why processing, not ingest, controls observability spend

The middle stage is where waste is converted into cost. Filtering removes telemetry that should never have existed, while transform logic strips high-cardinality fields that create explosive metric growth. Tail sampling is especially important for traces because it lets teams keep errors and slow requests after the full span is known, rather than guessing at the start. Without that processing layer, an observability stack becomes an expensive relay that pays to ingest everything and learn nothing new from most of it.

Practical implication: tune filters, transformations, and tail sampling before negotiating backend pricing or storage tiers.

Agent and gateway patterns in OpenTelemetry deployments

Most serious deployments split the pipeline into two tiers. Agents run close to workloads, usually as DaemonSets, and handle node-local work such as log tailing, initial filtering, redaction, and buffering. Gateways sit centrally and make decisions that need a global view, including tail sampling, cross-service routing, and final export. Tail sampling has a hard architectural constraint: all spans for a trace must meet in one place before the decision is made, which is why it belongs at the gateway, not on each node.

Practical implication: use edge agents for local hygiene and central gateways for policy decisions that require whole-trace visibility.

NHI Mgmt Group analysis

Telemetry pipeline governance is becoming a control problem, not a tooling problem. The article correctly treats the pipeline as the place where value and waste are separated before cost is incurred. That is a governance pattern security and identity teams should recognise, because once noisy or over-permissive telemetry lands in a backend, the organisation has already paid for its collection and often its retention. Practitioners should think about upstream decision rights, not just downstream storage efficiency.

High-cardinality telemetry is the observability equivalent of identity sprawl. When dynamic labels, user IDs, pod names, and request IDs multiply into millions of series, the system is no longer measuring reality cleanly. It is manufacturing operational overhead. That is conceptually close to unmanaged identity growth in IAM and NHI programmes, where uncontrolled uniqueness creates more governance burden than signal. Teams should treat cardinality as a lifecycle problem, not a dashboards problem.

Tail sampling creates a useful but fragile trust boundary in the pipeline. The article shows why decisions about which traces matter cannot be made at the start of a request. That same logic applies to identity telemetry and privileged workflows, where the decisive event is often only visible after the sequence completes. The named concept here is decision-late control: the point at which policy is applied must be late enough to preserve context, but early enough to prevent waste. Practitioners should design for that boundary explicitly.

Telemetry control is now part of operational resilience because bad signal handling becomes an incident multiplier. A pipeline that drops the wrong data under pressure, or keeps too much low-value data while a backend slows down, can obscure the evidence teams need during an outage. The article's reliability discussion is the key point: backpressure, memory limits, and buffering are not implementation details but resilience controls. Practitioners should treat telemetry pipelines as critical path infrastructure.

The identity angle matters because telemetry pipelines increasingly carry sensitive operational signal. Logs, traces, and metrics often contain secrets, user identifiers, service account references, and workload metadata. That means the same upstream controls used to reduce observability cost also support data minimisation and exposure reduction. For IAM, PAM, and NHI teams, the lesson is straightforward: the pipeline is part of the control surface, not just the data plane.

What this signals

Telemetry pipelines are now part of the same control conversation as secrets hygiene and identity telemetry governance. When teams can drop noise, remove sensitive attributes, and preserve only the signal they actually investigate, they reduce both cost and exposure. That is especially relevant where logs and traces carry service account names, tokens, or workload metadata.

Decision-late control: the best place to govern telemetry is after enough context exists to make the right choice, but before the observability bill is irreversible. That principle aligns closely with pipeline sampling, redaction, and routing choices, and it is also why identity and NHI teams should treat telemetry processing as a security control, not just an engineering optimisation.

For programmes that already track secret exposure and privileged activity, the practical next step is to align telemetry retention with data minimisation and response needs. If a field would not help an incident responder, it should not be reaching expensive long-term storage by default.

For practitioners

• Move filtering upstream of backend ingestion Drop health checks, readiness probes, and other low-value signals before they reach billable storage or indexing. Use receiver and processor rules to remove noise at the edge rather than paying to move it through the stack.
• Strip high-cardinality attributes before export Remove volatile keys such as pod_name, request_id, and user_id from metrics unless they are explicitly required for troubleshooting. In Kubernetes-heavy environments, do this before metrics reach the backend so the time-series explosion never occurs.
• Reserve tail sampling for gateway tier decisions Keep errors and slow traces with tail sampling, but only after all spans for a trace have converged in one collector instance. If trace fragments are split across nodes, the sampling decision will be incomplete and misleading.
• Design backpressure by signal criticality Apply disk buffering and blocking behaviour to audit logs and severe errors, while allowing low-value debug or info logs to be dropped under pressure. This prevents node failure without sacrificing the evidence that matters during an incident.
• Treat telemetry policy as living configuration Review pipeline rules whenever new services, labels, or debug modes appear, because static suppression logic quickly goes stale. Tie ownership to the platform team that sees cost and the service teams that create signal, so neither side can ignore drift.

Key takeaways

• Telemetry pipelines are the real control point for observability cost because they decide what data exists before backend billing begins.
• Processing choices such as filtering, transformation, and tail sampling shape both reliability and spend, especially in high-cardinality environments.
• Identity and secrets governance teams should treat telemetry as a security control surface, because the same upstream decisions affect exposure, retention, and incident evidence.

Key terms

• Telemetry pipeline: A telemetry pipeline is the path security data follows from collection to analysis and retention. In mature environments it must preserve context, maintain throughput, and avoid introducing blind spots as sources, formats, and volumes change over time.
• Cardinality: Cardinality is the number of unique attribute combinations a metric can produce. In observability systems, high cardinality turns a single metric into many time series, which quickly increases storage, query load, and cost when attributes such as request IDs or pod names are allowed to multiply.
• Tail Sampling: Tail sampling is a trace selection method that waits until spans for a trace are complete before deciding whether to keep them. It preserves decision quality because it can retain traces based on outcome, such as errors or latency, rather than guessing at the start of the request.
• Backpressure: Backpressure is the mechanism used to slow or control producers when consumers or brokers cannot safely absorb more traffic. In governance terms, it helps protect platform stability and prevents noisy or misbehaving clients from overwhelming shared event infrastructure.

What's in the full article

Sawmills' full article covers the operational detail this post intentionally leaves for the source:

• Receiver, processor, and exporter configuration examples for OpenTelemetry pipelines in production
• Practical guidance on filter, transform, and tail sampling ordering inside agent and gateway tiers
• Reliability tradeoffs for backpressure, memory limits, and buffering under backend outages
• Where the agentic operator model changes day-to-day telemetry maintenance and suppression-rule drift

👉 The full Sawmills article covers pipeline mechanics, topology choices, and cost-control decisions in more operational depth.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, IAM, and secrets management. It is built for practitioners who need to connect identity controls to broader operational security programmes.