Filtering telemetry pipelines can cut ingestion costs fast

At a glance

What this is: This is a practical walkthrough of using OpenTelemetry pipeline processors to reduce telemetry volume by filtering, deleting fields, and sampling before data reaches expensive destinations.

Why it matters: It matters because security and platform teams must control data growth without losing the records needed for detection, incident response, and auditability, especially where telemetry feeds SOC and IAM-adjacent monitoring.

By the numbers:

• The Splunk branch is down to 429 KB/m after filtering and field deletion, compared with 5.4 MB/m on the cold storage branch.

👉 Read Bindplane's walkthrough on filtering telemetry pipelines for cost control

Context

Telemetry pipelines often fail for the same reason many security data programmes fail: they collect too much, retain too much, and then pay to move and index noise. Pipeline-level filtering changes that economics by removing unnecessary records before they hit a billable destination, which is especially relevant where logs support security operations, audit trails, and access investigations.

The article also has an identity-adjacent angle because telemetry frequently includes session tokens, container IDs, pod UIDs, and build identifiers that function as operational secrets or traceable identity markers. In practice, that means data reduction and sensitive-field minimisation belong in the same governance conversation as secrets handling and workload visibility.

Key questions

Q: How should security and platform teams reduce telemetry costs without losing operational visibility?

A: Use a layered approach: filter obvious noise first, delete unnecessary fields from the records that remain, and sample only the streams where aggregate patterns matter more than individual events. Keep exception logs intact, and validate the effect in a live preview before changing production routing.

Q: When does telemetry sampling create more risk than it reduces?

A: Sampling creates more risk whenever the stream carries incident evidence, high-value errors, or security-relevant state changes. It is suitable for repetitive success traffic, but not for records that drive detection, auditing, or root-cause analysis. If losing three quarters of the data would change the conclusion, do not sample it.

Q: What do teams get wrong about filtering logs in observability pipelines?

A: They often assume filtering is only about cost, when it is also about governance and data minimisation. Another common mistake is using one broad rule for all traffic, which drops useful records or leaves noisy sources untouched. Good filters are specific, tested, and aligned to what downstream teams actually need.

Q: How do you know whether telemetry reduction is actually working?

A: Compare record volume and payload size before and after each processor, then check that the logs still contain the events needed for paging, investigation, and audit. If cost falls but detection quality or troubleshooting time worsens, the controls are too aggressive. Measure both efficiency and usefulness, not just ingestion reduction.

Technical breakdown

Why pipeline-stage filtering cuts cost before indexing

Telemetry processors operate after ingestion into the pipeline but before delivery to a destination. That placement matters because dropped records never become billable storage or searchable index content. In OpenTelemetry-based pipelines, this lets teams filter by severity, condition, metric name, or sampling ratio while preserving the original collection architecture. The key technical distinction is between reducing event volume and reducing event size. Filtering removes records entirely, while field deletion shortens the records that remain.

Practical implication: place reduction controls in the pipeline, not only in agents or apps, so cost control happens before data reaches expensive platforms.

How severity filtering differs from conditional exclusion

Severity filtering is a blunt control that keeps only events above a threshold such as warn or error. It is efficient when a stream is debug-heavy and the business value is concentrated in exception conditions. Conditional exclusion is more precise because it removes specific event patterns, such as health checks or repeated probe traffic, regardless of severity. The article also shows why boolean logic matters: conditions inside a group are ANDed, so distinct values usually need separate groups joined with OR. That is a common configuration trap.

Practical implication: use severity filters for broad suppression and condition filters for known noisy patterns, but validate boolean logic in preview before rollout.

Why field deletion and sampling solve different problems

Field deletion preserves event count while reducing payload size. It is useful for high-cardinality attributes such as session_token, container_id, and build_sha that inflate every record even when they are rarely queried. Sampling takes a different path by forwarding only a fraction of records, which is suitable when the statistical shape of the stream matters more than every individual event. Sampling is unsafe for error and fatal logs because it can erase the very evidence operators need during incidents. The mechanisms are complementary, not interchangeable.

Practical implication: delete noisy fields to shrink retained records, and sample only low-risk high-volume streams where partial visibility is acceptable.

NHI Mgmt Group analysis

Telemetry cost control is now a governance problem, not just an observability problem. Once logs, traces, and metrics support security monitoring, the question becomes which data is worth indexing, retaining, and protecting. Pipeline filtering is therefore part of data governance, because unnecessary telemetry can increase cost, privacy exposure, and investigative noise at the same time. Practitioners should treat data reduction as a policy decision, not only a tuning exercise.

High-cardinality telemetry fields create an unspoken data exposure surface. Session tokens, container IDs, and build identifiers are often operationally useful, but they also expand the amount of sensitive metadata that moves through systems. In identity-heavy environments, those values can become correlation keys for service accounts, workloads, and session activity. The result is not just larger invoices, but broader exposure if logs are overshared or retained without need.

Pipeline-side reduction aligns better with least-necessary-data principles than agent-only tuning. If every application team must change code to reduce telemetry, governance becomes inconsistent and slow. Central pipeline processors let security and platform teams apply a common policy across fleets while still preserving exceptions where they matter. That is the more scalable control model for large environments.

Sampling belongs in the observability design, but never as a default response to security signal loss. The article shows the right boundary: preserve enough low-risk volume to understand patterns, but do not sample away the records that support detection or incident reconstruction. For security teams, the practitioner conclusion is simple: define which telemetry classes are eligible for sampling before the pipeline is built.

What this signals

Telemetry reduction will increasingly be judged against governance outcomes, not simply storage savings. Security teams should expect stronger pressure to justify which log classes are retained, which fields are suppressed, and which streams can safely be sampled, especially where those records include identity-like identifiers or sensitive operational metadata.

Data minimisation at the pipeline layer: reducing what flows downstream before it becomes long-lived indexed data is a more scalable control model than retroactive cleanup. That matters for identity and security programmes because telemetry often contains session-linked or workload-linked markers that should not be retained longer than necessary.

For practitioners, the next maturity step is to define pipeline policy by data class, not by team preference. That means aligning observability design with retention, privacy, and incident response needs so cost reduction does not silently weaken security visibility.

For practitioners

• Apply severity thresholds on noisy production logs Use warn as the default cutoff for high-volume streams that mostly contain debug and info noise, but route any lower-severity data you still need to a cheaper destination instead of deleting it outright.
• Separate health-check suppression into distinct condition groups Model each probe endpoint as its own condition group and join the groups with OR, otherwise AND logic will make the filter match nothing and the noise will continue to flow.
• Delete high-cardinality fields before expensive indexing Remove fields such as session_token, container_id, k8s.pod.uid, thread_id, and build_sha from branches that feed premium destinations so the retained records are smaller without losing count.
• Restrict sampling to low-risk volume-heavy streams Use sample ratios only for traffic where aggregate shape matters more than each event, and keep error and fatal logs unsampled so incident triage still has complete evidence.

Key takeaways

• Pipeline processors can reduce telemetry cost materially without forcing application changes, but the control only works if teams distinguish between noise, volume, and evidence.
• Field deletion, conditional filtering, and sampling each solve a different problem, so using the wrong one can either waste money or destroy useful security data.
• The strongest operating model is policy-led telemetry reduction that preserves incident-grade records while removing high-cardinality overhead before indexing.

Key terms

• Telemetry pipeline: A telemetry pipeline is the path security data follows from collection to analysis and retention. In mature environments it must preserve context, maintain throughput, and avoid introducing blind spots as sources, formats, and volumes change over time.
• High-Cardinality Field: A high-cardinality field is a data attribute that takes many unique values, such as request IDs, pod UIDs, or session tokens. These fields often increase storage and indexing cost disproportionately because they repeat across large volumes of logs while adding little analytical value in most cases.
• Conditional Log Filtering: Conditional log filtering removes records based on explicit rule logic rather than broad severity alone. It is used to suppress known noisy patterns, like health checks or probe traffic, while preserving other events in the same stream that still matter for operations or security.

What's in the full article

Bindplane's full post covers the operational detail this post intentionally leaves for the source:

• Step-by-step processor setup for Filter by Severity, Filter by Condition, Delete Fields, and Sample Logs
• Live preview behaviour that shows before-and-after record counts as filters are edited
• Concrete examples of health-check suppression logic and the AND versus OR condition grouping pitfall
• The exact pipeline branch changes that reduce Splunk ingestion while leaving cold storage untouched

👉 The full Bindplane post shows the processor configuration details and live preview workflow.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course, the industry's only accredited NHI security programme. It is designed for practitioners who need to connect identity governance to the broader security controls their programmes depend on.