Notifications
Clear all
Topic starter
18/08/2026 2:11 pm
TL;DR: Filtering, deleting fields, and sampling at the pipeline layer can sharply reduce telemetry ingestion costs without changing application code, according to Bindplane. The governance question is not whether to collect less data, but how to preserve operationally useful signals while removing high-volume noise and unnecessary field baggage.
NHIMG editorial — based on content published by Bindplane: a walkthrough on reducing telemetry pipeline costs with filtering and sampling
By the numbers:
- The Splunk branch is down to 429 KB/m after filtering and field deletion, compared with 5.4 MB/m on the cold storage branch.
Questions worth separating out
Q: How should security and platform teams reduce telemetry costs without losing operational visibility?
A: Use a layered approach: filter obvious noise first, delete unnecessary fields from the records that remain, and sample only the streams where aggregate patterns matter more than individual events.
Q: When does telemetry sampling create more risk than it reduces?
A: Sampling creates more risk whenever the stream carries incident evidence, high-value errors, or security-relevant state changes.
Q: What do teams get wrong about filtering logs in observability pipelines?
A: They often assume filtering is only about cost, when it is also about governance and data minimisation.
Practitioner guidance
- Apply severity thresholds on noisy production logs Use warn as the default cutoff for high-volume streams that mostly contain debug and info noise, but route any lower-severity data you still need to a cheaper destination instead of deleting it outright.
- Separate health-check suppression into distinct condition groups Model each probe endpoint as its own condition group and join the groups with OR, otherwise AND logic will make the filter match nothing and the noise will continue to flow.
- Delete high-cardinality fields before expensive indexing Remove fields such as session_token, container_id, k8s.pod.uid, thread_id, and build_sha from branches that feed premium destinations so the retained records are smaller without losing count.
What's in the full article
Bindplane's full post covers the operational detail this post intentionally leaves for the source:
- Step-by-step processor setup for Filter by Severity, Filter by Condition, Delete Fields, and Sample Logs
- Live preview behaviour that shows before-and-after record counts as filters are edited
- Concrete examples of health-check suppression logic and the AND versus OR condition grouping pitfall
- The exact pipeline branch changes that reduce Splunk ingestion while leaving cold storage untouched
👉 Read Bindplane's walkthrough on filtering telemetry pipelines for cost control →
Telemetry filtering and sampling: what it means for pipeline governance?
Explore further
18/08/2026 3:04 pm
Telemetry cost control is now a governance problem, not just an observability problem. Once logs, traces, and metrics support security monitoring, the question becomes which data is worth indexing, retaining, and protecting. Pipeline filtering is therefore part of data governance, because unnecessary telemetry can increase cost, privacy exposure, and investigative noise at the same time. Practitioners should treat data reduction as a policy decision, not only a tuning exercise.
A question worth separating out:
Q: How do you know whether telemetry reduction is actually working?
A: Compare record volume and payload size before and after each processor, then check that the logs still contain the events needed for paging, investigation, and audit. If cost falls but detection quality or troubleshooting time worsens, the controls are too aggressive. Measure both efficiency and usefulness, not just ingestion reduction.
👉 Read our full editorial: Filtering telemetry pipelines can cut ingestion costs fast
