Join our Newsletter — 33% off our NHI Course

Did OpenTelemetry pipeline filtering cut data costs fast?

 

(@nhi-mgmt-group)
Member Moderator
Joined: 2 years ago
Posts: 19717
Topic starter  

TL;DR: Severity-based dropping, field deletion, and sampling can each reduce telemetry volume differently, according to Bindplane. Telemetry pipeline governance has to decide which records keep full fidelity so cost cuts do not erase evidence-bearing data.

Editorial analysis by NHI Mgmt Group, based on content published by Bindplane: “How to Reduce Data Costs with OpenTelemetry and Bindplane”.

Key questions

Q: Why do telemetry pipelines matter for governance rather than just engineering efficiency?

A: Because they decide what data is allowed to exist in the observability system at all.

Q: What mistakes do teams make when sampling log and trace data?

A: The most common mistake is sampling streams that contain rare but important failures, which destroys forensic value.

Q: How should security teams reduce telemetry overload without losing useful signals?

A: Start by separating data needed for compliance retention from data needed for live investigation.

Practitioner guidance

  • Implement severity-based filtering on expensive destinations Start by removing debug and info logs from high-cost sinks while preserving warn, error, and fatal records for operational response.
  • Use conditional exclusions for repetitive health checks Exclude known low-value traffic such as liveness and readiness probes, and keep each endpoint in its own condition group so the logic matches correctly.
  • Delete high-cardinality fields before they reach indexing Strip session_token, container_id, k8s.pod.uid, thread_id, and build_sha from branches that do not need those attributes for investigation.

Bottom line: Pipeline filtering can reduce ingestion spend quickly, but the control only works safely when teams distinguish between noisy telemetry and evidence-bearing telemetry.

Glossary terms: Telemetry, Telemetry pipeline, Field Deletion

Explore further

View Full Forum →  |  NHI Foundation Course →  |  Our Services →  |  Read the full analysis →


This topic was modified 1 day ago by NHI Mgmt Group

   
Quote
(@mr-nhi)
Member Moderator
Joined: 6 months ago
Posts: 20207
 

Telemetry governance is now a control-plane issue, not an afterthought. Once telemetry is filtered in-pipeline, the organisation has already made a decision about what evidence survives. That means cost, detection coverage, and auditability are linked at the point of collection, not just at the point of storage. For identity and access programmes, this is especially important because session data, token traces, and privileged activity logs may be reduced before they ever reach a SIEM. Practitioners should treat pipeline policy as part of security governance, not just observability tuning.

A question worth separating out:

Q: What should organisations keep at full fidelity in telemetry pipelines?

A: Keep error, fatal, privileged, and incident-response data at full fidelity whenever those records are needed for investigation or accountability. Sampling and aggressive filtering belong on repetitive, low-risk streams where summary behaviour is enough and evidence loss will not undermine the control objective.

👉 Read our full editorial: Filtering telemetry pipelines can cut ingestion costs fast



   
ReplyQuote
Share:

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.