When teams collect everything, storage costs climb quickly and analysis becomes noisier and slower. The pipeline can also waste compute on data that never supports a decision. Over time, this makes it harder to retain the right data for compliance, incident review, and troubleshooting. Filtering and aggregation help preserve the signals that matter while controlling cost growth.
Why This Matters for Security Teams
Telemetry is only useful when it can be turned into decisions. When teams retain every event, debug trace, and duplicate signal, the result is not better visibility but weaker operational clarity. Analysts spend more time searching through noise, and security engineers lose the ability to distinguish genuine anomalies from routine background activity. The NIST Cybersecurity Framework 2.0 emphasises risk-informed outcomes, which is exactly why data collection should be tied to response, investigation, and retention needs rather than habit.
This matters across cloud platforms, endpoints, SIEM pipelines, and application observability stacks. Low-value telemetry often enters the environment through default logging settings, duplicated collectors, or broad ingestion rules that were never revisited after deployment. The practical failure is not just cost growth. It is the gradual erosion of detection quality, because high-volume low-signal data can bury the very events that would support triage, forensics, or audit readiness. In practice, many security teams discover telemetry sprawl only after an incident review is slowed by irrelevant data rather than through intentional data governance.
How It Works in Practice
Effective telemetry filtering starts with deciding what each data source is for. Security teams should separate data needed for live detection, short-term troubleshooting, compliance retention, and long-horizon trend analysis. Those use cases rarely require the same granularity. For example, a SIEM may need normalised security events and high-fidelity authentication records, while routine application debug logs may be better aggregated at source or retained only for a short window.
Good practice is to define collection tiers before ingestion. That usually means combining allowlists, sampling rules, severity thresholds, and field-level filtering so that only events with operational value are forwarded at full detail. Aggregation can preserve useful patterns while reducing noise, especially for repetitive status messages, health checks, and known-benign system chatter. Teams should also review whether duplicate collection exists across EDR, cloud logging, and application telemetry, because the same event often appears in multiple pipelines without adding investigative value.
- Keep full-fidelity data where it supports detection, reconstruction, or regulatory retention.
- Aggregate or sample repetitive events that do not change decisions.
- Apply shorter retention to noisy debug streams unless a specific use case justifies longer storage.
- Document which data fields are required for incident response and which are optional.
From an identity perspective, telemetry should still preserve authentication, privilege change, and service-account activity because those records are often essential in NHI and PAM investigations. The key is not to suppress security-relevant identity events, but to remove surrounding noise that adds cost without improving confidence. These controls tend to break down when every team owns a separate logging standard because inconsistent schemas and overlapping collectors make rational filtering nearly impossible.
Common Variations and Edge Cases
Tighter telemetry control often reduces storage and analyst burden, requiring organisations to balance visibility against operational overhead. That tradeoff is especially sharp in regulated environments, where teams may be tempted to keep everything “just in case.” Current guidance suggests that this is rarely the best approach. Retention should be driven by legal, investigative, and security use cases, not by the assumption that more data is always safer.
There are edge cases where broad collection is temporarily justified. During an active incident, a migration, or a new detection-tuning cycle, teams may intentionally collect more data to establish baselines or reconstruct attacker behaviour. The important point is that this should be time-bound and reviewed. Similarly, high-value environments such as payment systems, critical infrastructure, or identity platforms may need more granular records for a defined subset of events, but even then the extra detail should be purposeful.
Best practice is evolving around telemetry minimisation, especially as organisations try to align observability with privacy, cost control, and resilience goals. The right answer is usually selective retention: keep the events that support a decision, reduce the rest, and revisit the policy as systems and threats change. This becomes harder when engineering teams optimise for “collect now, decide later” because data pipelines are easier to expand than to rationalise.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK and OWASP Non-Human Identity Top 10 address the attack and risk surface, while NIST CSF 2.0 and NIST AI RMF set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | GV.RM-01 | Telemetry volume should be governed by risk, not collection habit. |
| MITRE ATT&CK | T1005 | Collected data should support discovery of attacker activity and data access paths. |
| NIST AI RMF | AI-assisted analytics can amplify noise if input telemetry is poorly governed. | |
| OWASP Non-Human Identity Top 10 | Service-account and machine identity logs matter, but only when filtered for investigative value. |
Set telemetry priorities by business risk and security value, then prune data that does not support decisions.
Related resources from NHI Mgmt Group
- How should identity teams use event networking to improve fraud and risk programmes without collecting low-value contacts?
- How should teams reduce low-value phishing report tickets without weakening user reporting?
- How should teams verify accredited investor status without over-collecting personal data?
- How should identity teams implement interoperable age assurance without over-collecting data?
