Sampling creates more risk whenever the stream carries incident evidence, high-value errors, or security-relevant state changes. It is suitable for repetitive success traffic, but not for records that drive detection, auditing, or root-cause analysis. If losing three quarters of the data would change the conclusion, do not sample it.
Why This Matters for Security Teams
Telemetry sampling is attractive because it lowers storage, transport, and analysis costs, but the security tradeoff is not symmetrical. Once a team samples away logs, traces, or events that show failures, privilege changes, authentication anomalies, or incident indicators, it can weaken detection, delay triage, and reduce confidence in investigations. That is why the question is less about volume reduction and more about whether the discarded records carry evidential value. The NIST Cybersecurity Framework 2.0 emphasizes outcomes such as detection, response, and recovery, which depend on trustworthy telemetry rather than merely cheaper telemetry.
Security teams often get this wrong by applying one sampling rule across all data types, then discovering that the samples preserved the routine path while dropping the rare path that mattered. That is especially dangerous in environments with short-lived cloud workloads, identity-rich audit trails, and automated response actions, because the security signal may exist only in a small fraction of records. In practice, many security teams encounter the loss of critical evidence only after an access review, incident, or dispute has already required the unsampled data.
How It Works in Practice
Effective sampling starts with classifying telemetry by purpose, not by source. If the record supports threat detection, forensic reconstruction, compliance, or safety-related state transitions, it generally needs full retention or selective preservation rules. If the record is high-volume, repetitive, and low-consequence, statistical sampling can be acceptable. The practical question is whether the telemetry is operational noise or security evidence.
For example, authentication successes may be sampled in a high-traffic environment, but authentication failures, MFA step-ups, token issuance, privilege elevation, and policy denials usually need deterministic capture. The same logic applies to agentic AI and automation platforms: tool invocation logs, prompt inputs, policy rejections, model output decisions, and secret access events are often more important than routine status heartbeats.
- Keep full fidelity for events that may become evidence in incident response or legal review.
- Sample only after defining what conclusion would remain valid if records are missing.
- Use adaptive or risk-based sampling rather than one fixed rate for every stream.
- Protect chain of custody for retained telemetry so the preserved subset is defensible.
Current guidance suggests aligning telemetry retention with detection use cases, alert thresholds, and investigation needs, then validating those assumptions against real threat scenarios. Guidance from the OWASP community on observability and security logging reinforces the idea that incomplete event capture can undermine root-cause analysis, even when aggregate metrics still look healthy. These controls tend to break down when large-scale distributed systems mix high-volume success traffic with rare but security-critical events, because the sampling layer cannot reliably distinguish noise from evidence in real time.
Common Variations and Edge Cases
Tighter sampling often reduces cost and alert fatigue, requiring organisations to balance observability against storage, bandwidth, and analyst workload. The hardest cases are not steady-state web requests, but bursty environments where the rare event is the one that matters. That includes cloud control planes, identity providers, CI/CD systems, and AI agent orchestration logs. For these, best practice is evolving toward hybrid models that preserve full-fidelity security events while sampling only routine operational telemetry.
There is no universal standard for this yet, but a sensible rule is to avoid sampling anything that could change an access decision, explain an anomaly, or prove a sequence of actions. In AI-enabled systems, that includes prompts, tool calls, policy outcomes, and model-assisted actions where a small number of records may show prompt injection, data exfiltration, or unsafe automation. In identity-heavy environments, the same caution applies to privileged sessions, token minting, and step-up authentication events. Where legal, regulatory, or audit requirements apply, the retention rule should be stricter than the analytics rule, not the other way around.
Telemetry sampling becomes especially risky when teams assume that dashboards are equivalent to evidence. Metrics can stay stable while the underlying records needed for a forensic timeline have already been lost.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK and OWASP Agentic AI Top 10 address the attack and risk surface, while NIST CSF 2.0 and NIST AI RMF set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | DE.CM-1 | Sampling can hide security-relevant events from monitoring and detection. |
| MITRE ATT&CK | T1070 | Incomplete logs can mask evidence destruction or post-compromise cleanup. |
| NIST AI RMF | GOVERN | AI and agent logs need governance so evidence is not lost to cost-driven sampling. |
| OWASP Agentic AI Top 10 | Agent actions and tool calls can be security evidence and should not be casually sampled. |
Retain full-fidelity telemetry for events needed to detect, investigate, and respond to attacks.
