The common mistake is filtering without context. If you drop or keep events before enrichment, you risk removing the very evidence needed to separate routine activity from abuse. The better model is to enrich first, then make retention decisions based on the resulting security value.
Why This Matters for Security Teams
Filtering logs before ingestion looks efficient, but it can turn a defensive control into an evidence loss problem. SOC teams often assume they are reducing noise when they are actually removing the context needed for correlation, triage, and investigation. That matters most when attackers mimic normal behaviour, because the earliest clues are often low-signal events that appear unimportant in isolation. Guidance from the ENISA Threat Landscape consistently reinforces that modern intrusions blend into routine activity, which means security value is rarely obvious at the log source.
The practical issue is that ingestion filtering is frequently done for storage, licensing, or pipeline performance reasons rather than for risk-based curation. Once a security event is dropped upstream, it cannot be reconstructed from downstream tools, and that creates blind spots in SIEM, XDR, and incident response workflows. Teams also underestimate how often “boring” metadata, such as source, timing, process lineage, and authentication sequence, becomes the differentiator in an investigation. In practice, many security teams encounter filtering mistakes only after an incident has already forced them to ask for logs that were never ingested.
How It Works in Practice
The safer pattern is to treat ingestion as a two-step process: collect broadly, then enrich and tier the data before deciding what to retain long term. Enrichment can add identity context, asset criticality, geolocation, threat intelligence, and workflow tags that make an event materially more useful. At that point, retention policy can distinguish between high-value security telemetry and low-value operational chatter without destroying investigative evidence. This approach aligns with the control intent behind NIST Cybersecurity Framework functions for detect and respond, where visibility and analysis depend on complete enough telemetry.
In operational terms, the SOC should define which fields are mandatory, which event classes are high priority, and which can be sampled or summarised after enrichment. That usually means keeping:
- authentication events, privilege changes, and failed access attempts
- process and command-line telemetry on endpoints
- admin, audit, and configuration changes in cloud and identity platforms
- network and application logs needed to correlate a user, host, and session
Filtering should then happen on derived features, not raw evidence. For example, a routine service account login may be low priority until enrichment shows it came from an unusual host or followed a suspicious privilege escalation. The same logic applies to agentic AI and automation workflows, where a machine or agent identity may look noisy until it is linked to a high-risk action path or secret use. Where teams have strong data pipelines, the best practice is evolving toward policy-based retention instead of pre-ingestion deletion, but there is no universal standard for this yet. These controls tend to break down when log volume spikes faster than the pipeline can enrich events, because teams revert to blunt upstream dropping to protect storage and platform stability.
Common Variations and Edge Cases
Tighter filtering often reduces storage and licence costs, requiring organisations to balance operational efficiency against forensic completeness. The tradeoff is especially sharp in high-volume environments such as cloud platforms, identity providers, and CI/CD systems, where administrators are tempted to remove “known good” records too early. That approach can be defensible for duplicate metrics or clearly non-security telemetry, but current guidance suggests caution whenever the data might later support attribution, scope determination, or timeline reconstruction.
One common edge case is regulated retention: if logs support incident investigation, fraud review, or audit requirements, the value of a record is not determined only by day-to-day SOC use. Another is multi-tenant or shared-service architecture, where an event that looks generic at source can become critical once tenant, workload, or trust-zone context is added. For teams aligning with CISA resources and threat-led detection programs, the safer rule is to preserve raw telemetry until enrichment proves it is low value. That is particularly important in identity-heavy environments where NHI, service accounts, and delegated access can make routine activity appear deceptively normal. The real risk is not too much data, but too little context when a suspected compromise finally has to be proven.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK and OWASP Non-Human Identity Top 10 address the attack and risk surface, while NIST CSF 2.0, NIST Zero Trust (SP 800-207) and NIST AI RMF set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | DE.CM-1 | Continuous monitoring depends on retaining enough log evidence to detect abnormal activity. |
| MITRE ATT&CK | T1078 | Valid account abuse is harder to spot when upstream filtering removes supporting context. |
| OWASP Non-Human Identity Top 10 | NHI-6 | NHI misuse can hide inside normal service activity unless logs retain identity context. |
| NIST Zero Trust (SP 800-207) | SP 800-207, continuous verification | Zero Trust decisions rely on context-rich telemetry, not stripped-down raw events. |
| NIST AI RMF | MAP | AI-assisted log triage should preserve provenance and context for trustworthy decisions. |
Preserve and enrich telemetry before retention decisions so monitoring can detect anomalies reliably.
