Protect records that support accountability, such as authentication, privileged activity, service account usage, and incident-response evidence. Those logs should be exempt from aggressive filtering unless you have a documented control that preserves their evidentiary value. If a record would matter in a post-incident review, do not let cost optimisation erase it.
Why This Matters for Security Teams
Logging decisions are not just a storage problem. They shape whether a security team can prove what happened, reconstruct attacker movement, and separate normal operations from abuse. The highest-value records usually capture authentication events, privilege changes, service account activity, administrative actions, and response actions. When these records are reduced too early, investigators lose context and compliance teams lose evidence.
Current guidance in the NIST Cybersecurity Framework 2.0 emphasises that governance, detection, and response depend on reliable telemetry, not just broad collection. The practical issue is that many teams treat filtering as a routine storage optimisation step rather than a control decision. That is where mistakes happen, especially when logging pipelines, SIEM rules, or cloud-native defaults are tuned by cost rather than by investigative value.
The right question is not whether a log is noisy. It is whether the record may be needed to establish accountability, confirm a sequence of events, or support legal and operational review. In practice, many security teams encounter gaps only after a privileged compromise, rather than through intentional log design.
How It Works in Practice
Teams usually decide log retention and filtering by mapping each log source to a security purpose. Records tied to identity, privilege, and control-plane activity are normally protected first, because they are the most likely to answer who acted, what changed, and from where. That includes authentication successes and failures, MFA events, session starts and ends, elevation requests, account creation, policy changes, and activity from service accounts or non-human identities.
A sensible workflow is to classify logs into three buckets: must keep, can summarise, and can discard after short-term use. “Must keep” logs are those that support incident response, legal hold, fraud review, or audit trails. “Can summarise” logs may be aggregated after a defined period, provided the raw source remains available long enough to support investigation. “Can discard” logs are low-value operational noise that do not materially affect accountability.
- Keep raw authentication and privileged access events where possible, even if summaries are also stored.
- Preserve administrative and service account actions because these often explain lateral movement or automation abuse.
- Protect incident-response artefacts such as alert pivots, containment actions, and timeline records.
- Document any filtering rule so analysts can explain what was removed and why.
- Review filters after major changes to identity architecture, cloud logging, or agentic automation.
For most environments, the deciding factor is evidentiary value. If the record may be needed to prove access, reconstruct intent, or verify containment, it should not be subject to aggressive filtering. Teams can use SIEM and SOAR tooling to route, enrich, and prioritise these records, but not to eliminate them blindly. Where identity telemetry is involved, the handling of non-human identities is increasingly important because automation accounts often produce the clearest trace of a compromise or misconfiguration.
These controls tend to break down when logs are centralised across many SaaS, cloud, and endpoint sources because source-specific fields are flattened before analysts define what must remain immutable.
Common Variations and Edge Cases
Tighter log retention often increases storage, indexing, and investigation overhead, requiring organisations to balance evidentiary value against platform cost. That tradeoff is real, but it should not be resolved by stripping away the very records needed after an incident.
Best practice is evolving in environments that rely heavily on cloud-native telemetry, container orchestration, or agentic automation. In those settings, there is no universal standard for every field that must be retained forever. Instead, teams should prioritise the events that establish identity, privilege, and sequence, then apply shorter retention or summarisation to routine application noise. The challenge is especially sharp when service accounts rotate frequently or when machine identities generate high-volume events, because the useful signal can be buried in automation output.
Edge cases include privacy-sensitive logs, where minimisation is appropriate, and high-volume debug logs, where the raw content may be useful only briefly. Even then, security teams should avoid removing fields needed to link actions to a user, workload, or session. For regulated environments, the decision should also reflect legal hold, audit, and breach-notification obligations. When there is uncertainty, the safer path is to preserve the original record and control access to it rather than to filter it away.
In practice, the strongest programs define “never filter” categories up front, validate them against incident scenarios, and revisit them after each major platform or identity change.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Non-Human Identity Top 10 address the attack and risk surface, while NIST CSF 2.0, NIST AI RMF, NIST Zero Trust (SP 800-207) and NIST IR 8596 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | DE.CM-1 | Continuous monitoring depends on retaining trustworthy telemetry. |
| OWASP Non-Human Identity Top 10 | Service account and machine identity logs are often the most critical evidence. | |
| NIST AI RMF | Automated decisions and AI-assisted operations need auditable records. | |
| NIST Zero Trust (SP 800-207) | 5.2 | Zero trust relies on detailed verification and access telemetry. |
| NIST IR 8596 | Cyber AI operations can create new logging and evidence requirements. |
Preserve non-human identity activity logs that establish ownership, use, and abuse.
