They should calculate the full storage footprint, then decide which event classes truly justify long-term retention. For identity and security teams, the key question is whether the data will be searched, correlated, or only preserved for compliance. If it will not support a real use case, it should not be stored at premium cost.
Why This Matters for Security Teams
Extending log retention sounds simple until teams discover the real cost is not just storage, but indexing, backup, tiering, legal hold, and retrieval overhead. Long retention only helps when it supports a defined security or compliance outcome, such as forensic investigation, identity correlation, or incident reconstruction. The NIST Cybersecurity Framework 2.0 treats logging as part of operational resilience, which is useful because retention without a purpose often becomes expensive data hoarding.
The first step is to separate event classes that have active operational value from those kept only because someone is uncomfortable deleting them. Authentication events, privilege changes, admin actions, and security alerts usually justify longer retention more than routine application telemetry. Teams also need to test whether the retained data is actually searchable at the speed required during an incident, because years of frozen logs that cannot be queried quickly are not much better than no logs at all.
Identity and security teams frequently underestimate how retention decisions interact with privacy, access governance, and incident workflows. In practice, many security teams encounter the real retention problem only after an investigation has already been delayed by missing context or an overgrown archive that no one can use.
How It Works in Practice
The practical sequence is to define the use case first, then map it to the minimum viable retention period and storage tier. For example, a SOC may need 90 days of hot searchable logs, one year of warm searchable logs, and a longer compliance archive for a narrow set of identity and security events. That is very different from keeping every source at maximum retention by default.
Teams should inventory log sources by business value and investigative value, not by volume alone. The highest-priority classes often include:
- Authentication and federation logs, including failed logins and token issuance
- Privilege elevation, admin activity, and PAM session records
- Identity lifecycle events, such as provisioning, deprovisioning, and role changes
- Security detections, alert metadata, and incident response artifacts
- Cloud control-plane and management-plane logs where they affect access or configuration
Then the team should estimate storage in three layers: raw ingest, indexed search, and archive. This matters because “years of retention” often means years of immutable storage plus retention of enough metadata to make the records discoverable. If the question is compliance-driven, teams should validate legal or regulatory retention obligations and avoid retaining broader content than necessary. If the question is operational, they should confirm the data can be searched, correlated, and exported into SIEM or case management tooling when needed.
Current guidance suggests pairing retention policy with data classification, access control, and deletion schedules. Logs may contain secrets, identifiers, or sensitive behavioral data, so retention decisions should also consider who can read them and for how long. Control mapping should align to logging, monitoring, and data protection objectives, not just storage procurement. These controls tend to break down when log pipelines are fragmented across cloud accounts, SaaS platforms, and legacy appliances because no single owner can prove what is retained, where it lives, or whether it is still queryable.
Common Variations and Edge Cases
Tighter retention often increases cost and operational burden, requiring organisations to balance forensic value against storage, privacy, and retrieval constraints. That tradeoff becomes sharper when logs contain personal data, tenant data, or high-cardinality identity fields that make indexing expensive.
Some environments genuinely need extended retention, but the justification is narrower than many teams assume. Financial services, critical infrastructure, and regulated healthcare may need longer records for audit and reconstruction, while engineering teams may only need short retention for debug and threat detection. In cloud and SaaS-heavy estates, it is often better to retain a smaller set of high-value security events for longer than to preserve everything uniformly.
There is no universal standard for this yet across all sectors, so best practice is evolving. Teams should document what each log class supports: detection, investigation, compliance, or legal hold. If the answer is “none of the above,” the log should not be promoted to expensive long-term storage. For identity-heavy environments, that includes deciding whether authentication history, admin actions, and token events must remain searchable or only archived.
When the environment spans multiple jurisdictions, retention policy also has to account for data residency and privacy obligations. A long-retention design that ignores deletion rights, access review, or encryption key management usually creates more governance risk than it removes.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK address the attack surface, NIST CSF 2.0 set the technical controls, and PCI DSS v4.0 define the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | GV.RM-01 | Retention should be driven by risk and business value, not default accumulation. |
| MITRE ATT&CK | T1078 | Authentication and account abuse logs are key evidence for investigating valid account misuse. |
| PCI DSS v4.0 | 10.7 | PCI retention rules often require bounded storage and controlled access to audit logs. |
Prioritise retention of login, token, and privilege logs that help detect and investigate valid account abuse.
Related resources from NHI Mgmt Group
- What should security teams check before extending access controls to autonomous systems?
- What should security teams check before extending SPIFFE trust to another domain?
- What should teams do before extending workload identity to partners or multiple clusters?
- How should security teams validate GCP audit-log detections before relying on them in production?
