TL;DR: Full-fidelity log retention is being squeezed by rising ingest prices, growing telemetry volumes, and expensive retrieval, according to Anomali’s analysis of Microsoft, Dynatrace, and Splunk pricing and survey data. The practical shift is from hot-index dependence to decoupled storage and query, because long retention only works when searchability no longer inherits SIEM economics.
At a glance
What this is: NHI Mgmt Group's analysis, drawing on Anomali's blog post: The real cost of keeping full-fidelity SOC data is not storage alone but the premium paid to ingest, index and retrieve it.
Why it matters: This matters because IAM, NHI and security teams depend on long-retained telemetry for investigations, and retention only scales when searchability is no longer tied to hot-index economics.
Context
SOC retention becomes a governance problem when organisations can store data cheaply but cannot afford to search it when they need to. The technical issue is not whether logs exist, but whether historical evidence remains practically queryable across identity, cloud and endpoint investigations. For identity-heavy environments, that also shapes how far back teams can reconstruct authentication, session and privilege activity.
Anomali’s article frames the core trade-off as fidelity versus economics. That is the right lens for modern security operations because long retention is only useful if the platform can preserve analytic usefulness after data leaves the hot tier. The identity angle is especially relevant where investigators need years of authentication history, non-human identity activity and cross-system correlation.
Key questions
Q: Why do long-retention security archives become unusable in practice?
A: Because many platforms price ingest, indexing and retrieval together, so older data may be retained but not economically searchable. When the cost or delay of retrieving evidence rises, teams stop running broad historical hunts. The result is operational blindness, not true preservation. Retention only helps if the archive remains queryable at investigation time.
Q: How should security teams design retention for identity investigations?
A: Teams should preserve authentication, session and access data in a common schema before it leaves the hot tier. That lets human and non-human activity be queried together across long time ranges. The key decision is not only how long to store data, but whether the stored data can still answer incident questions months later.
Q: What breaks when telemetry is not normalised before analysis?
A: Cross-source correlation becomes unreliable, identity and network events are harder to match, and AI tools have to learn around inconsistent labels instead of analysing behaviour. The result is slower investigations, weaker automation, and more manual interpretation by analysts.
Q: Should teams keep all security data in a hot SIEM tier?
A: No. Hot tiers make sense for recent, latency-sensitive detections, but long-range investigations are usually better served by low-cost object storage and on-demand query. The practical balance is to reserve expensive indexing for the data that needs real-time speed and move the rest to a cheaper, searchable archive.
Technical breakdown
Why hot-tier log economics break long retention
Hot-tier SIEM economics charge for ingest, index and retention at the same time, which makes full-fidelity retention expensive long before storage limits are reached. The problem is structural: the more telemetry you keep searchable in the primary analytics tier, the more you pay per gigabyte to preserve a capability you may use only occasionally. Once teams push data into colder tiers, retrieval friction starts to determine whether investigations are even attempted. Practical implication: design retention around affordability of retrieval, not just the cost of keeping bytes somewhere.
Practical implication: Treat searchable retention as a separate control objective from storage retention.
How schema normalisation makes older identity data usable
Normalisation is what prevents archived logs from turning into a pile of incompatible formats. A common schema, such as an identity-focused event model, lets different sources like Windows, SaaS and cloud authentication data be queried together without translating every source at hunt time. That matters for identity and NHI investigations because the same question often spans human logins, service accounts, API access and workload activity. Practical implication: keep old telemetry queryable by mapping key authentication and access events into a consistent structure before it ages out of the hot tier.
Practical implication: Map identity and access events into one schema before they move to cold storage.
Why decoupled storage and query changes the SOC cost model
Decoupling compute from storage changes the economics of retention because storage becomes an object-storage problem and analytics become an on-demand compute problem. That removes the always-on penalty of a hot index sized for peak demand, while still allowing broad retrospective hunts when needed. It does not eliminate cost. It shifts it into a model where teams pay for compute when they ask questions, not every hour of the day. Practical implication: use serverless or elastic query layers for retrospective search and reserve hot indexing for recent, latency-sensitive detections.
Practical implication: Separate recent-detection workloads from historical-hunt workloads in your SOC architecture.
NHI Mgmt Group analysis
Full-fidelity retention is now an access problem, not just a storage problem. When retrieval is expensive or slow, the practical result is that evidence exists but is functionally unavailable to investigators. That creates blind spots across identity timelines, service-account activity and cross-domain correlation. The practitioner conclusion is to govern retentive usefulness, not simply retention duration.
Identity investigations depend on long-range queryability more than on raw log volume. A year of data that cannot be searched across authentication sources is operationally weaker than a smaller but normalised dataset. This is where NHI and IAM programmes intersect with SOC architecture: machine identities, delegated access and human logins all need a common evidentiary layer. The practitioner conclusion is to make schema consistency part of security operations design.
Retention economics are pushing the market toward decoupled security data architectures. That shift validates the move away from monolithic SIEM dependency and toward storage-plus-query models that preserve fidelity without paying hot-tier prices forever. It also complicates legacy reporting assumptions because “retained” no longer means “usable for hunt.” The practitioner conclusion is to test whether archived telemetry still supports real investigations, not just compliance storage.
What this signals
Retention fidelity will increasingly be judged by investigation viability, not by how many months a platform can technically store. For identity-heavy programmes, that means checking whether archived authentication and access data still supports real hunts after the hot window closes. If it does not, the organisation has bought storage, not security evidence.
Decoupled storage and query is becoming the architectural pattern that lets SOC teams keep more history without inheriting always-on SIEM economics. That matters for IAM and NHI work because historical access trails, service-account actions and delegated authentication flows all need to stay usable long after ingestion.
Identity data normalisation is the hidden control that makes retention economically defensible. Without a common schema, older logs are retained in theory but trapped in source-specific dialects in practice. The programme implication is to treat schema governance as part of evidence retention, not just data engineering.
For practitioners
- Separate hot detection from historical hunting Keep recent, latency-sensitive data in the primary analytics tier, but move older telemetry to low-cost object storage with elastic query for retrospectives.
- Normalise identity events before cold storage Map authentication, session and access events into a common schema so investigators can query human and non-human activity across sources later.
- Model retrieval as part of retention cost Include scan charges, restore jobs, query compute and egress in the retention budget, not just storage price per gigabyte.
- Test hunt viability on archived data Run realistic retrospective searches on data older than the hot window to confirm that the platform still supports identity investigations, not just storage.
Key takeaways
- Long-term SOC retention fails when the platform charges premium rates for both ingest and retrospective access.
- Historical investigations depend on whether authentication and access data remain queryable after the hot tier expires.
- The practical fix is to separate storage from compute and normalise identity events before they age into cold storage.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK addresses the attack and risk surface, while NIST CSF 2.0, CIS Controls v8 and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | PR.DS-01 — Data-at-Rest | The article centres on retaining security data economically without losing its usefulness. |
| DE.CM-01 — Anomalies and Events | Historical hunt value depends on the ability to detect and investigate events across long time windows. | |
| Recommendation — Apply PR.DS-01 to keep retained telemetry protected while preserving long-term usability. Use DE.CM-01 to ensure retained logs still support anomaly detection and retrospective investigation. | ||
| CIS Controls v8 | CIS-8 — Audit Log Management | The article is fundamentally about retaining and using logs for investigations. |
| Recommendation — Implement CIS-8 to centralise, retain and test log usability across the full investigation window. | ||
| MITRE ATT&CK | TA0007;TA0009 — Discovery; Collection | Long-range telemetry exists to reconstruct discovery and collection activity after the fact. |
| Recommendation — Map retained telemetry to TA0007 and TA0009 so hunts can reconstruct attacker discovery and collection paths. | ||
| NIST SP 800-53 Rev 5 | AU-11 — Audit Record Retention | Audit record retention is the clearest control family for long-lived security telemetry. |
| Recommendation — Use AU-11 to define how long audit records remain available for investigation and compliance. | ||
Key terms
- Hot Tier: The hot tier is the fastest and most expensive storage layer in a SIEM or data platform. It holds logs that security teams need to query immediately for correlation, alerting, and live investigation, especially when time-sensitive attacker activity is suspected.
- Cold Storage: Cold storage is lower-cost, non-indexed retention used for data that must be kept for compliance or forensics but does not need immediate correlation. In security operations, it preserves evidence without forcing every event into expensive real-time detection workflows.
- Schema Normalization: Schema normalization is the process of converting inconsistent raw log fields into a stable structure that downstream systems can reliably use. It reduces parsing drift, improves correlation accuracy, and prevents each tool from having to solve vendor-specific formatting problems on its own.
- Retrospective Hunting: Retrospective hunting is the process of searching historical telemetry for evidence that a known threat or indicator was present earlier in the environment. For NHI incidents, it is essential because compromised software can leave behind access, execution, or credential artefacts long before detection occurs.
Deepen your knowledge
NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It is designed for practitioners who need to connect identity controls to broader security operations and evidence retention.
Published by the NHIMG editorial team on September 14, 2026.
Updated on October 11, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org