Retention economics expose the real cost of full-fidelity SOC data

At a glance

What this is: This is Anomali’s analysis of why full-fidelity SOC retention becomes economically unstable once log volume, ingest pricing, and retrieval costs are considered together.

Why it matters: It matters because IAM, NHI, and SOC teams need searchable history for identity investigations, but cost pressure often forces retention decisions that weaken retrospective detection and access-forensics.

By the numbers:

• Microsoft Sentinel bills pay-as-you-go ingestion at $4.30 per gigabyte in its East US region, climbing toward $5.13 in higher-cost regions like Switzerland North.
• Dynatrace's 2026 State of Log Management report found log and telemetry volume rose 93 percent on average over the prior year.
• 150 percent

👉 Read Anomali's analysis of full-fidelity retention economics

Context

SOC retention strategy is no longer just a storage question. Once telemetry volume rises faster than budget and the cost of historical queries rises with it, organisations start trimming the very history they need for investigations, identity tracing, and post-incident reconstruction. In IAM and NHI programmes, that creates a practical gap between what should be knowable and what remains searchable.

The article frames a common operational tension: hot retention is expensive, cold retention is slow, and both can undermine access-led investigations when teams need to prove who or what authenticated, from where, and when. That problem is especially relevant where machine identities, service accounts, and federated access logs must be retained long enough to support forensics and governance review.

Key questions

Q: How should security teams design log retention when investigation history must stay searchable?

A: Design retention in two layers. Keep recent data hot for detection, but store long history in low-cost object storage with a normalised schema and on-demand compute. That preserves fidelity without forcing every retrospective query through premium SIEM indexing, which is what usually makes long retention unaffordable.

Q: Why does expensive retrieval undermine retrospective identity investigations?

A: Because investigators do not just need the data to exist. They need to query it quickly enough to test hypotheses across a long time window. When restore jobs, scan charges, and cleanup steps add friction, teams narrow the search and miss identity abuse that unfolded outside the hot window.

Q: What are the signs that a retention model is failing security operations?

A: The clearest signs are shortened hot windows, delayed historical hunts, and investigators avoiding broad retrospective searches because the process is too slow or costly. In identity-heavy environments, that usually means the environment is preserving telemetry but losing practical forensic value.

Q: When should organisations move from SIEM-centric retention to decoupled data-lake retention?

A: When telemetry volume, query cost, and retention requirements no longer fit inside a single ingest-and-index model. If the team needs months of identity history for investigations, but only a small portion is queried routinely, decoupled retention becomes the more workable architecture.

Technical breakdown

Why ingest-based retention pricing breaks investigation depth

Most SIEM pricing models charge heavily at ingest and index time, which means organisations pay premium rates to make data searchable long before they know whether that data will matter. The result is a retention curve that encourages short hot windows, then pushes older data into slower tiers where every query becomes a cost decision. For SOC teams, this is not a storage issue alone. It is a governance issue, because the ability to reconstruct identity activity depends on whether historical telemetry remains practical to query, not merely whether it still exists.

Practical implication: Separate retention economics from investigation economics so long-term telemetry stays queryable without forcing every query through premium SIEM pricing.

How normalised object storage changes the query model

Decoupled retention works when logs are stored in low-cost object storage and analytics compute is spun up only for the hunt. Normalisation matters because a year-old Windows sign-in, SaaS login, and cloud console event all need to become comparable if the search is going to span identities and platforms. Without a common schema, the data is retained but functionally fragmented. With one schema and serverless query, teams can search across months of authentication history without maintaining a hot index for every record.

Practical implication: Standardise identity and telemetry schemas before moving cold history into object storage, or long retention will still be operationally brittle.

Why retrospective identity investigations fail when retrieval is expensive

The real failure point is not that data disappears. It is that the cost and delay of restoring it suppress the very questions investigators need to ask. If a campaign has been active for months, teams must sweep DNS, proxy, and authentication logs across a wide time range to prove exposure or absence. When that search requires restore jobs, scan charges, and table cleanup, the analysis gets narrower than the incident demands. That is particularly damaging for identity-led investigations, where authentication trails often provide the only reliable chronology.

Practical implication: Build an investigation path that preserves searchable identity history for months, not just an archive that can be recovered in theory.

NHI Mgmt Group analysis

Retention economics is now an identity governance problem, not just a storage problem. When organisations cannot afford to keep authentication history searchable, they lose the evidence needed to answer who accessed what, from where, and by which identity. That weakens human IAM review, NHI forensics, and account abuse investigations alike. The practical conclusion is that retention architecture now belongs in identity governance conversations, not only in SOC platform planning.

Long-term telemetry only has value when it stays queryable across identity sources. A year of raw logs in multiple dialects is technically preserved but operationally fragmented. Normalised schemas create the bridge between logs and identity questions, which is why searchability matters as much as storage location. For practitioners, the real metric is not retained gigabytes, but whether access and authentication events remain analytically usable at investigation time.

Cold storage economics changes the control objective from always-hot visibility to recoverable fidelity. That shifts the design question from how much can be indexed continuously to how much can be reconstructed on demand. In identity-heavy environments, that distinction matters because a delayed investigation can still be effective if the data is complete and coherent. The actionable takeaway is to align retention tiers with the maximum historical window required for access forensics.

Decoupled retention supports the kind of evidence preservation that IAM and NHI programmes increasingly need. Service account misuse, federated login abuse, and cross-cloud identity traces rarely fit neatly inside a 30-day analytics window. If organisations want year-scale retrospectives without paying year-scale SIEM premiums, they need a retention model designed around identity investigations rather than alert storage. The practical conclusion is to treat searchable history as a control objective.

Full-fidelity data lake architectures are becoming the governance answer to telemetry growth. As log volume rises and hot-index costs climb, organisations will increasingly choose architectures that preserve fidelity while delaying compute until the hunt begins. That does not eliminate the need for real-time detection, but it does protect the back-end evidence layer that investigations depend on. Practitioners should view this as a durability decision for security evidence, not a tooling preference.

What this signals

Retention architecture is drifting into the same governance category as identity lifecycle management. Once historical access evidence becomes too expensive to keep searchable, teams lose the ability to verify whether controls are working after the fact. For identity programmes, that means retention design now influences assurance, auditability, and incident reconstruction in the same way that lifecycle offboarding influences standing access risk.

Searchable history will increasingly be treated as a control boundary. Organisations that cannot query enough identity and telemetry history will make narrower security decisions, especially when service accounts, federated identities, or cloud access need retrospective validation. The practical response is to align storage tiering, schema normalisation, and retention policy with the investigative window your IAM and SOC teams actually use.

For practitioners

• Separate hot detection from long-term retention Keep recent, high-value events in a fast analytics tier, but move older telemetry into low-cost object storage with a governed query layer so retention depth is no longer bounded by ingest pricing.
• Normalise identity telemetry before archiving Map Windows, SaaS, and cloud authentication events into a shared schema such as OCSF so investigators can query one model across identity sources instead of translating every source at hunt time.
• Define an investigation retention window by threat model Set retention periods based on the longest realistic access, identity abuse, or insider investigation window your team must support, then validate that the stored data remains searchable throughout that period.
• Model retrieval cost as part of detection design Include restore fees, scan costs, and table rehydration time in your SOC economics so the team understands when a ‘retained’ dataset is effectively unavailable during incident response.

Key takeaways

• Retention is a governance problem when data remains stored but stops being practically searchable.
• Rising log volumes and premium ingest pricing are pushing organisations toward architectures that preserve fidelity without keeping everything hot.
• Identity-led investigations need long historical windows, which makes decoupled storage and normalised telemetry a control decision, not a platform preference.

Key terms

• Decoupled Retention: A retention model that separates storage from analytics compute so data can remain inexpensive to keep while queries run only when needed. It preserves long-term evidence without paying continuous indexing costs, which is especially useful when investigations are occasional but require complete history.
• Hot, Warm, and Cold Tiers: Storage layers that trade speed for cost. Hot tiers support immediate investigation and detection, warm tiers slow down but remain queryable, and cold tiers minimise storage cost while making retrieval slower and more expensive, which changes how usable the data remains in practice.
• Normalised Telemetry: Security data transformed into a common schema so events from different systems can be queried together. In identity-led operations, normalisation makes authentication, access, and session records comparable across cloud, SaaS, and on-premises sources instead of leaving them trapped in source-specific formats.
• Searchable Fidelity: The degree to which retained security data remains complete enough and accessible enough to support real investigations. High fidelity alone is not enough if the data is too costly, slow, or fragmented to query when analysts need it.

What's in the full article

Anomali's full article covers the operational detail this post intentionally leaves for the source:

• Pricing examples showing how different retention tiers change the economics of keeping telemetry searchable over time
• Worked cost comparisons for interactive retention, archive, and object-storage query models at SOC-scale volumes
• A practical explanation of how schema normalisation makes year-old identity data queryable across Windows, SaaS, and cloud sources
• The specific assumptions behind Anomali's cost-reduction claim in a large financial-institution deployment

👉 The full Anomali post breaks down the retention math, query trade-offs, and deployment model in detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and identity lifecycle control. It helps practitioners connect identity decisions to the operational realities of security evidence, access risk, and auditability.