Long-term log storage beyond SIEM costs needs open formats

At a glance

What this is: Axoflow argues that long-term log storage should move out of SIEM-centric retention and into open, self-describing object storage that stays queryable over time.

Why it matters: This matters because IAM, SOC, GRC, and cloud teams still need years of retrievable logs for investigations, audits, and compliance without paying analytics-tier storage costs.

Context

Long-term log retention is a governance and cost problem, not just a storage problem. Security teams often need to keep logs for 3 to 5 years, and sometimes 7 to 10 years or more, while the original SIEM model was built for search, correlation, and alerting rather than deep archive economics.

The identity angle is indirect but real: logs preserve evidence for access reviews, privileged activity investigation, and incident reconstruction across human, service, and machine identities. Once retention horizons stretch into years, the control question becomes whether the archive remains readable, portable, and defensible without depending on a single analytics platform.

Key questions

Q: What breaks when teams use a SIEM as long-term log storage?

A: The storage model becomes misaligned with the workload. SIEMs are built for ingestion, correlation, and investigation, but multi-year retention turns them into expensive archives. The usual failure is cost growth, followed by portability problems when metadata, query indexes, or retrieval logic stay trapped in one platform.

Q: Why do long-retention archives need open data formats?

A: Open formats preserve access after the original toolchain changes. A self-describing archive can still be read, queried, and rehydrated years later without depending on one vendor's index or database layer, which lowers lock-in and reduces the risk of losing evidence accessibility over time.

Q: How should security teams separate analytics from retention storage?

A: Use the SIEM for short-lived analytics and investigations, then place historical evidence in object storage built for low-cost retention. That split lets teams keep queryable records without forcing every stored event to carry analytics-grade cost and operational overhead.

Q: What happens if retained logs cannot be rehydrated later?

A: Investigations and audits become dependent on the original platform remaining available and intact. If the archive cannot be reopened independently, the organisation risks losing the ability to reconstruct access history, privilege use, or incident timelines when they matter most.

Technical breakdown

Why SIEM retention becomes the wrong control plane

SIEMs are optimised for high-value, high-churn analytics: ingest, correlation, dashboards, alerts, and ad hoc investigation. That model breaks down when the same platform is asked to act as a long-term archive, because cost follows analytics capability. At multi-year retention scales, most records are rarely queried, yet they still carry premium storage overhead. The result is a control mismatch: a system designed for rapid security operations becomes a billable repository for dormant evidence. Long-term retention needs efficient writes, low-cost capacity, and a way to query data later without turning every stored event into an ongoing analytics expense.

Practical implication: Separate long-term retention design from detection design so archive economics do not inherit SIEM analytics cost.

How open Parquet storage changes log durability

Apache Parquet is a columnar, self-describing file format, which means the data contains enough metadata to be read without a separate database index. For retention, that matters because it reduces dependency on a proprietary engine and keeps the archive portable across tools and environments. Columnar layout also helps because queries can read only the fields and row groups they need, while compression can be applied efficiently at the column level. In practice, this is less about storage fashion and more about evidence survivability: logs remain inspectable years later even if the original toolchain has changed.

Practical implication: Use open file formats for evidence retention so archived logs remain readable outside the original platform.

Why separating storage from query engines matters for investigations

Modern analytics stacks increasingly separate the storage layer from the query layer. That distinction is important for security data because it lets teams keep immutable or long-lived objects in cheap storage while using different engines for review, audit, or incident response. Embedded query tools can run against the same dataset without requiring a dedicated database to stay online for years. Axoflow's model also uses an intermediate write path to convert streaming log traffic into efficient Parquet files. For practitioners, the mechanism shows that long-term retention and practical queryability are not mutually exclusive if the architecture avoids tight coupling.

Practical implication: Design the archive so investigators can query retained logs without keeping a full database stack alive.

NHI Mgmt Group analysis

Open-format archive design is becoming a governance requirement, not a storage preference. When retention stretches from months to years, the question is no longer only where logs sit, but whether they remain usable after tooling changes, mergers, platform exits, or cloud shifts. A self-describing format reduces operational dependency and strengthens evidence continuity. For identity and security programmes, that means retention strategy now belongs in architecture governance, not just infrastructure procurement.

The real control gap is archive portability. Many organisations assume retention is solved once data is written to a platform, but that assumption fails when the query path, metadata, and retrieval logic are proprietary. The article's model highlights a broader pattern: if evidence cannot be rehydrated outside one vendor's stack, the organisation has created a future investigation risk. Practitioners should treat portability as part of control design, not as a convenience feature.

Long-term log storage should be evaluated against evidence use, not ingestion volume. Security teams often size storage for how much data arrives, not for how often it must be defensible, searchable, and admissible later. That leads to the wrong economics, especially in regulated environments. The better question is whether retained logs can support audit, forensics, and access reconstruction without forcing the organisation to keep paying analytics-tier prices forever.

Open formats create the conditions for multi-tool governance of security data. Once logs are self-describing, teams can choose different engines for different tasks, including audit review, incident investigation, and future AI-assisted analysis. That flexibility matters because governance over security evidence should not depend on a single retrieval model. For identity-heavy environments, the long tail of access history is only useful if it can be queried in more than one operational context.

What this signals

Archive portability is the control question hidden inside long-term log economics. If retained logs cannot be moved, queried, or rehydrated outside one engine, the organisation has not really reduced risk, only deferred cost and dependency. The stronger design pattern is to treat archival logs as durable evidence objects, not as inactive SIEM rows.

Identity evidence needs a storage model that survives platform churn. Access reviews, privileged activity investigations, and incident reconstruction often depend on logs that outlive multiple tool generations. That makes open formats and independent queryability part of identity governance, not just a data engineering preference.

For practitioners

• Separate hot analytics from cold retention Keep SIEMs focused on detection and short-retention analytics, then move multi-year retention into cheaper object storage designed for archive economics.
• Adopt open, self-describing storage formats Store retained logs in formats such as Parquet so the archive remains readable and queryable even if the original platform is no longer available.
• Validate rehydration after platform loss Test whether teams can still query historical logs if the SIEM, local indexes, or original query service are unavailable.
• Design for audit and identity reconstruction Retain enough structure in the archive to support investigations into privileged activity, service accounts, and other identity events years later.

Key takeaways

• Long-term log storage fails when organisations force retention workloads into SIEMs designed for analytics and alerting.
• Open, self-describing formats such as Parquet reduce lock-in and keep retained logs readable years later.
• The practical split is to keep hot detection in the SIEM while moving archive evidence to lower-cost storage that still supports query and rehydration.

Key terms

• Log Retention: Log retention is the policy for how long logs are kept before archival or deletion. The practical question is not storage alone, but whether the organisation can preserve evidence long enough for forensics, compliance, and legal hold requirements without expanding exposure unnecessarily.
• Open File Format: An open file format is a data structure that can be read by multiple tools without depending on a single vendor's proprietary index or database layer. In security logging, this matters because it improves portability, reduces lock-in, and keeps archived evidence accessible over time.
• Columnar Database: A columnar database stores data by column rather than by row, which makes it efficient for analytics over large datasets. This layout improves compression and accelerates queries that scan a subset of fields across many records. It is commonly used for log analysis, reporting, and other read-heavy workloads.
• Object Storage: Durable storage designed for large-scale file and dataset retention rather than low-latency transactional access. In modern analytics architectures, object storage acts as the system of record for raw and materialized data, enabling stateless compute layers to read and process data without owning persistence.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect identity controls to broader security and retention decisions.