Long-term log storage beyond SIEM costs needs open formats

At a glance

What this is: Axoflow argues that long-term security log retention should move out of SIEMs and into open, self-describing storage built for cheap, durable retention and later rehydration.

Why it matters: For IAM, PAM, NHI, and broader security teams, the storage model affects auditability, evidence retention, and the ability to investigate identity-related events years after they occurred.

By the numbers:

• 72% of organisations have experienced or suspect they have experienced a breach of non-human identities, 46% confirmed and 26% suspected.

👉 Read Axoflow's analysis of long-term log storage beyond SIEM costs

Context

Long-term log storage is a governance problem as much as a storage problem. Security teams need to retain evidence for years, but the platforms they use for active detection are usually optimised for query-heavy analytics rather than low-cost archival retention. That mismatch drives expense, creates dependency on proprietary formats, and makes rehydration harder when investigators need older records.

For identity-heavy environments, the issue is not just logging volume. Audit trails for human access, service accounts, API keys, tokens, and other non-human identities often have to survive long enough to support investigations, regulatory reviews, and incident reconstruction. A storage layer that preserves logs in open formats supports those needs better than treating the SIEM as a long-term warehouse.

Key questions

Q: How should security teams store logs for multi-year retention without SIEM cost blowouts?

A: Use the SIEM for active analytics and keep long-term evidence in open object storage that is cheaper to retain and easier to move. That split lowers cost, reduces lock-in, and keeps historical records available for audits and investigations without forcing the archive to behave like a live detection platform.

Q: Why do SIEMs become the wrong place for long-term evidence retention?

A: Because SIEMs are engineered for search, correlation, and alerting, not for low-cost archival storage. When retention extends across years, organisations pay for capabilities they rarely use, while still needing durable access to older logs for compliance, incident reconstruction, and legal review.

Q: What breaks when long-term log archives depend on proprietary indexes?

A: The archive becomes tied to the original platform, which makes later access fragile if the vendor changes direction, the product is retired, or the environment is migrated. Years later, teams may still have the data but lose the practical ability to query or rehydrate it without the original system.

Q: How do organisations keep retained logs usable across platform changes?

A: Maintain open, self-describing storage formats and test retrieval workflows regularly so the archive can survive tool replacement. Pair that with clear ownership, cataloguing, and access rules, because retention only helps if the evidence remains understandable and governed when someone needs it.

Technical breakdown

Why SIEMs become expensive as retention periods stretch

SIEMs are designed for ingest, correlation, search, alerting, and dashboards. Those features are valuable for active operations, but they make the platform expensive when the main requirement is retaining data that is rarely queried. Once retention stretches into years, organisations end up paying analytics-grade pricing for storage that functions mostly as cold archive. The result is a control-plane mismatch: one system is asked to serve both detection and evidence retention, even though those jobs have different economics and access patterns.

Practical implication: Separate hot detection retention from cold evidence retention so SIEM spend is reserved for active analytics.

How open Parquet storage changes log governance

Parquet is a columnar, self-describing file format that can sit directly on object storage. Because the file contains the metadata needed to read it, teams are not dependent on a proprietary index or a running database to reconstruct the data later. Columnar layout also improves compression and selective reads, which matters when only a small subset of fields is needed for audits or investigations. For security logs, that means the archive remains portable, queryable, and less exposed to platform lock-in over time.

Practical implication: Use open file formats on controlled object storage when the retention objective is evidence preservation, not live search.

Why separating the query engine from storage matters for long retention

Modern data architectures increasingly separate durable storage from the query layer. That allows teams to keep logs in inexpensive object storage while using whatever query engine fits the task, whether that is a distributed engine, an embedded runtime, or a notebook workflow during audit work. The architecture is operationally simpler because the archive does not need a dedicated database lifecycle just to remain accessible. In practice, this makes long-term evidence easier to keep and easier to hand off across tools and teams.

Practical implication: Choose architectures where the archive survives tool changes without requiring a database migration.

NHI Mgmt Group analysis

Log retention is an identity governance issue when audit evidence outlives tool cycles. Security programmes do not just store logs for compliance, they preserve proof of who or what accessed critical systems. That matters directly for human identity, service accounts, and other non-human identities because investigations often depend on historical access traces that must remain readable long after the original control stack changes. Teams should treat archive design as part of governance, not as a storage afterthought.

Long-term retention in a SIEM creates a cost-control trap that weakens operational discipline. When organisations keep years of logs in analytics-grade platforms, they often overspend to preserve data they rarely query. That encourages shortcuts such as shorter retention windows, partial indexing, or fragmented export processes. The named concept here is retention-silo pressure: the point at which evidence retention, investigation needs, and platform economics diverge so sharply that governance becomes inconsistent. Practitioners should design retention tiers explicitly rather than let cost pressure define policy.

Open formats reduce lock-in, but they also raise the bar for stewardship. Parquet on object storage is useful because it preserves portability and future readability, yet that only works if organisations maintain cataloguing, access control, and lifecycle discipline around the archive. In identity terms, the archive itself becomes a governed asset that needs access policies and ownership. Teams should manage retained logs with the same seriousness they apply to privileged records and audit evidence.

The separation of storage and query capability points to a broader security architecture pattern. Detection systems need speed, but compliance archives need durability, portability, and low operating cost. That split mirrors how IAM and PAM programmes separate standing access from temporary access: one model is for continuous operations, the other is for controlled evidence or task-bound use. Practitioners should stop assuming one platform must satisfy both missions equally well.

For regulated environments, the decisive question is not whether logs are searchable today, but whether they remain usable years from now. That requirement affects investigations, legal hold, and identity forensics across both human and machine accounts. Organisations should evaluate whether their archive preserves meaning, not just bytes, because evidence that cannot be re-read has limited governance value.

What this signals

Retention-silo pressure will become a recurring governance issue as organisations separate detection workloads from evidence archives. The practical signal is not just lower storage cost, but a stronger ability to keep records readable across tool changes, migrations, and audit cycles.

The identity angle is easy to miss: logs that prove access by humans, service accounts, and other non-human identities only matter if they remain usable when a case is reopened. That makes archive design part of identity assurance, not just data management, and it aligns naturally with retention and audit expectations in frameworks such as NIST Cybersecurity Framework 2.0.

For practitioners

• Separate hot and cold retention tiers Keep the SIEM focused on alerting and recent investigation windows, and move long-term evidence into open object storage designed for retention. This reduces analytics spend and keeps older records available without forcing every query through the same platform.
• Preserve logs in self-describing formats Store retained logs in formats such as Parquet so the archive does not depend on a proprietary index or a running database to remain readable. That makes future audits and incident reconstruction less dependent on a single toolchain.
• Treat archive access as governed access Apply explicit ownership, access control, and lifecycle rules to the retained log store, including who can query, export, or rehydrate records. The archive is security evidence, not passive storage.
• Test rehydration before an audit needs it Run recovery drills that pull historical logs from cold storage into a working query environment, then verify the results against the original record set. If the archive cannot be rehydrated quickly, it is not operationally durable.

Key takeaways

• Long-term log retention fails when teams force evidence archives into SIEMs built for active analytics.
• Open, self-describing storage formats reduce lock-in and make old records usable long after the original platform has changed.
• Security and identity teams should treat retained logs as governed evidence, with explicit ownership, access controls, and rehydration testing.

Key terms

• Cold Log Archive: A cold log archive is a retention store for security records that are kept long after active investigation windows close. It prioritises durability, low cost, and future readability over live analytics, which makes it suitable for compliance evidence and incident reconstruction.
• Self-Describing File Format: A self-describing file format contains enough metadata inside the file for another tool to read it without depending on a separate database or proprietary index. In security logging, that property improves portability and reduces the risk that retained evidence becomes trapped inside one platform.
• Rehydration: The process of moving data back from compressed or deduplicated backup storage into a form that can support normal production workloads. It becomes a bottleneck when restore volume is large, because storage designed for backup retention often cannot sustain live application I/O at scale.

What's in the full article

Axoflow's full blog post covers the operational detail this post intentionally leaves for the source:

• The Parquet and object storage implementation choices behind the storage layer
• The Arrow Flight write path and how it reduces serialization overhead
• The query and rehydration workflow across DuckDB, Python notebooks, and other engines
• The operational tradeoffs of running the archive on-prem, in cloud, or in air-gapped environments

👉 Axoflow's full post covers the storage architecture, query model, and retention tradeoffs in more implementation detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and identity lifecycle control. It helps practitioners connect retention, auditability, and access governance to the wider identity programme.