Next-Gen SIEM vs Traditional SIEM: What Changed

"Next-gen SIEM" is a marketing label, but underneath it sit real architectural changes, chiefly how log storage is priced and separated from detection, and those changes decide what a security team can afford to see. A traditional SIEM collects, indexes and correlates events in one tightly coupled platform, usually priced on ingested volume. The newer pattern splits that platform into a hot analytics tier for detection and a cheaper data lake tier for retention, hunting and investigation, often in open formats, with behaviour analytics and AI-assisted triage on top. This guide is for SOC leads, security architects and buyers deciding whether that split is worth the change: what actually differs, the cost model behind each, where UEBA and AI triage help, why identity telemetry is the test case, and what can go wrong in a move. It does not repeat SIEM fundamentals (see the SIEM Guide) or migration execution (see the SIEM Migration Guide).

Products named as examples are listed alphabetically; the order is not a ranking, and many products now offer both patterns.

Key takeaways

The real change is decoupling, not AI. Separating cheap, long-term storage from the expensive real-time analytics tier lets teams keep high-volume logs they used to drop, at the price of slower queries and no real-time rules on lake data.
Cost models moved, not disappeared. Government guidance from the Australian Signals Directorate and partners notes most SIEM pricing is based on ingested volume; lake tiers add separate charges for processing, storage and per-GB scanned queries, so cost shifts from ingest to query.
Detections still need the hot tier. Whatever the label, alerting and incident management run on the analytics tier. Putting a high-value source in the lake by mistake removes it from detection.
UEBA and AI triage depend on identity data. Behaviour baselines and automated triage are only as good as the sign-in, directory and cloud control plane telemetry feeding them, including service accounts and workload identities.
Migration risk is the largest cost. Detection coverage loss during a platform change is more likely than a licence overrun; plan parity measurement before choosing.

The short answer

A traditional SIEM is still a sound choice when log volume is modest and stable, retention needs are short, the team has deep skills on the current platform and most data is used for real-time detection. A decoupled, lake-backed design pays off when volume is high or growing fast (cloud, network flow, DNS, endpoint telemetry), long retention is required for investigation or regulation, data has to feed more than one tool, or the team is dropping useful logs to stay inside an ingest licence. In practice the choice is rarely binary: most current platforms offer an analytics tier and a lake or archive tier, so the real decision is which sources go where, under which pricing meter, and whether storage sits in a format and account you control.

What actually changed

Dimension	Traditional SIEM pattern	Next-gen (decoupled) pattern
Architecture	Collection, indexing, storage, correlation and search in one platform	Hot analytics tier for detection; separate lake tier for retention, hunting and analytics
Storage format	Proprietary indexes	Often open columnar formats such as Apache Parquet in object storage
Data ownership	Data lives inside the SIEM	Lake may sit in the customer's own cloud account and serve several tools
Pricing basis	Ingested GB per day, events per second or a pre-purchased cap	Ingest for the analytics tier plus lake ingestion, processing, storage and query or compute meters
Schema	Platform-specific normalisation model	Platform model plus open schemas such as OCSF for shared data
Analytics	Correlation rules, saved searches	Rules on the hot tier; UEBA baselines; notebooks and ML over the lake; AI-assisted triage
Retention economics	Long retention is expensive, so logs are dropped or archived offline	Years of retention in the lake at low cost, queryable without restore
Main trade-off	Simplicity, fast search over everything ingested	Cheaper breadth, but slower lake queries and more design decisions

Ingest cost models

The joint executive guidance on implementing SIEM and SOAR platforms, first published on 27 May 2025 by the Australian Signals Directorate's ACSC with CISA and other partners, states that most SIEM pricing models are based on the quantity of data ingested, that some products sell a pre-purchased ingestion cap instead, and that uncapped products can bring very significant costs if ingestion is not carefully managed. It also observes that teams commonly feed a SIEM more log data over time. That growth is the pressure behind the decoupled pattern.

Model	Basis	Encourages	Risk
Ingest volume (pay-as-you-go)	GB ingested per day	Filtering and dropping data before ingest	Useful but noisy sources are dropped; cost spikes during incidents or misconfigured sources
Commitment or capacity tier	Pre-purchased GB per day at a lower effective rate	Predictable spend	Paying for unused capacity, or overage when volume grows
Workload or compute	Search and processing capacity rather than volume	Ingesting more, querying carefully	Heavy hunting or many scheduled searches consume capacity
Lake tier meters	Lake ingestion, processing, storage per GB per month, query per GB scanned	Keeping high-volume logs cheaply	Unbounded queries over years of data create a new, less visible cost
Entity or asset based	Users, hosts or assets protected	Collecting everything from covered assets	Mismatch when telemetry per asset grows (for example cloud workloads)

Two vendor examples show how meters split in practice. Microsoft documents Sentinel's analytics tier as pay-as-you-go per GB or commitment tiers starting at 100 GB per day (a tier can be raised at any time but lowered only once every 31 days), with all data retained free for the first 90 days. Its data lake tier is billed separately for lake ingestion and processing on lake-only tables, storage per GB per month at a stated uniform 6:1 compression rate, and queries per GB of uncompressed data scanned. Splunk's pricing page lists ingest pricing, workload pricing and activity-based pricing as options for its platform and Enterprise Security. The point for buyers is not which meter is cheaper but which one matches the team's behaviour: a team that ingests a lot and queries a little suits a different model from one that hunts constantly over long history.

Availability: pricing, tiers and meters change frequently and differ by region and contract. Confirm current terms on each vendor's pricing page and in the contract, not from this guide.

Data lake decoupling

How the architecture works

Events arrive through collectors or a pipeline layer. High-value, detection-relevant sources (identity, endpoint alerts, cloud control plane, privileged activity) go to the analytics tier, where rules, alerting, incidents and fast hunting run. Everything else goes to the lake tier: object storage in columnar files, queried with the SIEM's language, SQL engines or notebooks. Data can move upward when needed. Microsoft describes analytics-tier data as mirrored to the lake tier to keep a single copy, lake retention of up to 12 years, Parquet as the open format, and one-off or scheduled jobs that promote lake data into the analytics tier. Elastic achieves a similar effect inside its own platform: searchable snapshots back the cold and frozen tiers with object storage, and partially mounted indices in the frozen tier keep only recently searched data in a local cache, so searches that miss the cache are slower; Elastic documents searchable snapshots as requiring an Enterprise licence. Amazon Security Lake takes the lake-first approach: it stores data as Parquet in S3 buckets in the customer's account, normalises AWS sources to OCSF, and serves subscribers through either direct data access or query access via Lake Formation tables.

What decoupling gives you

Breadth: flow, DNS, proxy and verbose endpoint logs kept for months or years rather than dropped.
Investigation depth: historic queries for dwell time and scoping without restore jobs.
Tool independence: one copy in an open format can feed the SIEM, a data science team and another detection tool. OCSF, a Linux Foundation project since November 2024, provides a vendor-agnostic schema for that shared copy; see the OCSF Guide.

What decoupling costs you

Real-time detection on lake data: alerting and incident management sit with the analytics tier. Lake data supports scheduled queries, jobs and notebooks, which are later and coarser.
Query cost visibility: per-GB-scanned charges make a careless hunt over years of data expensive; set query guardrails.
Design work: someone must decide, per source, which tier, which schema and which retention, and revisit it when detections change. For the wider architecture see the Security Data Lake Guide.
A new attack surface: the lake, its buckets, catalogue and subscriber credentials need the same protection as the SIEM. See Securing the SIEM itself.

Threat model for each option

Traditional SIEM: the main risk is blindness by budget. Sources that would show lateral movement, data staging or cloud abuse are filtered out to stay within licence, and retention is too short to scope a long-dwell intrusion. The platform itself is a high-value target holding credentials for every connected source. Decoupled SIEM: the main risks are misplacement and sprawl. A source placed in the lake loses real-time rules; detection content assumes a field that the lake schema maps differently; more storage locations, catalogues and subscriber identities widen access to sensitive logs. Both designs share the identity risk: integration tokens, connector app registrations and forwarder credentials are non-human identities with broad read access, and an attacker who disables or edits logging (ATT&CK T1685.001 Disable or Modify Windows Event Log is one example) attacks both equally.

UEBA and AI triage

User and entity behaviour analytics builds baselines per user, host, IP address and application, then scores deviations. Vendor UEBA modules typically combine a near real time per-event priority score with batch anomaly scoring and peer groups drawn from directory group membership, and they rely on identity provider and directory logs as key sources. The lesson generalises: UEBA quality is a function of identity and directory data. For the discipline itself see the UEBA Guide.

AI-assisted triage, summarising alerts, suggesting next steps and drafting investigation queries, is now offered across most platforms. Treat it as an analyst productivity feature rather than a detection capability, and evaluate it on your own incidents: how often it is right, whether it shows its evidence, which data it can read, and what identity it acts with when it queries or takes actions. Agentic triage that can run queries or containment steps needs scoped permissions and logging like any other privileged automation. See the AI SOC Guide.

Capability	What to test	Red flag
UEBA baselines	Which sources feed it; whether service accounts and workload identities are profiled; peer grouping logic	Only human interactive sign-ins are modelled
Risk scoring	Whether scores are explainable and tunable	Opaque score with no contributing events
AI triage summaries	Accuracy on a sample of past incidents; citations to raw events	Confident summaries with no evidence links
Agentic actions	Permission scope, approval gates, audit trail	Broad write access under a shared identity
Lake analytics	Notebook and ML access to lake data; cost of compute	Compute billed per hour with no quota controls

Identity telemetry as the test case

Identity data is where the two designs are best compared, because it is high value, moderate volume and essential to both detection and UEBA. Place these sources in the analytics tier on any platform:

Identity provider sign-ins and audit: interactive and non-interactive sign-ins, service principal and managed identity sign-ins, consent grants, role assignments and conditional access changes.
Directory and Kerberos: Windows Security events 4624 and 4625 (logon success and failure), 4768 and 4769 (Kerberos TGT and service ticket requests) and 4720 or 4728 for account creation and group membership changes, collected from domain controllers.
Cloud control plane: role assumption, access key creation and policy changes, which surface T1078.004 Cloud Accounts and T1098 Account Manipulation.
Token use: OAuth application activity and token issuance, relevant to T1550.001 Application Access Token.
PAM and secrets platforms: privileged session starts, credential checkouts and vault access.

Non-human identities usually generate far more events than people, so they are the first candidates to be filtered out under an ingest model, and also the identities attackers most often reuse. The ITDR vs SIEM page covers where identity-specific detection sits; the Detection Engineering for Non-Human Identities Guide covers rules for service accounts and tokens.

Decision criteria

Question	Points to traditional	Points to decoupled
Daily volume and growth	Stable, modest volume	High or fast-growing cloud and network telemetry
Retention requirement	Months	Years, queryable
Logs currently dropped for cost	Few	Many, including flow, DNS or endpoint detail
Data consumers	SOC only	SOC, threat hunting, data science, compliance, other tools
Team skills	Deep on current query language and content	Data engineering capacity for schemas, pipelines and lake governance
Data residency and ownership	Vendor-hosted storage acceptable	Logs must stay in own account or region
Hunting pattern	Frequent fast search over recent data	Occasional deep search over long history

Small organisations: a single platform with a hot tier and a low-cost archive, or a managed detection service, is usually better than building a lake; the joint guidance stresses that running a SIEM needs skilled staff on a full-time basis and asks executives to decide whether they can implement it in-house at all. Large organisations: tiering is close to mandatory at scale; the decision is whether the lake sits inside the SIEM vendor's platform or in an account and format you control.

Worked scenario

A retailer runs an ingest-licensed SIEM at its cap. To stay within it, the team stopped collecting DNS and VPC flow logs and keeps 90 days of everything else. A review finds that two past investigations could not establish when an intrusion started. The team evaluates a decoupled design: identity provider, EDR alerts, cloud control plane, firewall denies and PAM events stay in the analytics tier; DNS, flow and proxy logs go to the lake with 13 months of retention; endpoint process telemetry goes to the analytics tier for 30 days, then the lake. Before committing, they model three costs: analytics ingest, lake storage after compression, and the per-GB query cost of their typical monthly hunts. They also check every existing rule's data source, and find six detections that rely on DNS fields, which they rebuild as scheduled lake queries with a documented, longer detection latency. Parity is measured by replaying three past incidents through both designs.

Migration risk

Execution belongs in the SIEM Migration Guide; the risks that should shape the decision are these:

Detection loss: rules translated between query languages or schemas silently stop matching. Sigma, a generic YAML signature format converted to platform query languages through pySigma backends and sigma-cli, reduces the rewrite but does not remove field-mapping work; see the Sigma Rules Guide.
Tier misplacement: a high-value source landing in the lake removes it from real-time rules.
Dual running cost: sending data to both platforms during parity testing doubles ingest for that period.
Lost history: historic data left in the old platform is unavailable when its licence ends unless exported.
Credential sprawl: new connectors, collector tokens and subscriber roles created in a hurry, with old ones never revoked.

Inventory sources with measured daily volume, current tier, detections that use each, and retention need.
Classify each source as detect (hot), investigate (lake), or comply (archive).
Model cost under each meter using your measured volumes and real query patterns.
Pilot with one data domain, such as network telemetry, in the lake; keep detections unchanged.
Measure detection parity by replaying known-bad samples and past incidents.
Cut over by domain, keeping the old path available until parity holds for an agreed period; roll back by re-pointing collectors.

Detection and measurement

Percentage of detection rules whose data sources sit in the analytics tier (target 100 percent for real-time rules).
Logs dropped for cost: count of sources not collected, reviewed quarterly.
Cost per GB retained and per investigation, split by ingest, storage and query.
Mean time to scope an incident's start date, using historic data.
Identity coverage: share of human, service and workload identities whose sign-in and activity logs reach the analytics tier.
Data pipeline health: sources silent beyond their expected interval.

Common failure patterns

Buying the label. Selecting on AI features without testing ingest, tiering and query costs on real volumes.
Lake as a dumping ground. Data lands in the lake with no schema, owner or retention rule, and nobody can query it when needed.
Unbounded query spend. Per-GB-scanned charges grow unnoticed because hunts and scheduled jobs have no guardrails.
Licence-tier blind spots. Security-relevant events that a source only records at a higher logging or licence tier were never collected, so the SIEM design assumed data that did not exist. Check which events each source's licence actually produces.
Service accounts filtered out. High-volume non-human sign-ins dropped to save cost, removing the identities attackers most often reuse.

Framework mapping

Framework	Reference	Relevance
NIST SP 800-53 Rev 5	AU-6(3) Correlate Audit Record Repositories; SI-4 System Monitoring; SI-4(2) Automated Tools and Mechanisms for Real-time Analysis	Correlation and real-time analysis in the hot tier
NIST SP 800-53 Rev 5	AU-11 Audit Record Retention; AU-9 Protection of Audit Information	Lake retention and protection of the lake itself
NIST CSF 2.0	DE.CM-01, DE.CM-03, DE.CM-09; DE.AE-02, DE.AE-03	Monitoring coverage and correlation from multiple sources
CIS Controls v8.1	8 Audit Log Management (8.9 centralise audit logs, 8.10 retain audit logs)	Centralisation and retention
ISO/IEC 27001:2022 Annex A	8.15 Logging; 8.16 Monitoring activities	Logging and monitoring controls
MITRE ATT&CK v20	T1078.004; T1098; T1550.001; T1685.001	Identity and logging techniques the telemetry must cover

Practitioner checklist

Measure daily volume per source before talking to vendors.
Map every detection rule to its sources and confirm those sources are in the analytics tier.
Model cost under ingest, commitment, workload and lake meters with real query patterns.
Decide where the lake lives, in which format, and who owns its access.
Keep identity provider, directory, cloud control plane and PAM logs in the hot tier, including non-human identities.
Test UEBA and AI triage on your own past incidents before relying on them.
Set query guardrails and budget alerts on lake scans and compute.
Plan parity measurement and rollback before any migration.

Next-Gen SIEM vs Traditional SIEM: Cost Models, Data Lakes and the Real Decision