Security Log Cost Optimisation Guide

Security logging bills grow faster than security budgets, and the usual response, cutting sources until the invoice fits, removes exactly the evidence an investigation needs months later. Cost optimisation done well is a detection engineering exercise, not a procurement one: you decide what each log is for, put it in the cheapest store that still serves that purpose, and prove after every change that detections still fire. This guide covers how to model where logging cost comes from, how to score sources by security value, how to tier storage and retention, which reductions are safe, and how to govern the result so savings do not quietly turn into blind spots. It is for SOC managers, detection engineers and security architects who own a SIEM or security data budget. The mechanics of pipelines and retention obligations have their own guides, linked below; this page is the cost decision method that sits across them.

Key takeaways

Know which cost you are cutting. Ingest, hot retention, long-term storage and query or rehydration are priced differently, and a saving in one often moves cost to another.
Score sources by use, not by volume. A small, high-value source like identity provider sign-ins stays in the analytics tier; a huge, rarely queried source moves to cheaper storage.
Reduce the copy, keep the evidence. Send a trimmed stream to the expensive tier and the full stream to cheap storage, so investigations can still reach the original.
Retention is set by obligation and dwell time. NCSC suggests a minimum of six months; CIS Control 8 sets a 90-day floor; some regimes require far longer. Cheap tiers make long retention affordable.
Test detections after every cut. A lower bill with fewer rules firing is a coverage loss, not a saving.

Where security logging cost comes from

Most platforms price along four dimensions. Map your bill to them before changing anything, because the right lever depends on which one dominates.

Cost driver	What grows it	Main levers	Risk if pulled too hard
Ingest (per GB or per event into the analytics tier)	New sources, verbose formats, duplicate forwarding	Route by value, trim fields, deduplicate, move sources to a lower tier	Real-time detections lose their input
Hot retention (searchable, indexed)	Long hot windows for every source	Shorter hot window per source, archive the rest	Hunts and investigations slow down or need rehydration
Long-term storage (archive, object storage, lake)	Long retention, uncompressed formats	Compression, columnar formats, lifecycle to colder classes	Restore takes hours or days; deletion before obligations end
Query, search and rehydration	Wide hunts over cheap tiers, scheduled searches, restores	Partitioning by time and source, scheduled job review, query budgets	Analysts avoid needed searches because they are charged per scan

Also count the hidden costs: collection agents and forwarders, network egress from cloud regions, licence tiers that gate which audit events a SaaS or cloud service writes at all (see the SaaS Audit Log Retention and Licensing Guide), and analyst time lost to noisy data. A cheaper log that doubles investigation time is not cheaper.

Scoring log sources by security value

NCSC advises choosing log sources through threat modelling and documenting why each one is collected, and names authentication, security control and DNS logs as must-have quick wins. Turn that into a scored inventory so cost decisions are explicit. The scoring model below is our recommendation.

Question per source	High value signal	Low value signal
How many active detection rules use it?	Several rules, including high severity	None
Is it used in investigations and hunts?	Queried in recent incidents or hunt reports	Not queried in the last 90 days
Does a regulation, contract or internal standard require it?	Named in an obligation with a retention period	No obligation
Can the same question be answered elsewhere?	Unique view (for example cloud control plane audit)	Fully duplicated by another source
Does it show identity or privilege activity?	Sign-ins, role changes, token issuance, PAM sessions	Performance and health telemetry

NCSC also notes you may not need every source if you can answer the incident questions another way, and that centralising everything is resource-intensive: centralise what you need for cross-source queries and use native consoles or APIs for the rest. Build the field-level view too: list which fields each rule and saved hunt actually uses, so trimming decisions are based on evidence. The SIEM Log Source Onboarding Checklist covers recording this at onboarding.

Tiering: matching each log to the right store

Most SIEMs and cloud logging services now offer at least three tiers with different prices and capabilities: a full analytics tier for real-time rules, a lower-cost tier with limited query features, and an archive or data lake tier for long retention. Names, features and prices differ by platform, so confirm them in your platform's current documentation. Our recommended placement by data type:

Data	Primary use	Recommended tier	Reduction allowed
Identity provider sign-ins, directory and role changes, PAM sessions, token issuance	Real-time detection, identity threat detection	Analytics	Field trimming only
Cloud control plane audit logs	Detection of privilege, logging and policy changes	Analytics	Field trimming only
EDR alerts and key process telemetry	Detection and response	Analytics for alerts; lower tier or lake for full telemetry	No sampling
Firewall allow flows, DNS, web proxy	Hunting, beaconing analysis, scoping	Lower tier or lake, with summaries or indicator matches in analytics	Aggregation for the analytics copy only
Application debug, health checks, verbose diagnostics	Troubleshooting	Operations tooling, not the security store	Drop from the security pipeline after review
Everything required by regulation	Legal hold, regulatory retention	Immutable archive	None beyond compression

Before moving a source down a tier, check three things: can scheduled detections still run against it (many lower tiers restrict this), how long does a restore or rehydration take during an incident, and what does a wide query cost. The Security Data Lake Guide covers lake design and the SIEM Guide covers the analytics tier.

Filtering and reduction without losing evidence

Filtering mechanics, ordering and redaction are covered in the Security Data Pipeline Guide; here is the cost-focused view of which reductions pay off and where they bite.

Start from defaults, then remove the least useful: this is NCSC's advice for finding the cost and benefit balance, and it beats starting from a blank sheet because you see what each removal costs in detection terms.
Deduplicate with a count: repeated identical events from retries or duplicate forwarders are pure waste. The OpenTelemetry Log DeDuplication Processor (alpha stability, as of November 2026) emits one log with a count of the duplicates, which keeps rate-based detections workable.
Trim fields, not events: verbose vendor fields that no rule or hunt uses are the safest saving. Keep command lines, script blocks and URLs whole; the malicious part is often at the end.
Never sample security events: sampling authentication, audit, EDR or cloud control plane events breaks thresholds and counts. Keep sampling for traces and performance metrics.
Make filter failures visible: in the OpenTelemetry filter processor, the default error_mode of ignore logs errors and continues, silent continues without logging, and propagate drops the payload. Prefer the mode that logs, and alert on filter errors, so a broken condition does not silently drop or pass data.
Drop at the source with care: turning a log off at the device is the biggest saving and the hardest to undo, because nothing is kept anywhere. Reserve it for data with no security use after review.

Retention is where cost and obligation collide. Set it per source and tier, starting from the longest requirement that applies. Full retention obligations and privacy limits are in the Log Management and Retention Guide.

Reference point	What it says
NCSC introduction to logging for security purposes	Hold logs for a minimum of six months, longer if budget allows, because attacks can go undetected for months; review retention over time
CIS Controls v8.1, Safeguard 8.10	Retain audit logs for a minimum of 90 days
OMB M-21-31 (US federal agencies, August 2021)	Minimum of 12 months in active storage and 18 months in cold data storage for most log categories; full packet capture kept for 72 hours

The M-21-31 split between active and cold storage is a useful model even outside government: keep a searchable window long enough for most investigations, then a cheaper, slower archive long enough for late-discovered intrusions and legal needs. Our recommendation is to set the hot window from your own incident data (how far back did investigations actually search last year?) rather than from a vendor default.

Protect the archive: NCSC advises restricting write access, making changes detectable and protecting logs from tampering. Cheap storage must still be immutable and held in a separate account, because attackers target logs (ATT&CK T1685 Disable or Modify Tools).
Test restores: run a timed restore from the archive at least quarterly, as we recommend, so you know the real delay before an incident depends on it.
Delete on schedule: keeping logs past their obligation adds cost and privacy exposure. Apply lifecycle deletion once every requirement has lapsed.

Implementing a cost optimisation programme

Baseline: export volume and cost by source, tier and query workload for the last three months. Record the current detection rule count and a detection coverage map (for example by ATT&CK technique).
Inventory use: for each source, list the rules, hunts, dashboards and obligations that depend on it, and the fields they use.
Score and decide: apply the value scoring and assign each source a tier, hot window, archive period and allowed reductions. Record the owner and reason for each decision.
Build a detection regression test: replay sample events or run attack simulations for the rules affected by each change, before and after.
Change in small batches: move or trim a few sources at a time, run the regression test, compare rule firing rates for two weeks, then continue.
Roll back cleanly: keep the full stream flowing to cheap storage during each change so a bad filter can be reverted and the gap backfilled.
Govern: require a value case for every new source and a detection review for every new drop rule; review the whole inventory every six to twelve months, matching NCSC's review cadence for logging strategy.

Decision guidance

Cut ingest or retention first: if ingest dominates, tier and trim; if storage dominates, shorten hot windows and move to archive. If query cost dominates, fix partitioning and scheduled searches before touching data.
Commitment pricing: committed volume tiers lower unit cost but punish over-commitment; size them on the post-optimisation baseline, not today's volume.
Small organisations: keep identity, cloud control plane, endpoint alerts and email security in the analytics tier, archive the rest, and use the platform's native tiers rather than building a pipeline.
Large estates: a pipeline layer that routes by value pays for itself at scale and reduces lock-in; see the Security Data Pipeline Guide. Add showback so teams that add verbose sources see their cost.

Where identity fits

Identity logs are usually small and high value, and they are the last place to cut. Sign-ins, token issuance, role assignments and service principal changes are how most cloud and SaaS intrusions are found, and non-human identity activity often appears nowhere else. At the same time, the pipeline's own collectors, forwarders and API integrations hold credentials with read access to every source; scope them tightly and rotate their secrets. See the Identity Telemetry Requirements guide.

Detection and measurement

Cost per used GB: spend on data that at least one rule, hunt or obligation uses, divided by total spend; aim for this share to rise.
Detection coverage held: ATT&CK techniques or priority use cases covered before and after each change; any drop needs a written acceptance.
Rule firing rate change: per affected rule, two weeks before and after a change.
Restore time: measured time to search or rehydrate archived data in a drill.
Silent source alerts: sources that stop arriving, which NCSC advises alerting on.
Unowned sources: sources in the analytics tier with no owner or documented use.

Framework mapping

Activity	NIST CSF 2.0	NIST SP 800-53 Rev 5	Other
Deciding which logs are generated and collected	PR.PS-04	AU-12	CIS Control 8
Storage capacity and tiering	PR.PS-04	AU-4	CIS Safeguard 8.3
Retention by tier	PR.PS-04	AU-11	CIS Safeguard 8.10; OMB M-21-31
Protecting archived logs	PR.PS-04	AU-9	ATT&CK T1685
Monitoring and alerting on missing sources	DE.CM-09	AU-5, SI-4	-

Common failure patterns

Cutting by volume: the biggest sources are removed first, regardless of how many detections depend on them.
Dropping without a copy: filters applied before any archive means the data is gone for every future investigation.
Tier moves that break rules: a source moved to a lower tier no longer supports the scheduled detections that relied on it, and nobody notices until an incident.
Retention set to the default: hot windows and archive periods follow product defaults rather than obligations or real investigation needs.
Untested archives: the restore path is first used in a live incident and takes days.
One-off clean-ups: without a value case for new sources, costs return within a year.

Practitioner checklist

Break the logging bill into ingest, hot retention, long-term storage and query cost.
Build a source inventory with rules, hunts, obligations and fields used for each source.
Score every source by security value and assign a tier, hot window and archive period.
Keep identity, cloud control plane and endpoint alert data in the analytics tier.
Route the full stream to immutable, low-cost storage before applying reductions.
Never sample security events; deduplicate with counts and trim unused fields instead.
Run a detection regression test before and after each change.
Set retention from obligations and your own investigation history, and test restores quarterly.
Alert on sources that stop arriving and on filter errors.
Review the inventory every six to twelve months and require a value case for new sources.

Security Log Cost Optimisation Guide: Tiering, Filtering and Retention Without Blind Spots