Log Management and Retention Guide

Most investigations stall on the same problem: the log that would have answered the question was never generated, was overwritten after seven days, or was deleted by the attacker. Log management prevents this: deciding what each system records, moving it reliably to central storage, keeping it long enough at an affordable cost, proving it has not been altered, and disposing of it on schedule. This guide covers collection and forwarding, log rotation, retention periods from standards and regulation, storage tiers, integrity controls and the logging of service account, API key and workload identity activity. It is for security engineers, platform teams and compliance owners, and complements our guidance on detection platforms: see the SIEM Guide.

Key takeaways

• Retention is a per-category decision. Identity, cloud control plane and security tool logs justify a year or more; verbose debug and flow data rarely need more than weeks in searchable form.
• Federal retention rules changed in 2026. OMB M-26-14 rescinded M-21-31 and now asks US agencies to keep logs actively searchable for at least 6 months and retrievable for a year after creation.
• Defaults are not a policy. Microsoft Entra keeps sign-in and audit logs for 7 days on the Free tier and 30 days on P1 or P2, and Google Cloud Data Access audit logs are off by default for most services.
• Integrity must be provable. Forward logs off the host quickly, store them where the systems that produce them cannot delete them, and hash or sign them so tampering is detectable.
• Non-human identity activity is easy to miss. Service principal, managed identity, access key and token events are separate log categories that are often not collected at all.

What log management covers

NIST defines log management as "the process for generating, transmitting, storing, accessing, and disposing of log data." The final NIST SP 800-92 Guide to Computer Security Log Management dates from 2006. SP 800-92 Rev. 1 Cybersecurity Log Management Planning Guide was released as an initial public draft on 11 October 2023 (comments closed 29 November 2023), and as of October 2026 the CSRC page still lists only that draft. Treat Rev. 1 as draft guidance.

Rev. 1 is a playbook whose four plays make a good programme structure:

• INV, update logging-related inventories: log source types, logging infrastructure, logging use cases, requirements and work roles.
• TS, define the target state: what must be generated, how it is stored and transferred, who can access it and how it is disposed of, including "how long each log event should or must be preserved at the log source."
• GRC, document gaps and their root causes: for example a missing log traced to misconfiguration, poor enforcement or lack of storage.
• PMG, plan to mitigate the gaps: costed project plan, compensating controls, revised policies and stakeholder review.

The draft gives no retention periods; that decision sits with you.

Deciding what to log

The 2024 joint guidance Best practices for event logging and threat detection, led by ASD's ACSC with CISA, NCSC-UK and other partners, notes that "event log quality refers to the types of events collected rather than how well a log is formatted." Start from the questions an investigator must answer (who, what, when, from where, with what result) and work back to sources.

Each event should carry timestamp, host, identity, action, object and outcome, ideally as "key-value-pairs" as the joint guidance recommends.

What not to log

Logging of excessive data (CWE-779) wastes storage and creates a second copy of sensitive data. OWASP lists authentication passwords, access tokens, encryption keys and other primary secrets as data to remove, mask, hash or encrypt before logging, and suggests hashing session identifiers when you still need to correlate a session. Under GDPR, data minimisation (Article 5(1)(c)) applies to logs containing personal data, while Recital 49 recognises processing "strictly necessary and proportionate" for network and information security as a legitimate interest. Sanitise event data against log injection.

Collection, forwarding and centralised logging

Centralised logging means getting a copy of each event off the system that produced it before an attacker or a full disk can remove it. The joint guidance states that "Forwarding event logs to a centralised and secure storage capability prevents the loss of logs." The NCSC adds a caution that "Taking all logs to a centralised source is a never-ending (and resource intensive) job", so centralise the priority sources and know where the rest can be queried.

• Syslog: RFC 5424 defines the message format. Plain UDP 514 is unauthenticated, unencrypted and drops messages silently under load; use syslog over TLS (RFC 5425, TCP 6514) between relays and the central syslog server.
• rsyslog and similar relays: use disk-assisted action queues so events buffer locally when the destination is unreachable, and RELP (with TLS where supported) so messages are not lost when a connection drops.
• Agents: Windows Event Forwarding, vendor collectors and OpenTelemetry collectors handle structured logs better than syslog, and each holds credentials to the log platform.
• Cloud-native export: organisation-level CloudTrail trails, Azure diagnostic settings and Google Cloud log sinks, configured so new accounts and subscriptions inherit them.
• Filter in the pipeline, not at the source: the joint guidance suggests "filtering event logs before sending them to a SIEM or XDR". Keep the full copy in cheaper storage.

Log rotation and local overflow

Local logs are a buffer, not an archive, but must not fail in a way that loses evidence. On Linux, logrotate controls rotation with directives such as daily, rotate, maxage, size and compress; prefer the default create-and-signal method over copytruncate, which can lose lines written during the copy. For systemd-journald, set SystemMaxUse so the journal cannot fill the disk. On Windows, size the Security log for several days at peak and choose "overwrite events as needed" only once forwarding is proven; event 1104 records a full Security log. NIST SP 800-53 control AU-5 (response to audit logging process failures) asks you to decide in advance whether a full audit log overwrites the oldest records, alerts or halts the system, and OWASP warns that logging must not "deplete system resources, for example by filling up disk space."

Time synchronisation

Correlation across sources fails if clocks drift. The joint guidance says "Time servers should be synchronised and validated throughout all environments", recommends UTC, ISO 8601 timestamps with the year first, multiple accurate time sources, and notes that "Timesharing should be unidirectional." M-26-14 requires "a consistently accurate timestamp" traced to a designated time source. Monitor NTP offset and record both event time and ingestion time.

How long to keep logs

There is no single correct retention period. The joint guidance warns that "Default log retention periods are often insufficient" and that "it can take up to 18 months to discover a cyber security incident", and advises that "Logs that are crucial in confirming an intrusion and its impact should be prioritised for longer retention." Set retention per log category, take the longest applicable requirement, then confirm that any personal data in it can still be justified.

M-26-14 draws a useful distinction: "searchable" data is usable immediately, while "retrievable" data needs preparation such as thawing cold storage. PCI DSS makes the same split. Write your schedule in those two columns. A practical default without a stricter sector rule: 90 days searchable for detection sources, 12 months retrievable for identity, cloud control plane, security tool and edge logs, and longer only where a regulation, contract or legal hold demands it. Sector regulators may set longer periods: confirm with legal counsel.

Storage tiers and log storage costs

The joint guidance recommends "data tiering such as hot and cold storage". Reducing cost without losing telemetry mostly means changing where and in what form data is kept, not deleting it.

Never filter identity, privilege change or logging configuration events to save money; they are small and decisive. Document the archive schema: an archive nobody can parse in two years is not retention.

Log integrity and tamper resistance

Attackers remove the evidence of what they did. MITRE ATT&CK v20 groups this under T1685 Disable or Modify Tools in the Defense Impairment tactic (TA0112), including T1685.001 Disable or Modify Windows Event Log, T1685.002 Disable or Modify Cloud Log, T1685.005 Clear Windows Event Logs and T1685.006 Clear Linux or Mac System Logs. The technique explicitly covers "interfering with logging pipelines and forwarding mechanisms (e.g., SIEM ingestion)", and T1685.002 describes downgrading a Microsoft 365 licence from E5 to E3 to disable mailbox audit logging. ATT&CK records that Salt Typhoon "has disabled logging and disabled the distribution of logs to log aggregators": see the Salt Typhoon Telecom Intrusions 2025. In the Cutting Edge campaign against Ivanti Connect Secure, ATT&CK notes that threat actors disabled logging on the appliances: see the Ivanti Connect Secure Exploitation 2024.

• Immutable logs: write archives to WORM storage (for example S3 Object Lock or Azure immutable blob storage with time-based retention) so no administrator can delete them early.
• Separate the log store: use a dedicated cloud account, subscription or project with its own administrators; the joint guidance asks for "a separate or segmented network with additional security controls".
• Cryptographic verification: the joint guidance calls for "methods of cryptographic verification to ensure the integrity of event logs in-transit and at rest." AWS CloudTrail log file integrity validation, for example, hashes each file with SHA-256, delivers an hourly signed digest file, and chains each digest to the previous one. It must be enabled per trail.
• Least privilege on logs: separate read, write and delete roles, and log access to the logs themselves.
• Alert on gaps: treat a source that stops sending as a security event. Watch for Windows events 1102 (log cleared) and 4719 (audit policy changed), and cloud events such as CloudTrail StopLogging, DeleteTrail and Google Cloud UpdateSink.

Where identity fits

Service accounts, workloads and integrations perform a large share of actions in cloud and SaaS estates, and their activity is logged in categories that are easy to miss.

• Collect every sign-in category: Microsoft Entra separates interactive user, non-interactive user, service principal and managed identity sign-ins, and the legacy view "only includes interactive user sign-ins." Export all four through diagnostic settings. See the Cloud Workload Identity Guide.
• Enable data access logging where it matters: in Google Cloud, "Except for BigQuery, Data Access audit logs are disabled by default because they can generate large volumes of data", while Admin Activity logs are always written. Enable Data Access logs for secrets, key management and sensitive data services at least.
• Record which credential was used: the caller identity alone is not enough when one service account has several keys. Keep the access key ID, token or certificate identifier in the event so a leaked credential can be scoped. See the API Key Management Guide.
• Log credential lifecycle events: creation of service principal secrets, IAM access keys and service account keys, and role assignments to non-human identities. M-26-14 asks agencies to collect logs that determine "the identity used for performing operations in applications and on systems."
• Protect the logging pipeline's own identities: collectors, forwarders and SIEM connectors hold API keys and service accounts with read access to everything. Vault and rotate them. See the Service Account Security Guide.
• Keep secrets out of logs: application and CI/CD logs often capture tokens and connection strings. Scan for credential patterns and mask at source.

Implementing a log management programme

1. Inventory: log sources, destinations, retention, owners and legal obligations (the NIST INV play).
2. Write the retention schedule: one row per log category with searchable and retrievable periods, legal basis and disposal method. Get legal and privacy sign-off.
3. Fix generation first: audit policy, command-line auditing, cloud data access logs and all sign-in categories. Pilot on a small ring and measure volume first.
4. Build the pipeline: TLS transport, buffering at relays, a raw copy to immutable storage and a filtered copy to the detection platform.
5. Apply integrity controls: separate log account, object lock, integrity validation and restricted delete rights. Start object lock in governance mode, where privileged roles can still remove objects, and move to compliance mode once retention settings are confirmed, because compliance mode cannot be shortened.
6. Test: generate known events and confirm they arrive parsed in time; time a restore from the archive; simulate a collector outage and confirm buffering and silence alerts.
7. Roll back safely: keep previous forwarding paths running in parallel until the new path has matched event counts for a full reporting cycle; a rollback is then a routing change.
8. Dispose on schedule: expire data automatically unless a legal hold applies, and record the disposal.

Choosing an approach

• Small organisations: use native platform logging, export identity and control plane logs to one storage account with lifecycle rules, and prefer a managed log service to running a cluster.
• Large organisations: run a dedicated pipeline layer that routes, filters and enriches before any analytics platform, a data lake or object store as the system of record, and the SIEM as a consumer of selected data.
• Self-managed or SaaS: self-hosted stacks such as Graylog, Elasticsearch or OpenSearch give control over residency and cost, but you own index lifecycle, snapshots and hardening. SaaS platforms remove that work but price by ingest or retention: model your top sources over the full retention period.

Log review, monitoring and metrics

Log review means scheduled checks and alerts that prove logging works, plus detections over the data. Measure:

• Source coverage: percentage of inventoried assets with logs arriving centrally. M-26-14 sets collection coverage thresholds that rise from 50 percent at Level 1 to 95 percent at Level 4.
• Silent sources: number of sources with no events beyond their normal maximum gap, and time to restore them.
• Ingestion latency: 95th percentile time from event creation to searchability.
• Retention compliance: categories whose actual oldest searchable and retrievable record meets the schedule.
• Restore time: measured time to make archived data searchable for an investigation.
• Cost per GB by tier and source: to identify high-volume, low-value data to move or summarise.

Common failure patterns

• Relying on platform defaults: 30 day identity log retention means an intrusion found after a month has no sign-in history.
• Logs deletable by the attacker: logs in the same account, with the same administrators, as the systems they record.
• Logging disabled without anyone noticing: as in the Salt Typhoon and Ivanti cases above, edge devices and appliances stop forwarding and no alert fires. Monitor for silence, not just for events.
• Licence tier decides what is recorded: the Microsoft Storm-0558 Key Breach was first spotted by a government customer using mailbox access audit events that then came only with premium logging. Check what your tier records.
• Only human sign-ins collected: abuse of a stolen workload credential leaves no trace centrally.
• Archives nobody can restore: undocumented formats, lost encryption keys or restores that take weeks.

Framework mapping

Practitioner checklist

• Maintain an inventory of log sources with owner, destination and retention.
• Publish a retention schedule with searchable and retrievable periods and legal basis per category.
• Export identity sign-in logs, including service principal and managed identity categories, beyond platform defaults.
• Enable cloud data access logging for secrets, key management and sensitive data services.
• Forward over TLS with local buffering; retire plain UDP syslog across untrusted networks.
• Store a raw copy in a separate account with object lock and integrity validation.
• Alert on log clearing, logging configuration changes and silent sources.
• Mask passwords, tokens and keys before they reach logs, and scan the log store for credentials.
• Test a restore from the archive at least annually and record the time taken.

Standards and references

• NIST: SP 800-92 Rev. 1 (Draft), Cybersecurity Log Management Planning Guide
• OMB: M-26-14, Ensuring Effective and Efficient Agency Logging and Network Visibility to Defend Against Evolving Cyber Threats
• ASD's ACSC, CISA and partners: Best practices for event logging and threat detection
• NCSC: Introduction to logging for security purposes
• MITRE ATT&CK: T1685 Disable or Modify Tools
• AWS: Validating CloudTrail log file integrity
• Google Cloud: Cloud Audit Logs overview
• OWASP: Logging Cheat Sheet

Related NHI Mgmt Group resources: SIEM Guide · Windows Event Log Guide · Digital Forensics (DFIR) Guide · Incident Response Guide · Service Account Security Guide. Retention periods in this guide summarise published standards and regulations and are not legal advice.