A SIEM is one of the largest stores of personal data in most organisations, yet it is rarely designed, documented or protected as one. Authentication logs, proxy and DNS records, email gateway events, endpoint telemetry and cloud audit trails all identify people: usernames, IP addresses, device IDs, browsing history, message metadata and sometimes content. Data protection law does not stop security monitoring, and the GDPR explicitly recognises network and information security as a legitimate interest, but it does require a lawful basis, minimisation, defined retention, protection and transparency. This guide shows how to build security logging that meets those duties without losing detection value: what personal data sits in each log source, how to minimise and pseudonymise at ingestion, how to set retention, who may re-identify, and how to handle subject access and suppliers. It is for SOC leads, detection engineers, SIEM owners, DPOs and privacy engineers. Workforce monitoring beyond security logging, such as productivity tools and covert monitoring, is covered in the Employee Monitoring and Privacy Guide.
Key takeaways
- Security logging is lawful processing, not an exemption. Legitimate interests (GDPR Article 6(1)(f), with Recital 49 on network and information security) usually covers it, but only to the extent strictly necessary and proportionate, and only with a documented assessment.
- Minimise at the source, not in the SIEM. Drop payloads, bodies and fields with no detection use before ingestion; what never lands cannot leak, be requested or need deletion.
- Pseudonymise analyst views, keep a controlled path back. Keyed pseudonyms for user identifiers preserve correlation, and a logged, approved re-identification step keeps investigations possible.
- Retention is a decision per log class. Hot, warm and cold tiers with different access rules, aligned to threat dwell time and legal requirements, beat a single "keep for seven years" setting.
- Logs leak too. Secrets, tokens and personal data in logs are a breach waiting to happen, as the exposed DeepSeek log database showed, so protect log stores like production data.
The legal basis for security monitoring
Under the GDPR and UK GDPR, Recital 49 states that processing personal data to the extent strictly necessary and proportionate for ensuring network and information security is a legitimate interest of the controller. That supports Article 6(1)(f) as the usual lawful basis, alongside legal obligations in some sectors (for example security and incident reporting duties under NIS2 or financial regulation). It does not remove the other principles.
- Legitimate interests assessment: the ICO's monitoring workers guidance recommends the three-part test of purpose, necessity and balancing, documented in a legitimate interests assessment, and notes that workers can object to processing under this basis. Write one assessment per monitoring capability (EDR, proxy, email security, UEBA), not one for "security".
- Data minimisation and storage limitation: GDPR Article 5(1)(c) and 5(1)(e) apply to logs as much as to customer data. The ICO says personal information obtained from monitoring workers must not be kept for longer than necessary.
- DPIA: the ICO says a DPIA must be carried out before processing likely to result in high risk, and gives keystroke monitoring as an example. UEBA, DLP with content inspection, TLS inspection and full packet capture usually meet that threshold. See the DPIA Guide.
- Transparency: workers must be told about monitoring in an accessible way; covert monitoring is exceptional, authorised by senior management, time-limited and covered by a DPIA. Customers and visitors whose IP addresses and sessions are logged need coverage in the privacy notice.
- Special category data: monitoring can capture sensitive data incidentally, such as emails to a trade union representative or occupational health. Where that is likely, the ICO says you must identify a processing condition.
- Status note: the ICO's monitoring workers guidance is under review following the Data (Use and Access) Act and may change.
What personal data sits in each log source
| Log source | Personal data typically present | Detection value | Minimisation options |
|---|---|---|---|
| Identity provider and authentication | Username, UPN, IP address, device, location, user agent, MFA method | Very high: account takeover, impossible travel, MFA fatigue | Keep; pseudonymise usernames in analyst views; coarse location after the hot tier |
| Web proxy and secure web gateway | Full URLs including query strings, user, IP, categories | High: command and control, phishing clicks, exfiltration | Strip query strings by default; keep domain and path; category only for sensitive categories (health, religion) |
| DNS resolver logs | Queried names, client IP or host | High: C2 domains, tunnelling | Keep names; map client to host, not person, where possible |
| Email gateway | Sender, recipients, subject, attachments, sometimes body | High: phishing and malware delivery | Keep headers and verdicts; do not ingest bodies; hash attachments |
| Endpoint detection and response | Process command lines, file paths with names, user, network connections | Very high | Keep; mask file names under personal folders after the hot tier |
| Cloud and SaaS audit logs | Actor, IP, object names, sometimes content previews | Very high: data access, privilege changes | Keep events; drop content previews and message excerpts |
| Application logs | Customer IDs, emails, request bodies, session tokens | Medium to high: fraud, abuse, API attacks | Never log tokens or passwords; log customer IDs not emails; drop bodies |
| DLP and content inspection | Matched content snippets, file names, recipients | High for insider and exfiltration cases | Store match type and count, not the matched content, unless an incident is opened |
| Full packet capture | Everything, including content | High for short-term forensics | Short rolling window only; OMB M-21-31 sets 72 hours for US federal agencies |
IP addresses, device IDs and usernames are personal data where the organisation can link them to a person, which a security team almost always can. Machine and service account events are not personal data in themselves, but they often carry the name of the human who owns or used the account.
Minimising at ingestion
The OWASP Logging Cheat Sheet lists data that should usually not be recorded directly in logs, and should instead be removed, masked, sanitised, hashed or encrypted: session identifiers, access tokens, authentication passwords, database connection strings, encryption keys and other primary secrets, payment and bank data, sensitive personal data such as health and government identifiers, data above the logging system's classification, and data a user has opted out of or not consented to.
- Filter in the pipeline: apply drop and mask rules in the log shipper or pipeline (for example the collection agent, a stream processor or the SIEM's ingest pipeline) before data is indexed. Rules at the search layer leave the raw data on disk.
- Use schemas: normalise into a schema such as OCSF, ECS or ASIM, and maintain an allow list of fields per source. New fields default to dropped until reviewed.
- Secret scanning on logs: run detectors for API keys, tokens, JWTs and passwords on log streams and alert the owning team when one appears; the fix belongs in the application, not only in the filter.
- Prevent log injection: structured formats such as JSON, input length limits and output encoding, as OWASP recommends, also stop personal data being smuggled into fields that bypass masking.
- Cost alignment: minimisation reduces ingest licences and storage, which is often the argument that gets it funded.
Pseudonymised analysis with controlled re-identification
Most detection logic needs to know that events belong to the same user, not who the user is. ENISA's 2021 report on advanced pseudonymisation includes cybersecurity use cases such as telemetry and reputation systems, and the EDPB's draft Guidelines 01/2025 describe keeping pseudonymisation secrets separate, access-controlled, logged and rate-limited.
- Tokenise identifiers at ingestion: replace usernames, email addresses and employee IDs with a keyed pseudonym (HMAC with a key held in a KMS), consistently across sources so correlation still works.
- Keep enrichment separate: department, role and privilege level can be added as attributes without the name, which keeps UEBA peer grouping working.
- Re-identify on a case, not on a whim: analysts request identity resolution against an open incident ticket; the request, approver and result are logged; senior approval is needed for re-identifying people in HR-sensitive contexts.
- Exempt where justified: privileged and administrative accounts, service accounts and accounts already linked to an open incident can stay in clear for speed.
- Test: pilot with one data source and one use case (for example impossible travel) and confirm detection rates and analyst time are unchanged before rolling out.
Pseudonymised logs are still personal data for the organisation. The gain is in reduced exposure to analysts, suppliers and anyone who steals the SIEM data. For method selection and key handling, see the Pseudonymisation Techniques Guide.
Retention and tiering
| Tier | Typical content | Typical period | Access |
|---|---|---|---|
| Hot (searchable) | Normalised events used for detection and triage | 30 to 90 days | SOC analysts, detection engineers |
| Warm | Reduced fields for threat hunting and investigations | Up to 12 months | Senior analysts and incident responders on request |
| Cold archive | Compressed raw or reduced logs for forensics and legal needs | 12 to 30 months, or as required by law | Incident response with approval; immutable storage |
| Short window | Full packet capture, content inspection snippets | Hours to days | Named forensic staff |
| Legal hold | Records tied to an incident, claim or investigation | Until released by legal | Incident and legal teams |
- Justify periods with dwell time and duties: intrusions are often found weeks or months after they start, which justifies months, not days, for authentication and network metadata. Document the reasoning in the log retention schedule.
- Know external minimums: US federal agencies must keep most log categories for 12 months in active storage and 18 months in cold storage under OMB M-21-31, with full packet capture at 72 hours; sector rules (PCI DSS, financial regulators) also set minimums. These are justification for the period, not permission to keep everything.
- Delete for real: enforce retention with index lifecycle policies and storage lifecycle rules, and include backups and exported copies.
See the Log Management and Retention Guide for platform-level retention design.
Access, suppliers and transfers
- Least privilege in the SIEM: role-based access with data-level restrictions, so HR investigation data, executive mailboxes and DLP content are visible only to named staff. Log and review all searches in the SIEM itself, which is what OWASP means by recording and monitoring all log access.
- Protect log integrity and confidentiality: immutable or write-once storage for archives, encryption at rest and in transit, and separation between the SOC's log access and the administrators whose actions are being logged.
- MSSPs and cloud SIEMs are processors: they need an Article 28 contract that covers purpose limits, sub-processors, location, retention, deletion at contract end and support for subject requests. The ICO notes that using a third-party tool does not transfer responsibility. See the Cybersecurity for MSPs and MSSPs Guide.
- International transfers: follow-the-sun SOCs and offshore analysts are transfers of personal data. Cover them with the appropriate transfer mechanism and a transfer risk assessment. See the International Data Transfers Map.
- AI in the SOC: sending log extracts to an LLM service is another disclosure. Apply the same minimisation and pseudonymisation to prompts, and check the provider's retention and training terms. See the Agentic AI in the SOC Guide.
Subject access and other rights
- Logs are in scope: the ICO says monitoring data about workers must be made available to them on request, subject to exemptions. Plan how to search the SIEM by person, including through pseudonyms.
- Use exemptions narrowly: where disclosure would prejudice an investigation or reveal detection methods, apply the relevant exemption case by case and record why, rather than refusing all log data as a category.
- Objections: under legitimate interests individuals can object, but compelling legitimate grounds such as protecting systems and other users will usually override an objection to core security logging. Record the assessment.
- Erasure: retention limits are the main erasure mechanism for logs; deleting individual entries on request can damage evidential integrity. Explain this in the privacy notice. See the DSAR Guide: Handling Data Subject Requests.
Where identity fits
Identity data is both the most valuable and most sensitive part of security logs. Two practical points matter. First, map non-human identities (service accounts, workload identities, API clients) to owning teams rather than named people in the logs, which preserves accountability without adding personal data. Second, the SIEM's own ingestion connectors, API keys and service principals have read access to almost everything; manage them as privileged NHIs with rotation, scoping and monitoring. See the Cloud Logging and Audit Trail Guide.
How to implement: a phased approach
- Inventory log sources: for every source feeding the SIEM, data lake or MSSP, list the fields, the personal data categories, the detection use cases that need them and the current retention.
- Document the lawful basis: write legitimate interests assessments per capability and DPIAs for high-risk ones (UEBA, DLP, TLS inspection, packet capture), and update privacy notices for staff and customers.
- Cut fields with no use case: drop or mask fields that no detection rule, hunt or investigation playbook uses. Run in parallel for two weeks and compare alert volumes before switching off the old feed.
- Introduce pseudonymisation for one source: start with proxy or SaaS logs, prove detection is unchanged, then extend.
- Set and enforce retention tiers: configure lifecycle policies per log class and verify deletion, including in backups.
- Lock down access: role-based and data-level access in the SIEM, search auditing, and review of who can re-identify.
- Contract and transfers: review MSSP and SIEM vendor contracts and transfer mechanisms.
- Roll back safely: keep previous pipeline configurations versioned so a masking rule that breaks a detection can be reverted quickly, and fix the detection rather than removing the control.
Detection and measurement
| Metric | What it shows | Target |
|---|---|---|
| Log sources with a documented field inventory, use case and retention | Accountability coverage | 100% of SIEM sources |
| Secrets or tokens detected in log streams per month | Application logging hygiene | Falling towards zero, each fixed at source |
| Share of user identifiers pseudonymised in analyst views | Exposure reduction | Rising to all non-privileged users |
| Re-identification requests without a linked incident | Misuse of the re-identification path | Zero |
| Data held beyond the retention schedule | Storage limitation in practice | Zero, verified quarterly |
| Mean time to answer a subject access request covering logs | Rights handling readiness | Within the statutory period |
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| GDPR and UK GDPR | Articles 5(1)(c), 5(1)(e), 6(1)(f), 15, 28, 35; Recital 49 | Minimisation, retention, lawful basis, access rights, processors, DPIA |
| NIST SP 800-53 Rev. 5 | AU-3(3) Limit Personally Identifiable Information Elements; AU-9; AU-11 | Limiting PII in audit records, protecting and retaining them |
| NIST SP 800-53 Rev. 5 | PT-2, PT-3, SI-12, AC-6 | Authority and purpose for processing, retention, least privilege on log data |
| NIST CSF 2.0 | PR.PS-04, DE.CM-01, DE.CM-03, GV.OC-03 | Generating logs, monitoring networks and personnel activity, legal and privacy requirements |
| ISO/IEC 27001:2022 | Annex A 8.15 Logging; 8.16 Monitoring activities; 5.34 Privacy and protection of PII | Logging and monitoring controls that respect PII obligations |
| CIS Controls v8.1 | Control 8 Audit Log Management | Log collection, retention and review. See the CIS Control 8 Audit Log Management Guide. |
Common failure patterns
- Logging everything because storage is cheap: full URLs, request bodies and email content ingested by default, with no use case, creating a store nobody can justify to a regulator.
- Secrets and personal data in application logs: tokens, passwords and chat content written to debug logs and then exposed. In the DeepSeek database exposure, an unauthenticated ClickHouse log table held over a million entries with plaintext chat history and API keys.
- One retention period for all logs: seven years for everything, or the SIEM default, with no reasoning recorded and no deletion from backups.
- Unrestricted analyst search: every analyst can search any person's browsing and email metadata, with no audit of searches.
- Forgotten processors: MSSPs, cloud SIEMs and AI assistants receiving logs without processor terms or transfer assessments.
- Refusing all access requests for logs: treating security logs as automatically exempt instead of assessing each request.
Practitioner checklist
- Inventory personal data fields in every log source and link each to a detection or investigation use case.
- Write a legitimate interests assessment per monitoring capability and DPIAs for high-risk ones.
- Drop or mask fields without a use case in the pipeline, before indexing.
- Block tokens, passwords and secrets from logs at the application, and scan log streams for them.
- Pseudonymise user identifiers in analyst views with keyed pseudonyms and a logged re-identification workflow.
- Set retention tiers per log class, document the reasons, and verify deletion including backups.
- Apply role-based and data-level access in the SIEM and audit every search.
- Put Article 28 terms and transfer assessments in place for MSSPs, cloud SIEMs and AI services.
- Prepare a tested process to answer subject access requests that include log data.
Standards and references
- EUR-Lex: Regulation (EU) 2016/679 (General Data Protection Regulation)
- ICO: Data protection and monitoring workers
- OWASP: Logging Cheat Sheet
- NIST: SP 800-92 Rev. 1 (Initial Public Draft), Cybersecurity Log Management Planning Guide
- OMB: M-21-31, Improving the Federal Government's Investigative and Remediation Capabilities Related to Cybersecurity Incidents
- ENISA: Data Pseudonymisation: Advanced Techniques and Use Cases
- EDPB: Guidelines 01/2025 on Pseudonymisation
- NIST: SP 800-53 Rev. 5
Related NHI Mgmt Group resources: Employee Monitoring and Privacy Guide · Log Management and Retention Guide · Pseudonymisation Techniques Guide · GDPR Compliance Guide for Security Teams